I don’t think early elementary school it makes any difference to be in a rich versus upper middle class district. They’ll be finger-painting and ABCing the same either way. Maybe by high school the difference in AP classes and peer influence will make that difference important.
And socially, do you want to be the poorest kid in CV or have a background similar to peers? Parents of small tots tend to socialize with their playmates.