We’ve lived all over the world, but, as Natives, have always kept our “home base” in San Diego, simply because our entire family likes it better than any other place in the world. We know that may not be true for others, and everyone has to do what they feel is best.
As other posters have mentioned, if living in CA becomes a financial issue, it probably is better to look elswhere–especially if your kids won’t be able to afford to buy homes here, or will have to settle for a compromised quality of life.