You certainly don’t have to apologize for enjoying the San Diego lifestyle for over 20 years and wishing to move forward again. We have friends who are in the same situation–long-time San Diego residents with great homes that were paid off long ago (even though we’re all only in our 40’s)and lotsa cash–waiting for the right time to move up again.
Like you, they are fully aware of the real estate situation in San Diego County, and are being very careful. They have everything they want, are grateful for what they have, and would never wish anything but the best for others.
Probably best just to ignore miserable people, and focus on advice from those who have achieved their ultimate goals. After all, they are the ones who have proven themselves in life.