Apart from La Jolla, which is amazing if you are independently wealthy and don’t have fight traffic on your way to work, MH is my favorite place to live. It is a beautiful area near restaurants. You can be anywhere in the city within 15 min. There is only 1 very big problem, I can’t afford it.
I love downtown too and would consider living in a high rise, but I have too many toys that need a garage.
As far as moving from TX, you are right about affordability. I could live in a McMansion there. Although you should enjoy being in your house because it is too hot to go outside. I am willing to sacrifice an amazing house to be within 10 min of the beach, and live in an air conditioned climate.