There is a certain class of rich that does retire here, namely many ex-professional athletes. The more recent retirees (last 10-15 years) have a very comfortable nest egg and gravitate to this area.
I realize that a baseball pitcher is probably not the cultured type of rich person that you are talking about in terms of saving a housing market. However, it’s not hard for some of these guys to have $60-100 million dollars in career earnings these days. RSF and coastal areas are chock full of these guys.
I’d say many of these guys retire here because relatively speaking they are fairly young to have amassed that fortune and want a good place to raise kids. In general folks that would rather be somewhere like RSF which has big lots, and tucked away properties.