We considered Santa Barbara, along with The Bay Area, years ago, when housing in CA was still “affordable,” since we have family and friends in both locations.
Ultimately, we opted for Rancho Santa Fe, which is a better match for our preferred lifestyle–and appreciation in both areas has been great–so no complaints here, but, as has been mentioned, SoCal coastal towns can be challenging for new buyers.