We live in Rancho Santa Fe, not far from 4S, and have friends who live in both of the areas you mentioned.
If you will be commuting to a job on a daily basis, you should definitely consider choosing the location closest to your employer.
San Diego traffic has become pretty bad over the last few years, and you don’t want to waste any more time than you have to in gridlock.
If the employment consideration is not an issue, then you need to visit both locations and determine which best suits your personal lifestyle–schools, shopping, golf courses, beach access, etc. You might also want to check out Carmel Valley (92130).
We just happen to prefer the RSF, Del Mar, Torrey Pines areas for schools, beaches, golf, etc. These areas seem to fit our lifestyle best, but everyone is different.