The downside is that you add restrictions to how much expansion you can do in the future.
Depending on the size of the lot and the location how big of a deal this is could vary, particularly in La Jolla. For instance, in the Muirlands with a big view lot, a lot of the value of an older home would be that it could be torn down and a big McMansion built in it’s place. But with a small lot in the village, you might not be able to build much bigger anyway.
So, evaluate not only what you might want to do in the future, but also what future owners might want to do. You’re basically trading potential future expansion for low taxes today.