I think building isn’t so bad, especially if you understand the process and can manage it yourself.
The problem comes in getting all the approvals and government stamps needed to pull a permit. THe difficulty of this varies greatly by neighborhood. I’m sure in north county there are some places which are easy to get a permit and some places which are nightmares. I would focus energy on figuring out all the nasty rules that can stop a design in its tracks.
I recommend getting help from an architect who is used to moving projects through the government. You don’t need a great architect, but you need an architect who really knows the rules and/or how to get around them.
Also, if you get a construction loan, you have to make sure you get the project done before the construction loan expires. Keeping things moving fast enough can be difficult and stressful.
If you “have your eye” on a project that is already permitted, I say go for it.
