I say do your planning, wait for the right time, and go for it. I had similar thoughts back around 2000 and kick myself for not going through with it. I envisioned buying 5 acres or so in Valley Center and building my own home. What you have not heard so far is the value of creating something with your own ingenuity and two hands. Like learning a second language or obtaining a higher education degree, it is one of those things that you will cherish for the rest of your life and NOBODY will be able to take it away from you!
Like all complex matters, I recommend breaking it down into more manageable components. I don’t have any real-world experience in this area, but you need a roof, walls, electrical and plumbing at the very least. Start from there and work your plan around it.
One thing that I think is worth mentioning is the value and importance of a basement. For those of us born in the south west, that is another floor level below ground. I think a basement holds significant value in terms of added living space and ongoing energy savings. The basement needs little if any cooling in the summer, and (if there’s a fireplace there) the heat will rise to warm the entire house. If you are considering building your own home from scratch, why not go all out and consider adding a basement to it? As always, check with local building codes to ensure this is a viable option.