Chris,
Much of Olivenhain has 2 acre minimum lot size zoning which wont change. Another area you might really like is the Elfin Forest which is just east of Olivenhain. Much of it is unicorporated and in the county. Minimum lots sizes can be even larger out there and it is even less likely to be developed than Olivenhain and can be quite a bit cheaper. I’ll try to find an mls # to post if there is anything for sale there so you can get an idea of what its about and where it is.