Starting in the mid 80s I had cabins in the Pioneertown/Pipes Canyon area. If you don’t know the area it is on the way to Joshua Tree. Extremely beautiful, but not for everyone.
Cabins used to be cheap, and my first one was a beauty on 2 1/2 acres for $16k. As time went on, prices went up, and in 2005 I sold my last place — which had a well and everything — because it had appreciated. Then the Sawtooth fire hit and devastated the area.
Still, I think the high desert is fantastic, and in Joshua Tree you can get some nice, affordable acreage and build something to suit your architectural fantasy.
The desert is for you if you like quiet and star-filled skies. Not much to do except hike and that is exactly the point.