I think the main point is that we could be allocating our personal resources in places other than housing. That extra 20% of our monthly income could be invested more wisely in sectors other than housing (construction, financials, realtor's BMW 740i, etc.).
I'd be very interested in seeing some numbers showing what percentage of people's monthly earnings actually went to illegal immigrants in the construction industry. It is well known that many in the construction industry here in SoCal are here illegally, or being paid under the table, or both. Could it be that the additional monthly expenditures for housing have actually hurt our economy because many of those dollars don't go back to the government in the form of taxes, or don't go into paying health care costs? Would it add even more strain to many people who have to pay more for these services?