Illegal or not, they are still spending and contributing to the local economy. If large numbers of people leave because of a reduction in jobs, it will have an impact. They are lower wage earners, so you could argue that the impact would not too large. However, all of the those people have to pay rent, buy groceries and shop at Walmart.
The loss of jobs in real estate will have an impact, too. How large that will be is also debatable.
Another big unknown is whether consumer spending will dry up when people quit feeling house rich and start feeling house poor.