We are richer when house prices go down.

It is relative.

If no one owned housing, then it would benefit everyone. If everyone owned housing, then everyone would in fact be poorer if house prices went down.

As it stands, the market peaked exactly when there were a record number of home owners.