Capitalist, free market economies only work when everyone makes rational financial decisions for themselves, ideally within the confines of the law.
When an institution chooses to lend money to an individual, there is a contract that exists between the institution and the individual. Both parties go in knowing the rules of the game and agree how to play. Both accept certain risks. It’s been hammered here in many threads, but when a bad loan is issued, in most cases both the lender and the borrower have at least some fault in the matter.
Borrowing from the bank is not like borrowing from your parents or brother. It is a business transaction, and default is always an option in business. Obviously it isn’t without cost, as that person isn’t going to be able to borrow money for a home again for a very very long time.
As for hurting the rest of us, I actually think at this point if more people walked away from homes they couldn’t afford it might help us all out in the long run (or at least those who would benefit from lower housing prices). More defaults means more inventory for the banks, which means they need to both negotiate more with underwater buyers and unload more of the properties they are holding for lower prices.