If the house had kept it’s value – market stayed flat rather than declining – would you be anxious to move out of your house? Or would you be content to stay and make the house home?
It sounds like you can comfortably afford the payments on your income. Especially since you were able to do the loan mods. So the only factor is that your neighbors are buying similar houses for less – or paying the same amount to get bigger houses… It doesn’t sound like it’s an issue of you not being able to afford the house you bought.
I’m old school – I really dislike the notion of treating your primary residence as an investment – I think you should buy the house you plan to live in because it meets your shelter needs, your school district needs, your nesting needs. And you buy long term…
My advice – filtered through the thought process of buying a house for shelter, not as an investment, is that you should stay, pay your mortgage, and be grateful for a nice home to live in.
As far as the crowded kids… Like CA Renter, my kids share a bedroom – it’s by choice. Our house is similar in size (2k sf) and is plenty large enough to raise a family in. You’d be surprised how little space you need to raise a family. My husband grew up one of 6 kids in a 1200 sf house with 1 bathroom. He didn’t suffer psychic harm.