I”m going to go against the majority here and recommend that you not buy. When I read your 8 bullets, I saw 8 reasons not to buy, the biggest of which is your and your wife’s luke warm feelings about the house. “It would be something to do” is not a reason to buy.

Also, just because you can afford the monthly does not mean its a wise financial decision. If prices drop 20 %, you just wiped out your ability to trade up to that house that you are more than “luke warm” on. Your down payment is gone (the down payment you really don’t have).

And you have been renting for more than 10 years and don’t have the down payment yet? You are rationalizing in an irrational matter when you argue that borrowing from your retirement account is “OK” because the government “will take it” anyway. Come on.