A lot of variables to consider, your age, your risk tolerance, kids (college fund), desires (possibly wifes desires here too), where is the money coming from? In my case I have less to pay off at a lower rate, however I can still deduct and not get hit by caps. Personally I would pay it off if the money is coming out of the stock market, but hey I’ve been wrong many times before so I think you should consult the wife. If the money is sitting in a money market account then I would say it is probably a wash and don’t lose any sleep over it. Max out 401k/IRA/college fund and then count your blessings.
You should have already refi’d into a 15 year to get the best rate and deduction, if you didn’t then that’s one more thing you can kick yourself for.