I love having my house paid off. You will be able to save $2500/mth in not having a mortgage, plus gain about $2500/$3000 month in rent while you are not there. That is $60K/yr you will be able to save, plus whatever else you save with the higher paying job you took. You will still be able to deduct expenses, taxes, and depreciation against your rental income.
In the early 90s, the Japanese probably thought their interest rates would rise also. They’re still waiting. We are in a deflationary cycle as credit contracts around the world.
Take the security and move on. I still have money in a savings account that I am looking to invest. It is, in my opinion, a difficult and frustrating time in history to invest money and be sure you are doing the right thing. If you need that money in the future, you can always mortgage the house again. It isn’t as if you have thrown it in a trash can. With $400k down, you have already assumed all the downside risk if prices keep falling.