It doesn’t have to be all or nothing, you can pay down 50% of the mortgage and get a modification to keep the same term. Might cost $500 but you then get a 50% lower payment.
With the remainder you can invest in a combination of Vanguard Muni and closed end muni bond funds to make 3-5% tax free.
The monthly savings can the be put into two index funds 50% muni bonds and 50% total stock market.
I personally like: VPGDX Vanguard Managed Payout Fund Investor Shares
Later if you decide you want to pay it all off, you can or you can continue with stocks or used some of the funds to buy more property.
We paid off our first home about 5 years ago. It allowed us to rent it out and buy a bigger house (short sale) in 2013. The intersting thing was, the rent from the first home covered the mortgage on the new one. So we kept doing this three more times to get to 6 properties, of which 2 are free and clear. We owe about 1.5M on properties worth 4.2M now and the cash flow is good enough so work is optional.
I continue to work but to buy the things I wouldn’t be comfortable selling assets to buy like vacations, new cars, water treatments, major landscape projects, additional solar panels etc.
So paying off a mortgage can be a great step to financial freedom, but don’t sweat it too much as there are many options.