The yield on your residence’s home equity is your use and enjoyment of your house. In government statistics this is called owner equivalent rent. It is, in fact, growing if the rental value of your house, which you avoid by owning the house, is growing.
When you paid down the mortgage, your interest cost went down but your use/enjoyment did not.