It’s simple, too many people haven’t been good for the money.
So, given the so many people aren’t good for the money, rates have to be that much higher to provide the risk adjusted return needed to buy the mortgage instead of just buying a treasury bond.