I can’t comment on whether or not you should sell and move into a 500K-600K home. One would need far more information to give reasonable advice.
What I would say is that buying a house for $650K and selling it six years later for even $1.0M is not bad return for your money at all. If the rest of your personal particulars (health, personal well-being) indicate that selling and moving (where?) are in your best interests, I would think that the real estate transaction is is an additional factor working in its favor.
