If you and your husband decide to cancel fight for your deposit on the grounds that the sellers rep and mortage guy are being misleading.Accuse them of teaming up on you if they are in anyway affiliated. Ask the builders sales Rep if they have ever sold a model like the one you are interested in for less or if the would ever consider it in the future.Ask them what they would do for you if that happened,especially since you feel you are being mislead.
Put it in writing registered mail.Quote some of the things the posters here are saying. If they offer you a lower price ,which they might take it or tell them you don’t trust them and that the price would have to be 20%or more lower to make sense stand your ground. CC and send a copy to the mediation center of the local NAR if you have to.Maybe take them to small claims. I know I would get my deposit back.