Removing contingencies requires you to actively remove them in writing, tehy do not disappear with the passage of time so your agent is most likely correct. If you dont remove them, the listing agent can send you a “notice to perform” which allows them to cancel if you do not remove the contingency by that deadline (usually 24 to 48 hours). In that case you could lose the house but not your deposit.