You can easily back out especially since you have not formally released the contingency AND the square footage discrepancy is causing you hearthache. Simply ask your agent to send a cancellation. Sign it and request your full deposit back.
One is never FORCED to sign anything. You obviously cannot cite the appraisal as being the reason but you can cite that the square footage not matching the tax records and not having an adequate explanation should do the trick. If you are not sure of the measurements Cornerstone Appraisals will go out and only measure the home for like 100 bucks.