Yes, refining is a problem and a bottleneck at times that can drive up prices, but it doesn’t tell the whole story. At least not yet. Crude oil costs about $75/42=$1.8 / gallon. If you add transportation, taxes, gas-stations, credit-card fees, you are already at about $2.50, so refining costs about $0.70. Also, recently most of the refining is done in the Middle East and even in Europe to make up for our shortage in refineries. Yes, we could earn more, and we currently give away some profit, which is stupid, but it’s not the only problem. Higher crude prices have their validity, and some it is justified simply by inflation.
