Any listing agreement may be cancelled at any time. The first course of action is to simply address the issue directly with the agent. Simply call him up and ask him to send you a cancellation of listing agreement. This is a simple 1 page CAR (California Association of Realtor) form. When fully executed, (signed by both parties) it essentially terminates your listing agreement.
In the event that the agent resists or does not agree, then the next step is to go to the broker and ask the broker for the same thing. In all likelihood the broker will ask if you would like to use another agent, etc… If you do want to use another agent then great, the problem will be solved, if not then stick to your guns and demand the broker release you from the listing agreement. Make sure that the broker sends you the cancellation of listing form.
In the event that the broker disagrees then let the broker know that your next move will be to file a formal complaint with the California Dept of Real Estate. Thus the broker has a choice of retaining an unhappy client by force and getting a complaint lodged against him, or simply letting you out of the contract. 99 times out of 100 the brokers let people out of the contract. There are other steps that you can take if he does not but I would be willing to bet that he will if you go this far.
If you want I can look over your listing agreement to let you know if there are any ramications readily visible in the document. Even if there are you will most likely get the broker to drop them. For instance, there was a couple who had an agreement with Remax in the spring that showed a 2k expense fee to be paid in the event of cancellation. Well they did cancel and no they did not pay that expense fee. It took many phone calls and such but in the end it worked out and they did not have to consult a lawyer or file a complaint with the DRE. If you are firm and stick to your guns it will be fine.
The fact of the matter is that yes you can cancel at any time. The practical aspect is that if you do it correctly and use the proper documentation and show the broker you know the law, it will be easier and less time consuming.