I drive down there all the time, most recently in June for the Baja 500.

You just have to at least look tough and no one will bother you. Leave the tight metrosexual gear at home and dress like a man.