I’ve often been fascinated by this, most particularly on weekdays (and some weekends) when returning from Legoland right after it closes.
Just some offhand guesses:
1. People returning from LA. They leave early and/or spread out enough to avoid causing a morning rush, but in trying to get back before sundown or dinner, they create an afternoon rush.
2. People returning from places like Legoland that close around 5. 🙂
3. Something in the way the traffic pattern and road design combines makes it more vulnerable to heavy traffic than the northbound lanes.
4. A more general effect I’ve noticed is that the afternoon rush hour delays on any freeway are more spread out than the morning rush hours. I’ve never really figured this out.