I don’t see what’s so horrible about downtown. Good view of the waterfront and marina, close to the airport if you’re someone who needs to fly out frequently (takeoffs happen the other direction), lots of pretty 1920s/30s buildings.
Coming from NYC, I didn’t get a super-diverse vibe from downtown. More of a business district and less busy after hours. North Park and the areas around it feel much more diverse to me.