I think the answer is that polygamy is well outside the framework of marriage. It’s not hard to see how a white guy and a black woman can have a normal marriage. I think there are ample examples that a gay couple can have as close to a normal marriage as possible without opposite plumbing; controversy aside, it’s become rather commonplace.
On the other hand, polygamy in the US is neither common place, ethically acceptable (by which I mean “is it OK if the boss knows?”; the gay engineers I work with don’t advertise, but they do have family pics up), nor does it map (legally or taxably) onto the way marriage works now. It opens so many cans of worms (partial divorce, child custody, rewriting tax codes) that it can’t easily be done without extreme legislative intervention. It’s also much harder/impossible to frame as a civil rights/discrimination issue since polygamy is rather indisputably a choice.
All that said, I do have to ask, why is polygamy indisputably wrong? Polygamy and polyandry have been practiced by dozens of cultures throughout history. The groups favoring it now tend to subjugate their women, but it’s not clear to me that this is a necessary flaw. It’s also not clear to me that I want government regulating this sort of thing… as was pointed out above, just because I’m a white, straight, married American male doesn’t mean I will always be on the right side of public tyranny.