I would say that most buyers are better off having some representation in the transaction. I think any conflicts of interest could be resolved by changing the compensation structure a little.
Pick an arbitrary price point that’s well below what the buyer is prepared to spend and base their agent’s commission on that rather than the actual sale price they negotiate. Bump the percentate up a little to make up the difference. That eliminates all motivation for steering the buyer to properties above that price point so that the agent can concentrate on matching the buyer to the right property at the right price.
Either that, or become a much more active participant in researching and analyzing the sales data so that you can make a more informed decision on your own, hence reducing the need to rely so heavily on your agent’s advice.