“Don’t really care for” is pretty vague. I’ve had co-workers I didn’t really care for but had perfectly functioning professional relationships with. You couldn’t have gotten me to go have a drink with them for less than say $1,000, but we were able to function at work.
On the other hand, there are people you just don’t get along with at all. There’s no reason you should feel obligated to pay someone so you have to be around them. I don’t personally think you’d be out of line switching agents if your dislike of the first one is more of an extreme distaste rather than mild annoyance.
