Right now in my house we have a Christmas dinner guest who is physically and mentally disabled, and lives in a group home of this sort.
I helped his relatives move him into his group home. At the time the neighbors had hurtful protest signs across the street. Since then there have been no problems at all.
Whenever you buy a house there is always a risk that someone you don’t like will move in nearby. Maybe you are “right” maybe you aren’t. I have a relative who felt obliged to sell his co-op in New York because of noisy new neighbors.