I had the same requirements 2 years ago when looking for a house. I did buy the house with a small pool/spa because I liked everything else. I have to say, I was hesitant first, but it’s the best decision.
I have 2 children (4.5 yrs-old & 5 months-old), and the first thing I did when moving in was a safety pool fence. (I found one on Craigslist & installed it myself to save a buck).

Go for it if you like the house, if you don’t like the pool, it might cost you $10K to get rid of it/fill it with dirt.