This was one of my husband’s concerns as well when we were looking for a house, especially when they were younger.
We bought a house with a pool, and couldn’t be happier. Our kids were a bit older when we finally bought (9, 8, and 5 at the time) which helped, and we had them in swim lessons for quite a while before we moved in.
Do know that it will cost quite a bit to maintain the pool. I grew up with pools and had rented a pool house in my 20s, but didn’t remember the cost to run the pump being as much as it seems to be now (~$150/month for a main pump and smaller pump for the pool vacuum??? We’re wondering if our meter is not working properly). Also, factor in ~$100-$120/month for a pool guy if you don’t do it yourself (we’re seriously considering taking this on ourselves since our pool guy maybe vacuums maybe twice a year and hardly brushes things down — he’s definitely not worth the money). Then, you’ll have the maintenance issues that you’ll have to budget for over the years like re-plastering, pump repairs/replacement, etc.
No matter what, a pool fence is a must when the kids are as young as yours, IMHO. This should run about $2,000-ish if you hire a contractor to supply and install it.
It really depends on where the house is, too. Is it in the coastal fog area, making the pool less desirable; or is it in the inland area, which will make the pool more desirable and valuable upon resale. Do you and your family think you’ll enjoy swimming on a regular basis? If so, it would probably be worth it to buy the pool house. For those who really enjoy swimming and playing outdoors, I think pools are wonderful. Most people who have a pool and kids love the fact that your kids and their friends will probably be hanging out at your house…which makes it a plus if you want to know with whom they’re associating and what they’re doing.