Don’t be a fearful little fraidy-cat is how. Speak to the people over the phone first — if they seem like scumbags, you can always be unavailable for the test drive.
Meet them in front of your house. It’s just another house, and they won’t be seeing the inside, only what’s visible from the street.