I’ve actually had a neighbor (female) tell me that if someone knocked on their door, they wouldn’t answer. Lots of people buy in a Condo/Townhouse project for exactly that level of privacy. We get about 5% participation at mine and I’m certainly not helping things.
The fact that the board is willing to sue isn’t surprising as normally the most activist of one’s neighbors run for the board. You might try just talking to your immediate neighbors, people who at least know of your existence, to get them to come with you to a board meeting. That would give your objection more weight.
Josh