Isn’t the main issue that you then subject yourself to the HRB. Basically an HOA with Governmental Authority to review and authorize any modifications to your home. Like an HOA it’s volunteer based and YMMV depending on the board. But, in some areas, they dictate right down to the colors you can paint.