IANAL, but I would recommend that you get a copy of the HOA’s bylaws, that should spell out the HOA’s authority and or limitations of the HOA’s ability to restrict individual home owners access to the community facilities (HOA clubhouse & pool), specifically the steps to amend the bylaws or rules of the HOA facilities. You should have been given a copy when you bought the house. Then consult an attorney who deals with property law, specifically restrictive covenants.
(IANAL) I am not a Lawyer,