Reasonable Accommodation process seems really intrusive need advice

I hope that this is okay to post here because it has not been confirmed but it's the worst kept secret at my office. I have had a reasonable accommodation for remote work since 2021. Our agency has had a steep increase in the number of RAs being issued since probably the beginning of 2024.

Recently we were told that all new RA requests had been put on hold since spring of 2024 while they were waiting to implement a new step in the process before approving new ones. Last month the new plan was leaked from someone in the EEO office. They are bringing an agency doctor on staff to review RA requests. They were waiting for funding approval but now they have it and are in the process of either hiring or contracting this person. The plan outlined the steps - if a person's doctor submits an accommodation of remote work then the agency doctor is supposed to review it and decide if that's accurate or not. The plan requires all new and renewals to sign an agreement to allow the agency doctor to directly contact your private physician and access your medical records to verify the accommodation. I'm just not sure I feel comfortable with this. I need the RA to be able to work, but I just don't trust someone that works at my agency having access to my medical records. I love my job but I already know how gossipy people are in some of these offices and I just don't think that things will stay private. I'm also worried it would be used as some sort of retaliation if any of my personal medical history is available. What would prevent that doctor (or a nosy person in the office with access) from providing it to my supervisor or someone else in my chain of command or if I apply for a new job in the agency sharing it with potential supervisors? I'm sorry this is so long-winded. I guess my question is, is this legal? I'm not asking for legal advice I know that this is a community of feds but I was just hoping that maybe an EEO staffer or someone could provide some input on this?

Edited to add- I don't care about the doctors discussing my RA request what I do care about is the scope of what they request and how they get it and how the information is protected. I have medical issues unrelated to my RA that I don't think the agency needs to know about and don't know how to limit the medical records to the specific condition. Those of you that have worked through this experience give me some peace of mind so thank you. It just isn't how my initial approval or renewals went so I was fearful of the privacy issue.

For those who think I am trying to fake an RA I'm not. I am also ticked that people abused it and and made it harder than it needs to be.