Partner feeling worse about themselves as I lose weight

I am a few pounds away from my vanity goal weight and I have been working really hard to get fit and improve myself. My partner has not gained significant weight, he is about the same as he was when we met 5 years ago. He did have a period were he lost some weight, but has sense gained it back. He is maybe slightly over weight by BMI, but i dont know his exact weight. He said last night that he has been feeling like he wants to sleep in full pajamas to cover himself.

Which I took as him both not wanting to see himself and not wanting me to see him. Now I have never known him to be a very confident person to begin with, but it seems like it has only gotten worse as I've gotten leaner. He has said before that I will get so attractive I realize I don't want to be with him anymore. Which is 100000% not true, he is my person and I think he looks hot as he is.

I cook healthy meals at home, he eats what I eat. But it is a different story at work. He either has days he eats normal or days he doesn't eat at all then smashes half a giant tub of peanut butter in one sitting late at night cause he is starving. I don't say anything in those moments. I only talk about his eating if he has given himself a headache or acid reflux from not eating anything at all.

I tell him all the time that I find him attractive. When he self depreciates I give him a compliment and encourage him to be kind to himself. I've told him we can go to the gym together. I've offered for him to do my home work outs with me. But I only offer those whe. He has made a comment. For example I'd say "I think you are super hot. We can start going to the gym together if you think that would help your opinions of yourself." Is that not what I should be doing?

I know I am not responsible for how he feels about himself, but I don't know what to do. I want him to feel confident in himself. I want him to want to be naked around me. He is a very attractive man. I don't know what I could be doing better to make him see that I don't see him any differently than I used to. I am sure I'm not the only person who has had to deal with this during their weight loss.

I have body issues myself that have not gone away with weightloss. I understand not feeling confident in yourself. I know I am critical of myself still around him and I'm starting to feel like maybe that is causing him to think that I feel that way towards him? I just don't know what to do to help, cause I feel like I'm making his confidence lower just by being around.

Tdlr: I've lost weight, my partner has staid the same. His self confidence has gone down and gotten worse the closer I get to my goals. I'm looking for advice on how to help