Grammar help

I have a confusing situation with pronouns:

The main character (3rd person limited), is in an argument with an unknown person. Both are male. The main character has amnesia and knows neither of their names.

"He (other person) pushed on his (mc) chest and knocked him (mc) down to the ground. He (other person) then said to him (mc), "You lasted longer than I thought you would." How can I make this more clear without switching POV to the first person?