Men busting into the closed bathroom

I've had this happen with at least 3-4 guys I was dating. I'll be in the bathroom with the door shut and they feel free to come in without even knocking. They always apologize and quit doing it when I call them out, but it's always a "sorry, but".

"Sorry, but my ex was never bothered by it."

"Sorry, but I thought you were in the shower."

And? The door is closed, does it matter wtf I'm doing in here?

Has anybody else noticed this pattern?

Edit: to everyone asking why I don't lock the door, there isn't one. Wouldn't have expected so many victim blaming responses to this post. It isn't about me needing to lock the door, it's about men needing to not invite themselves in when the door is closed.