If your relationship status doesn't define your self worth, why do people use "no wonder you're single" as an insult?

I feel like the people who tell you that you shouldn't define yourself by how many rejections you take or how many people like you are the SAME people and use "no wonder you're single" as an insult soon as you say something they don't like. To me it just seems like duplicitous BS.