yes, i think you're right, at least
the last two. i just googled the first one, it is uncommon but it exists.
so just to summarize if the guy knows he is sending it to one person, but he doesn't know this person's gender, he should use "dear sir or madam" if he is sending it to more than one person, or if he's not ure who is going to see the letter, he should use "to whom it may concern" like you said , he should use "dear sirs" or dear madams" for two (or more) men or two women, correspondingly. and of course, using "Dear Mr/Mrs ___ " is always the best choice if you know the name.