Why is nice diction only associated with females?

my voice doesn't sound feminine or homosexual and i'm heterosexual. my voice sounds just as proper as someone from england and suave as a frenchman but a lady on the phone "will that be all mam"i asked around and others said my voice sounded masculine she knew very well i was a man she just felt she had to attack me out of wanting it to only be that only females can have a nice sounding voice.