For sure! I have had that with people assuming I am gay yet I have never been attracted to anyone of my birth gender.
But some people can read subtle physical flaws and assume such things. For example the accountant at my work pointed out that a guy at my workplace walks like a girl. Nobody else noticed that now we all notice it, do we assume he is gay or trans? Nope, not that it matters tbh.