male and female: black and white, or what?

When are you male/female?
This is a simple question, especially at fist thought, but actually so much more complicated. Is it what you look like? what you feel like or whether your cells contain XX or XY?
if your answer is what you look like, or the genetics, what happens to those unfortunate people "born in the wrong body?" Are these transvestites what they look or what they feel like?

Another scenario: you are female, have always been female, have been brought up as a female, and have female secondary sexual characteristics. But when going to the doctor to get advice on how to get pregnant, you dont only find out that you are infertile, but that your genotype is XY. are you male or female?

And what happens to the very few unfortunate babies born with XY but a partially functional testosterone receptor: half the testosterone works, the other half is changed to oestrogen due to the high androgen concentration. you get a mixture of both male and female genitalia. So what are these babies? male? female? and who decides?

and what about the hoard of other issues: sexual orientation, marriage, perspective of yourself and stigma of society.
finally if you are a doctor and your patient has this XY despite looking XX issue: do you tell them and risk deep psychological trauma, or do you not tell them, even though you know?