OK, let's ignore the canon and talk in terms of the brand. The (highly successful) brand is that he's a white, upper middle-class British man.
Also, he's called James, so him being a woman would be pretty weird.
The brand is that he's an English (Scottish?) gentleman with impeccable taste and a decent line in running shooting punching driving and shiz. And he's cool.
He could easily be a she. Or Afro-Caribbean, or Anglo-Indian. It makes no difference now.