It is indeed. It's like how it's also racist to point out that someone clearly isn't a child or for that matter a refugee when the only excuse for letting them enter the country is that they are a child and/or a refugee. I'm sure Gary Lineker and Lily Allen could explain why doing this makes you a racist, but I'm not sure.
Essentially, I think it's that if someone is poorer and browner than you, then questioning their right to live in your country is racism.