As you well know, the charge is that they colonised and subjugated England (as it is now called), expelling the Celts into Cornwall and Wales.
The historical evidence for a sort of mass Romano-Celtic exodus to the west is pretty sketchy, actually. The suggestion is that most Celts stayed where they were, inter-married and were subsumed within Anglo-Saxon society.