Except of course that the "other policies" that the Tories are campaigning on include massive increases in public spending (popular with traditional Labour voters) while Labour are pledging to not only reverse Brexit but to keep freedom of movement (massively unpopular with traditional Labour voters).
So your assumption that former Labour and now Brexit Party voters would sooner return to Labour than vote Tory is not necessarily correct. For many of these people, the Tories now better represent them than the metropolitan liberal elite-dominated Labour.