Yes, but deselecting remainers from tory seats will make it more likely that the tories win those seats at an election (although in fact the tory party has no mechanism for deselection, merely for insisting candidates be presented for reselection). A 'centrist' party represents very little threat to the Tories in terms of votes. For Labour, on the other hand, the rise of any centrist party would mean an almost certainly unsurvivable loss of vital votes in marginals.