Those permissions are not granted on a case by case basis. They were granted in perpetuity by treaty at Maastricht (and subsequently Lisbon) and were never subjected to public approval or disapproval. Blair promised a vote on Lisbon, of course, but then withdrew that commitment as he knew he'd lose badly.
You surely cannot be arguing that a decision taken unilaterally by governments that left office years before many voters were even born and which is effectively immutable (other than by leaving the EU, of course) can remain democratically legitimate in perpetuity? A key principle of the British constitution is that no Parliament may bind its successors in perpetuity and yet that is - in effect - precisely what signing these treaties did. As such, those 'permissions' are not democratically legitimate. The price paid to remain part of the EU was to abandon our system of democratic and parliamentary scrutiny over vast swathes of our statute book. To those of us who believe in democracy, it was not and is not a price worth paying.
Dismantling those treaties - because they subvert our domestic legislative process and allow laws to be passed pretty much in perpetuity without proper democratic scrutiny - is very much where the impetus for Brexit came from.