The only logic for it not being a penalty was that Saka never touched or controlled the ball. Which, frankly, is a nonsense according to the rules but I agree you tend not to get those.
And the 'he went down to easily' muppets haven't seen it clearly. James puts his leg across Saka which causes Saka to trip and then James pins his leg causing him to go down. It was a foul every time anywhere else on the pitch.