Yep.He doesn't have the mandate. The Conservative party do.
This is something many people forget. You vote for the person standing in your constituency. The party with the most seat wins, forms a government and the person who is the leader of that party is installed as prime minister.
At no point are you voting for a prime minister. It's just something you get given for being the leader of the governing party.