You are ignoring one major factor - Events!
Nobody knows entirely what will happen in the next 5 years.
The Tories will have to make a success of Brexit. They won't be able to blame Starmer. He was never in favour of it.
Also they have to make a success of getting out of this current health crisis with a solution that fits their ideology better than a solution that the Labour Party might have suggested. For example, the whole notion of big government to the rescue is anaethema to many of them.