Well that’s no good now is it?
The music mogul, 53, was asked by audience members at yesterday’s Britain’s Got Talent auditions in Cardiff if he would be rejoining the show.
A source told The Sun: "Simon shook his head. People weren't happy."
And we don’t blame them.
[Related article: Simon Cowell returning to the UK X Factor to boost ratings?]
Simon Cowell left the series in 2011 to concentrate on the US version instead with ratings for the UK show subsequently declining ever since.
Last year’s X Factor audience figures proved to be at an all time low- even with wacky Nicole Scherzinger proclaiming newly coined words such as ‘JaJazzle’ and ‘Schermazing’ in every other one of her comments.
The current lineup- Gary Barlow, Tulisa, Nicole Scherzinger and Louis Walsh- were heavily criticised this year for not having enough chemistry.
After the show wrapped the usual names to replace members of the panel were banded about- namely Sharon Osbourne and Cheryl Cole.
But without Simon Cowell, will ratings decline even more?
Executive producer of the show Richard Holloway previously admitted that he was desperate to get Simon back behind the UK judging desk.
He said: "There is not one element we will not be looking at carefully - from judges and start times to auditions and glitzy productions.
"I'd love him to come back. He's the best judge in the world, never say never! If we can find a way to reintroduce him, we will. But it is impossible for him to be in two places."
Alas, it looks like Simon is staying put in the US.