And sure enough, reports have surfaced today that the Irish judge is facing the axe at the end of this series.
And according to The Sun the 59-year-old panelist is set to get the chop.
Louis was forced into an embarrassing apology last night after he branded Misha B a bully on Saturday night's show.
And a source told the tabloid: "It doesn't take a genius to work out that Simon not being on has hit the ratings.
"ITV are happy with the panel. They think they're great. But they realise when it comes to judges, there's isn't anyone like Simon. He IS the X Factor.
"The likelihood is that Louis will be gone next year - and Simon could make a welcome return."
Louis was left out of the panel briefly in 2007 - and replaced by choreographer Brian Friedman - before his position on the panel was reinstated.