And it seems that the two really weren't helping matters after they spent the evening together on Thursday.
According to The Mirror, the two, who met on the show, spent most of Thursday evening flirting with each other.
A source said: "They had a great evening and really enjoyed each other's company, there could definitely be a future between them."
And according to the paper, a romance has been on the cards for several weeks with Caroline advising James not to start anything up until the X Factor had finished.
[Related article: Caroline Flack gets over Harry Styles split with pals NOT Joey Essex]
[Related article: James Arthur wins the X Factor 2012]
The source continued: "James built up a reputation as a ladies' man over the series and made no secret of the fact he was enjoying all the female attention he was getting.
"But while he has been playing the field he has always had feelings for Caroline. She made it clear nothing could happen while the show was still on, but as the weeks went by they got closer and closer.
"At the wrap party on Tuesday — which was held at the Hippodrome Casino in London — they spent almost the entire evening together and there was a lot of flirting. They left separately but arranged to meet again."
However, according to the Daily Mail, a representative of Caroline Flack has denied the romance.
They said: "It's definitely not true and it won't be changing anytime soon."
Didn't Caroline initially deny her romance with former X Factor contestant and One Direction member Harry Styles way back when?
And look how that turned out.