Costars Jennifer Aniston and Gerard Butler are making headlines with speculation over their off-screen romance – but a usually humorous Butler sets the record straight on one thing he doesn’t find funny.

“That is just annoying. People say I’m always dating so and so, and sometimes it’s three people in one day,” the actor, 39, tells PEOPLE.

“I’m trying to make movies, work hard and do my best. Then this kind of thing happens and people start to associate me more with that. It gets ridiculous.”

Aside from those pesky rumors, Butler – who also stars with Katherine Heigl in The Ugly Truth, opening July 24 – has nothing to complain about when it comes to his current gig.

“She is one classy lady,” he says of Aniston. “Everyday I go to work with her I’m always surprised about how cool, easy and down to earth and real she is.”

He adds, “I’m in New York right now filming and I’m in heaven. I can’t always say that about movies, but to be filming this romantic comedy with Jennifer and a story that made my side split when I read it – I’m lucky. I’m happy as a pig in s—, as they say.”

As far as Aniston’s current love life goes – a source close to the actress, 40, says she’s still happily talking to Bradley Cooper. But when it comes to working with Butler, “They get along really well, but Jen only likes him as a friend,” says a source. “She can’t see herself dating him.”