Boy George Dishes on Madonna: 'She's Too Full of Herself to Mention Me'

"Every gay man loves Madonna and there’s no one gayer than me," writes Boy George in his new book, Karma: My Autobiography.

He adds, "Her fans always try to defend her when I make a joke or a jibe. They are mostly missing the point."

The singer doesn't hold back when he details the many times he says he's been snubbed by the star (not to mention a cease-and-desist when Culture Club did a version of Madonna's enduring hit "Vogue"), but he's also clear-eyed about his own attitude, the way fame can affect a person and how a star's public persona is distinct from their private life.

"I love her music and I love Madonna and I love all the people I've written about," he tells PEOPLE. "I suppose when you write things about other artists, you remember that perhaps there's been times in your life when you weren't friendly to everyone you met."

The two singers have had their issues in the past

In the book, George details the first time he met Madonna as a case of mistaken identity.

"This is where Madonna alleges she met me, and I was bitchy," he writes. "She describes me as head to toe in Westwood." But he notes that, at the time, he was wearing Sue Clowes exclusively. To George's recollection, that couldn't have been him.

Another time, the "Karma Chameleon" singer claims Madonna snubbed him at the opening of Palladium, a then-new club in New York by Steve Rubell, who also founded Studio 54. "Madonna arrived with Sean Penn and pretended she didn’t see me," he writes.

The pop icon has also reportedly snubbed George's friends, which didn't raise her profile in his mind. Upon meeting her and extending his hand, Madonna once told George's friend, "'Ugh, I don’t know where it’s been.' He replied, 'By my side all night.' I told him he should have poked it in her eye," George writes in his memoir.

But George doesn't hold back his appreciation of the "Like a Prayer" singer's music, despite their rocky past.

"Madonna is everything it says on the tin but she adds new ingredients every day," he writes. "I know for a fact she’s too full of herself to even mention me. She once said, ‘Boy George was mean to me in the '80s and he’s still mean.’ To be fair, I haven’t really been given the opportunity."

He has been vocal about loving some of her records, and jokes that, "Not knowing anything about her I would assume she’d have issue with the word ‘some.' ... I have a sense of humour, I don’t think Madonna does, despite some of the clothes she wears (LOL)."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

George is trying to be nicer—to himself and others

When asked about their interactions, George chalked a lot of it up to the difference between a star's public persona and how they act in private. Over the years, he said he's learned to be nice not only to his fans, but to himself.

"Not only is it easier to be nice as it's better for you," he says. "I don't have those situations with the public anymore because I'm just always friendly. And nicer to myself. You are part of the process."

At the end of the day, he says it comes down to the person behind the brand. “I love all the drama of, you know, the pop star behavior. I love to watch Madonna. I mean, the Madonna brand is genius," he writes. "But outside of that, you know, it's how you act. You know, because you want, as I say in the book, you want your artists to be weirdly beautiful and strange and all that, but you also want them to be kind of ordinary."

And he's not giving up on Madonna, either. "Like Bette and Joan, we should have been friends," he says. "There’s still time."

For more from Boy George, pick up the latest issue of PEOPLE, on newsstands everywhere now.