Ian McKellan may have played Lord of the Rings’ Gandalf and Magento in the X-Men series, but he was certain to stay away from the Harry Potter franchise.

According to the British actor, he turned down the role of Dumbledore after Richard Harris, who played the iconic wizard in Harry Potter and the Sorcerer’s Stone and Harry Potter and the Chamber of Secrets, passed away in 2002. For McKellan, he didn’t want to take on the role because he knew Harris didn’t like him very much.

“When they called me up and said would I be interested in being in the Harry Potter films, they didn’t say what part,” he shared during an interview with BBC’s Hardtalk. “I worked out what they were thinking and I couldn’t. I couldn’t take over the part from an actor who I had known didn’t approve of me.”

Before his death, Harris had publicly criticized McKellan for supposedly picking easy and “passionless” projects. “I’ve seen these so-called ‘nice’ actors. Very able fellows like Ian McKellen and Kenneth Branagh,” he said. “But they’re like bank managers. So sweet and careful. Who needs them?”

Ironically, Branagh starred alongside Harris in the second Harry Potter movie, playing the fame-hungry and egotistical Professor Gilderoy Lockhart.

Following Harris death, Michael Gambon was cast as the late actor’s replacement. He played the Hogwarts headmaster for the next six movies.

“Well, sometimes when I see the posters of Mike Gambon, the actor who gloriously plays Dumbledore,” McKellan mused, “I think sometimes it is me.”