The filmmaker, who has returned to direct a sequel 24 years after the original was released, shared the revelation ahead of question marks surrounding Phoenix's professional conduct.
Earlier this year, the Oscar-winning Joker star dropped out of a new Todd Haynes film with days to go, despite cast and crew being in position to shoot. The project was cancelled due to Phoenix's last-minute decision.
After that, James McAvoy claimed that he was only cast in M Night Shyamalan's 2016 thriller Split as Phoenix had quit the role weeks before the film was scheduled to begin production.
It turns out Phoenix was also on the cusp of quitting Gladiator also, with Scott saying the actor started getting cold feet while in costume on set.
"He was in his prince's outfit saying, 'I can't do it.' I said, 'What?'" the director told The New York Times.
Lead star Crowe, who played Maximus in the Best Picture-winning epic, was not happy with his co-star, with Scott revealing that he exclaimed: "This is terribly unprofessional."
Fortunately, Scott convinced Phoenix to stay on in the role of Commodus, which would go on to earn him an Oscar nomination for Best Supporting Actor.
He told the outlet: "I can act as a big brother or dad. But I'm quite a friend of Joaquin's. Gladiator was a baptism of fire for both of us in the beginning."
Scott worked with Phoenix again on the 2023 film Napoleon, which Phoenix is also rumoured to have threatened to quit - but Scott remained coy on that suggestion in the new interview
When asked to comment on his decision to quit Haynes's project, Phoenix told Screen International at Venice Film Festival in August: "If I do, I would just be sharing my opinion from my perspective and the other creatives aren't here to say their piece and it just doesn't feel like that would be right, I'm not sure how that would be helpful."
Meanwhile, Haynes' long-time producer Christine Vachon called the situation "tragic", stating: "The idea that his time was wasted and that a movie is not a result of those years of working closely with Joaquin... That is the tragedy to me. And that I can't get over, that we as a cultural community, lost the opportunity to have another movie by Todd Haynes. That is just criminal."