Casey Affleck reflects on his connections with longtime friends like Matt Damon.

The 49-year-old actor spoke with PEOPLE during the Center at West Park‘s 2nd Annual Fall Benefit in New York on Thursday, November 14, where he was asked about the importance of his friendship with Damonwho was honorary president at the evening’s event.

“Someone said that old friends are the leaven of life,” Casey told PEOPLE. “The older I get, the more it means to me.”

“People you’ve known for a long time become more and more important as you get older, and they are to me, for sure,” he continues.

Never miss a story: subscribe to PEOPLE’s free daily newsletter to stay up to date with the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

As for his presence at this benefit evening which celebrates the arts, the Manchester by the sea The star says in part: “(Art) was something I loved very early on, and I think it does a lot for society.”

“The arts entertain us and sustain culture, and the arts are the mirror we look into when we want to understand who we are and what we do,” adds Casey.

Casey and Damon, 54, have been friends for over four decades, as has Casey’s older brother. Ben Affleck. The trio has collaborated on numerous occasions, including on the 1997 film Goodwill Hunt and, more recently, this year The instigators.

“We have this shared history of 43 years,” Damon told PEOPLE in August of his longtime bond with the Affleck brothers. “We grew up together.”

The PEOPLE Puzzler crosswords are here! How quickly can you solve it? Play now!

Boston natives “shared experience and relationship, trust and love” as Damon said, this is what makes collaboration such a natural process.

“There’s an underlying understanding of who we are,” he told PEOPLE, adding: “In our business, there’s a whole language that’s created to try to protect people’s feelings because their ego is involved.”

He also said that he and Casey “don’t waste time on diplomacy, which I love.”

Damon’s memories of chasing his Hollywood dreams alongside Casey and Ben, 52, in the years before Oscar-winning success Goodwill Hunt involved not being “afraid to be vulnerable with each other as young men” in Boston.

As he reminded PEOPLE, “being an actor, putting yourself in these positions and constantly being rejected – knowing that someone else is living the same life and making the same sacrifices, that sense of shared experience, it That’s why we would band together.” and spend time together and talk about it.