George Clooney reflects on his humble beginnings with Hollywood icons

George Clooney is taking a trip down memory lane, all the way back to his early days in Hollywood.

The 64 year old actor caught up with PEOPLE at the Los Angeles premiere of his latest film Jay Kelly on October 23, where he shared fond memories of his late aunt, singer Rosemary Clooney.

“I remember being on the road [...] I was my Aunt Rosemary's driver when she was singing,” Clooney recalled, adding, “I got to drive [her] and Tony Bennett, and all of these really wonderful singers around for a year.”

The Ocean’s 11 star said that being around such talent gave him invaluable insight into fame.

“I got a really good life lesson in success and failure, and handling it. I felt very lucky that I got that lesson early in time,” he said.

Clooney even shared that he was once scolded by Frank Sinatra, something he laughs about now. “I got yelled at by Frank Sinatra!” he joked.

Reflecting on his enduring friendships, George Clooney added, “It’s helpful to have people that know you long before you were defined by something else — by movies you've done or work you've done.”