Former Secret Service agent Clint Hill shares never-before-told stories and rare photos of the first lady in his new memoir, My Travels with Mrs. Kennedy

He’s a Secret Service agent who served under five presidents — Eisenhower, Kennedy, Johnson, Nixon and Ford — yet still to this day, one of the questions Clint Hill gets asked most often is: What was Jacqueline Kennedy really like?

As the first lady’s dedicated Secret Service agent, Hill rarely left her side for four years. They shared an “almost indescribable bond,” he says — a bond he explores in his new memoir, My Travels with Mrs. Kennedy, co-written with his wife, Lisa McCubbin Hill. It’s a sequel to their 2012 bestseller Mrs. Kennedy and Me, and his most personal book yet.

The idea for the latest book came about three years ago, while he and Lisa were cleaning out his home in Alexandria, Virginia, to prepare it for sale. There, they discovered an old dusty steamer trunk that hadn’t been opened for nearly 50 years.

Inside was a collection of photos, mementos, and handwritten notes from his travels to Paris, Ravello, Pakistan, India, Venezuela, Morocco, Mexico and more with the woman he still calls “Mrs. Kennedy.”

The discovery brought back long-ago memories — some magical, some very private, others very painful. A time capsule of his four years with the most famous woman in the world.

Writing the book, which includes nearly 200 rare photos, was “emotional” for the agent, who was just 28 years old and bore a resemblance to a young Paul Newman when he was first assigned to watch over the soon-to-be first lady, herself only 31 years old.

By her side, he saw her outside-of-state dinners and official visits. “Once she got away from the White House, she was more open and carefree,” he recalls.

And when he dropped her off for a Buckingham Palace lunch with Queen Elizabeth in 1962, he also saw her as a young mom who could share stories about raising young children while being on the world stage.

Then there was the 1963 trip to Morocco, at the invitation of King Hassan, where Jackie and her sister Lee Radziwill dined with the king’s brother. Afterward, a tray was passed around with tea and special confections, including “major,” which they tried, unaware the dessert contained cannabis.

“Major, as it turns out, was the Moroccan version of hash brownies,” writes Hill. Just one of the many never-before-told stories in the new book.

There were also casual days with her young children, Caroline and John, at their country home in Middleburg, Virginia, where he says, “She was just one of the locals. She wasn’t the star and that made her happy.”

Hill also accompanied President John F. Kennedy and the first lady to their fateful Dallas trip on Nov. 22, 1963.

When the first shots rang out, he ran and leaped onto their limousine, risking his life to save theirs. His courage is forever immortalized in the Zapruder film of JFK’s assassination. Nearly 60 years later, he says, “I still have a sense of not having fulfilled my responsibility.”

Hill hopes the new book “offers a better understanding of who Mrs. Kennedy really was,” he tells PEOPLE. “She was a one-of-a-kind unofficial ambassador and she made a global impact.”

He adds: “There was no one else like her.” Nor would there ever be a bond quite like theirs.

As Hill went through the old trunk, Lisa asked him, “Were you in love with her?” He answered: “I wouldn’t call it love. We had a bond, a bond that was really almost indescribable. But it never could have been love. She was the wife of the president and I was there to protect her. And we both knew that.”

Share on Facebook