Matt Lauer Remains 'Withdrawn' 5 Years After Today Exit: 'Talking to People from His Past Is Painful'

Matt Lauer is continuing to tighten his inner circle five years after being ousted from the Today show.

A source tells PEOPLE exclusively that the former morning show anchor, 64, has "kind of withdrawn from several friendships in the last year," particularly following the release of his former co-host Katie Couric's memoir, Going There.

"He was really upset by Katie Couric's book," the insider reveals. "She shared their private text messages and she semi-slammed him. It made him lose trust."

The source continues, "His level of trust has just diminished with a lot of people he considered friends and much of that was because of Katie's book, and because talking to people from his past is painful."

RELATED: Katie Couric on Being 'Shocked' by Matt Lauer's Behavior: 'It Just Seemed So Callous'

Brent N. Clarke/FilmMagic Matt Lauer

The insider adds that Lauer is in "transitional time" with his two older children — Jack, 21, and Romy, 19 — who are now in college, and his youngest, 16-year-old Thrijs, who is in high school. The source notes he is a "semi-empty nester, not working."

A rep for Couric did not immediately return PEOPLE's request for comment.

For more on this story, listen below to our daily podcast PEOPLE Every Day.

During an appearance on Today in Oct. 2021, Couric, 65, said that when she heard about the allegations against Lauer, it was "really, really hard and it took me a long time to process what was going on."

RELATED VIDEO: Katie Couric on Being 'Shocked' by Matt Lauer's Behavior: 'It Just Seemed So Callous'

"The side of Matt I knew was the side of Matt I think you all knew. He was kind, generous, considerate, a good colleague," she said. "I got more information and learned more about what was going on behind the scenes. And then I did some of my own reporting, talked to people, tried to excavate what had been going on. It was really devastating and also disgusting."

"I think what I realized is that there was a side of Matt I never really knew. I tried to understand why he behaved the way he did, and why he was so reckless, and callous, and honestly abusive to other women," Couric continued.

James Devaney/WireImage Matt Lauer and Katie Couric

In a 2021 PEOPLE cover story, the television personality reiterated how she was "shocked" after learning of the allegations, and called Lauer's behavior "grossly inappropriate."

"That's not the Matt I knew," Couric said. "There's a duality in human beings, and sometimes they don't let you see both sides."

RELATED: Katie Couric Stands by New Memoir on Today: 'I Wanted to Share the Messy Parts' of My Life

Kevin Mazur/Getty Matt Lauer and Katie Couric

RELATED: Katie Couric Says She Has 'No Relationship' with Matt Lauer Today, Calls His Behavior 'Disgusting'

Lauer was fired by NBC in November 2017 after a former network employee, later identified as Brooke Nevils, reported that he sexually harassed her. Several other women came forward with similar accusations shortly thereafter.

Never miss a story — sign up for PEOPLE's free weekly newsletter to get the biggest news of the week delivered to your inbox every Friday.

Nevils went public to Ronan Farrow for his 2019 book Catch and Kill: Lies, Spies, and a Conspiracy to Protect Predators. To Farrow, Nevils alleged that Lauer raped her in his hotel room at the 2014 Sochi Olympics, where she was working for NBC's Meredith Vieira. Lauer has said all of his encounters were consensual.

If you or someone you know has been sexually assaulted, please contact the National Sexual Assault Hotline at 1-800-656-HOPE (4673) or go to rainn.org.