Tom Brady Wants to 'Be a Good Father' But Football Is 'Such a Big Draw' to Him: Source

Despite being incredibly focused on his football career, Tom Brady still "wants to be a good husband and father," a source tells PEOPLE.

"Tom is a great guy, everyone thinks so, including Gisele [Bündchen]. He doesn't mean to be so preoccupied," the insider says. "He wants to be a good husband and father; he still gets excited when talking about Gisele and the kids."

Brady, who revealed his decision to unretire from the NFL in March, and Bündchen, 42, are parents to son Benjamin, 12, daughter Vivian, 9 and Brady's son Jack, 15, from a previous relationship.

"Football is such a big draw to him," adds the source. "Something's got to give, and it has always been his family. They have to be so patient for him."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Gisele Bundchen Instagram

RELATED: Gisele Bündchen Talks Tom Brady's NFL Return as She Says She Stepped Back in Career to Raise Kids

The source also notes that Bündchen "feels like Tom needs to be home" as their kids are getting older.

"From a family standpoint, these are critical years. The kids are getting older, Ben is 12 now," the insider adds.

https://www.instagram.com/p/CS4izlXgY83/

The Brazilian model opened up about her feelings about Brady returning to the NFL in a cover interview for ELLE's October issue, out this week.

"Obviously, I have my concerns—this is a very violent sport, and I have my children and I would like him to be more present," she said of her husband. "I have definitely had those conversations with him over and over again. But ultimately, I feel that everybody has to make a decision that works for [them]. He needs to follow his joy, too."

Rumors of a rift between Brady and Bündchen increased last month after Brady was excused from practicing with the Buccaneers "to deal with some personal things," head coach Todd Bowles said at the time.

On Monday, multiple sources told PEOPLE that Bündchen is "frustrated" with Brady's decision to unretire.

"Going back to Tampa after the retirement put a strain on the personal side of his life," a source said. "Gisele is frustrated and sick of his career coming before their family, who has always supported him."

The couple spending time apart from one another "has happened before," the source explained, adding that "they are trying" to make the relationship work.