Paterson allegedly implored Kennedy not to suddenly withdraw from contention, New York magazine reported.
"They had never heard her talk so tough," said an unnamed family adviser quoted in an excerpt of the book "Ted Kennedy: The Dream That Never Died." "They told her that, if she was getting this worked up getting the job, they didn't want to see what she would be like in the trenches of a political campaign."
Kennedy's daughters Rose and Tatiana and her 15-year-old son John were concerned that their mother's political career was bringing stress into the home and changing her behavior.
“Rose pleaded, saying, ‘Mom, you are above this." according to the advisor.
Author Edward Klein wrote that the meeting was a wake up call to Kennedy, who called Gov. David Paterson hours later to withdraw.
"If Paterson had called and offered her the job an hour earlier, she would have accepted," the adviser said in the book except. "But after that conversation [with her kids], she wouldn't have taken the job if Paterson had come begging."
.Kennedy was considered the favorite to fill Clinton's vacated Senate seat, which was ultimately given to unheralded upstate Congresswoman Kirsten Gillibrand.
Kennedy had expressed interest in the Senate seat in December after being urged to throw her hat in the ring by her uncle Ted. Her decision to withdraw was a big disappointment to Sen. Ted Kennedy, who had hoped his niece would follow the family's legacy in national politics