The meeting is set for Thursday night and follows a flurry of calls overnight in which many of the leaders reconsidered their support for Paterson. Those calls were also voiced in a similar summit Saturday in Manhattan's Harlem neighbourhood.
The adviser spoke on the condition of anonymity because of the sensitivity of the issue. The adviser says Sharpton is expected to say he's rethinking his support for New York's first black governor.
Sharpton spokeswoman Rachel Noerdlinger confirmed the meeting but declined to comment further on Paterson. The leaders will also discuss issues including health care, education and jobs, she said.
Paterson represented Harlem for 20 years in the state Senate before becoming lieutenant governor in 2006, then governor in 2008, when Eliot Spitzer stepped down during a prostitution scandal.
The news about the Manhattan meeting contrasted with a statement Thursday by an organization of black police officers who stepped up to show support for Paterson. The group, 100 Blacks in Law Enforcement Who Care, urged an end to what it called a "rush to judgment."
Paterson is snared in two scandals that threaten his job - he is being investigated over contact with a woman who had accused a top aide of domestic violence and he faces ethics charges over taking free baseball tickets.
Paterson was accused Wednesday of breaking ethics laws when he sought and obtained free Yankees tickets for the 2009 World Series and then may have lied about his intention to pay for them, according to a state report.
He faces penalties of nearly $100,000, and the case was referred to the Albany County prosecutor's office and the state attorney general for possible criminal investigation into whether Paterson or anyone else gave false answers to questions by the Public Integrity Commission or backdated a check to pay for the tickets.
The damage was mounting in the wake of the scandal plaguing Paterson over contact he and others in his administration had with a woman who accused a top Paterson aide of domestic violence.
The ethics charge isn't directly related to the scandal over the aide. But the panel said the aide, David Johnson, was one of Paterson's four guests, along with Paterson's son and a son's friend, getting tickets for the Oct. 28 World Series game provided by the Yankees.
Four days later, also in the Bronx, Johnson was accused of domestic violence by his then-girlfriend, a case that now threatens Paterson's job and administration.
At issue is whether Paterson or anyone from his staff or security detail influenced the woman's decision not to press charges after she told police that Johnson roughed her up.
"My side of the story will not be unsourced, it will not be in inaccurate, it will be the truth," Paterson said Wednesday, taking a swipe at some media reports.
But the ticket scandal may ultimately be more damaging to the governor, especially given the timing.
Paterson told investigators that he always intended to pay the $850 for tickets for his son and the son's friend. They were paid for with a postdated check, and the governor paid for them only when confronted by a reporter for The New York Post, the report said.