In a speech to Justice Department employees marking Black History Month, Holder said the workplace is largely integrated but Americans still self-segregate on the weekends and in their private lives.
"Though this nation has proudly thought of itself as an ethnic melting pot, in things racial we have always been and I believe continue to be, in too many ways, essentially a nation of cowards," said Holder, nation's first black attorney general.
Race issues continue to be a topic of political discussion, Holder said, but "we, as average Americans, simply do not talk enough with each other about race."
He urged people of all races to use Black History Month as a chance for honest discussion of racial matters, including issues of health care, education, and economic disparities.
Race "is an issue we have never been at ease with and, given our nation's history, this is in some ways understandable," Holder said. "If we are to make progress in this area, we must feel comfortable enough with one another and tolerant enough of each other to have frank conversations about the racial matters that continue to divide us."
He told hundreds of Justice Department employees gathered for the event that they have a special responsibility to advance racial understanding.
Even when people mix at the workplace or afterwork social events, Holder argued, many Americans in their free time are still segregated inside what he called "race-protected cocoons."
"Saturdays and Sundays, America in the year 2009 does not in some ways differ significantly from the country that existed almost 50 years ago. This is truly sad," said Holder.
As a presidential candidate last year, Barack Obama gave a landmark speech on race relations during the hotly-contested Democratic primaries as he tried to separate himself from the angry rhetoric of his then-pastor. Holder cited that speech by Obama as part of the motivation for his words Wednesday, saying Americans need to overcome an ingrained inhibition against talking about race.
"If we're going to ever make progress, we're going to have to have the guts, we have to have the determination, to be honest with each other. It also means we have to be able to accept criticism where that is justified," Holder told reporters after the speech.
Holder is headed to Guantanamo Bay early next week to inspect the terrorist detention facility there. Obama has assigned Holder to lead a special task force aimed at closing the site within a year.
Holder's Justice Department will have to decide which suspects to bring to U.S. courts for trial, which to prosecute through the military justice system, and which to send back to their home countries.