Travis Zajac scored the other goal while Kevin Weekes made 26 saves for the Devils, who have won three of their last four games.
"We found a way to win here," said New Jersey head coach Brent Sutter. "We didn't play all that great the first two periods. We played much better in the third; we played with some assertiveness, competed, played hard, and did a lot of good things in the third period that we need to do to be successful."
Manny Malhotra scored the lone goal and Steve Mason stopped 26 shots in the loss for the Blue Jackets, who had a three-game winning streak snapped.
"We played well, but we left the game out there," said Columbus head coach Ken Hitchcock. "They caught their break and they took advantage of it. Good teams do that. This is a tough loss for everybody, but we can move forward."
Columbus starts a three-game road trip in Vancouver on Sunday.
Trailing, 1-0, in the third period, the Devils scored twice in a span of 1:13 to take the lead.
The first goal came on the power play as R.J. Umberger attempted to clear the puck from the deep left boards, but it skirted off the end of his stick and went right to Zajac, who snapped a quick shot from the low left circle past a surprised Mason at the 10:06 mark.
A bit later, Madden attempted a wraparound from the left side that sat on the goal-line and he tucked it in from the right side before Mason was able to fall on it. The play was reviewed as the left post was coming off its moorings as the puck crossed the line, but a Columbus player was the one who dislodged the net -- and the goal was upheld.
Weekes made the lead stand up with a couple of big saves down the stretch as the Devils skated out of the building with a win.
The first period went scoreless as Weekes made eight saves in the frame while Mason countered with six.
Columbus took the 1-0 lead at the 12:46 mark of the second period. Kristian Huselius grabbed the puck along the right boards in neutral ice, skated into the New Jersey end and from the slot dished it to the low left side for Malhotra, who snapped a quick shot on net that fooled Weekes.
Mason stopped all nine shots he faced in the second period to keep New Jersey scoreless.