"Obviously, whatever I do, it can't be compared to a gold medal winner in figure skating in the Olympics [or] a Pussycat Doll who… is so talented," he said, referring to Evan Lysacek and Nicole Scherzinger, the contestants he thought would be the two ultimately vying for the title.
The 80-year-old history-maker, the second man to set foot on the moon, added that "Dancing" pushed him to his limits.
"My whole life has been about reaching for excellence, but now it was about trying," he said. "[It was] probably right at the top, as far as being demanding."
Still, he didn't think he'd go so soon, even after the judges gave a weak score to his waltz with partner Ashly Costa.
"I really thought that, being two points behind the next lowest grade, that I had enough votes to overcome that," he said. "I really felt that I was in for at least another week."
Buzz also addressed another competitor, Kate Gosselin, who has hit the headlines once more with the announcement of her new show, "Twist of Kate," and her latest legal drama with ex-husband Jon Gosselin.
"I think the public eats that up," he said of Kate. "The show loves controversy. You attract a lot of people by having unusual individuals that are trying hard."
As for his "Dancing" moonwalk, he said you'd never see it happen up in space.
"The moonwalking has a name given to it by Michael Jackson," he said. "I've never seen anyone on the moon walk backwards. You can't see where you're going!"
Related Content from AccessHollywood.com:
VIEW THE PHOTOS: 'Dancing With The Stars' Season 10, Week 2!
VIEW THE PHOTOS: 'Dancing With The Stars' Season 10 Class Photos
PLAY IT NOW: Buzz Aldrin Gets The 'Dancing' Boot & More Drama For Kate Gosselin
PLAY IT NOW: 'Dancing' Recap: Goodbye, Buzz Aldrin