The reason you could argue he doesn't belong is because he didn't play a team sport... And that in itself could make him the greatest athlete ever. All three of those other guys, as great as they are, still rely on an entire team to reach the highest level.
People like to write off boxers thinking it's easy... Like you just go into the to ring and throw punches... Maybe block a few. But there's a variable that's involved that the other three can't say they've had to deal with. When they have an off day their teammates are expected to help bail them out. You have an off day as a boxer, you're in for a world of hurt.
@LazaRRus