It's probably Lincoln. The man was born in a log cabin and knew how to do an honest day's work unlike almost all of the other Presidents (including Teddy). His wrestling feats are the stuff of legends. And he was by far the tallest President at 6' 4", at a time when that was like a foot taller than the average.
Washington was also a very large and famously strong man for his time but he was getting on in years when he became President. Jackson's all bluster with a glass chin. Grant had health problems by the time he was elected and was never much of a brawler. JFK had a bad back and wouldn't have been able to run the length of a basketball court.
Truman should get a lot more attention. He (like Lincoln) grew up doing manual labor and tended his family farm into the early 1920s. He was an artillery man in WWI, and fairly young when elevated to the Presidency. He had no recorded fights that I know of but saw battle, I think he could go far.
The modern presidents would have an advantage due to modern medicine, better nutrition, and generally greater height. Ford, Obama, Clinton, Carter, and both Bushes would be contenders I think. Of those U think Ford, GWB, and Obama have the best shot at getting to the late rounds.
Lincoln wipes the floor with all of them in the end.