Wesley grew up playing drums, bass and piano. Then his father, a band leader, promised him that if he could play trombone he could join his band.
Wesley was just 12 years old and he's never looked back.
He started off with Ike and Tina Turner in the 60s. And when funk legend James Brown needed a trombonist, Wesley got his big break.
"James Brown already had a funky saxophone player, Maceo [Parker]. But at that time there was only one funky trombone player and that was me," Wesley told RFI ahead of a recent concert at the Blues sur Seine festival in Mantes-la-Jolie north of Paris.
The first thing Brown said was "Can that guy dance?". Luckily Wesley passed the test.
He joined Brown's band The JBs and became Brown's musical director.
He insists Brown was the creative one, and that he just "filled in the gaps".
But Wesley played on some of the big hits - "Say it Loud, I'm Black and I'm Proud" was his first - and even wrote some of the songs, such as Breakin' Bread and Sportin' Life.
They were tumultuous years - the men parted company in 1975. But Wesley still respects the way Brown changed music.
"James Brown knew things that we didn't, that's why I revere him 'til today," he says.
Wesley moved over into jazz, joining Count Basie's orchestra in 1978.
These were happier times. He says the album To Someone (1988) marked the height of his career as a trombonist, when "my chops were at their best".
He moved over from sideman to frontman full-time, forming his own band Fred Wesley and the new JBs in the late 90s. They do some of the old JBs stuff like Pass the Peas, alongside new material like Peace Power.
He no longer has the energy to keep doing the long solos of the past, let alone to dance like he did in the Brown years, he says, but he enjoys passing on his knowledge to the younger generation.
"My playing days are vastly coming to an end," he says. "What I can do now is pass on my knowledge and skill and the kids benefit".
Ahead of the Blues sur Seine concert, Wesley gave a masterclass.
He blows a bluesy riff for students at the local conservatoire to improvise on.
"Blues makes you feel better about feeling bad," he tells us.
And we all feel a lot better hearing Wesley play.