The son of Hall founders Allan and Sandra Jaffe, Ben was raised in and around Preservation Hall. Jaffe marched in his first Carnival parade at the age of nine alongside his father and parrain (godfather) Harold Duke Dejan. When he was thirteen, the lease on the building that housed the Hall was up for renewal. His father sat the family down and asked Ben whether they should continue. By 1984, many of the original musicians who had played at the Hall in its early days had passed away, yet the soul of the Hall remained the same. The young Jaffe replied, Of course you continue!
Less than ten years later, the day after graduating from Oberlin College, Jaffe flew off to play upright bass with the Preservation Hall Jazz Band in France and began overseeing operations at the Hall. He now serves as the Hall's creative director and plays tuba (sousaphone), upright bass, and banjo in the band. Jaffe remastered and released several archival recordings of the early Preservation Hall Jazz Band, including some by Sweet Emma Barrett and Sister Gertrude Morgan. The band's newer recordings include collaborations with the Blind Boys of Alabama, Pete Seeger, Tom Waits, Trombone Shorty, Mos Def, Lenny Kravitz, The Del McCoury Band and many others.