The son of Hall founders Allan and Sandra Jaffe, Ben was raised in and around Preservation Hall. Jaffe marched in his first Carnival parade at the age of nine alongside his father and parrain (godfather) Harold "Duke" Dejan. When he was thirteen, the lease on the building that housed the Hall was up for renewal. His father sat the family down and asked Ben whether they should continue. By 1984, many of the original musicians who had played at the Hall in its early days had passed away, yet the soul of the Hall remained the same. The young Jaffe replied, "Of course you continue!"
Less than ten years later, the day after graduating from Oberlin College, Jaffe flew off to play upright bass with the Preservation Hall Jazz Band in France and began overseeing operations at the Hall