Barretto was born on 29 April 1929, in the Brooklyn borough of New York City, of Puerto Rican parents. Noted for his many years as a prominent Latin bandleader, his music career actually began as a studio performer on the conga for jazz recording sessions.
He was raised in the Latin ghettos of East Harlem and the Bronx, in an environment filled with music of Puerto Rico but with a love for the swing bands of Ellington, Basie and Goodman. He escaped the ghetto by joining the United States Army when he was 17 years old, but he did not escape the music.
Influenced by a record of Dizzy Gillespie, “Manteca”, with conguero Chano Pozo