Conniff was born in Attleboro, Massachusetts, and learned to play the trombone from his father. He studied music arranging from a course book.
After serving in the U.S. Army in World War II (where he worked under Walter Schumann), he joined the Artie Shaw big band and wrote many arrangements for him. After his stint with Shaw, he was then hired by Mitch Miller, then head of A&R at Columbia Records, as their home arranger, working with several artists including Rosemary Clooney, Marty Robbins, Frankie Laine, Johnny Mathis, Guy Mitchell and Johnnie Ray. He wrote a top 10 arrangement for Don Cherry's "Band of Gold" in 1955, a single that sold more than a million copies. Among the hit singles he backed with his orchestra (and eventually with a male chorus) were "Yes Tonight Josephine" and "Just Walkin' in the Rain" by Johnnie Ray; "Chances Are" and "It's Not for Me to Say" by Johnny Mathis; "A White Sport Coat" and "The Hanging Tree" by Marty Robbins; "Moonlight Gambler" by Frankie Laine; "Up Above My Head," a duet by Frankie Laine and Johnnie Ray; and "Pet Me, Poppa" by Rosemary Clooney. He also backed up the albums Tony by Tony Bennett, Blue Swing by Eileen Rodgers, Swingin' for Two by Don Cherry, and half the tracks of The Big Beat by Johnnie Ray.
In these early years he also produced similar-sounding records for Columbia's Epic label under the name of Jay Raye (which stood for "Joseph Raymond") amongst them a backing album and singles with Somethin' Smith and the Redheads, an American male vocal group.
Between 1957 and 1968, Conniff had 28 albums in the American Top 40, the most famous one being Somewhere My Love (1966). He topped the album list in Britain in 1969 with His Orchestra, His Chorus, His Singers, His Sound, an album which was originally published to promote his European tour (Germany, Austria, Switzerland) in 1969. He also was the first American popular artist to record in Russia—in 1974 he recorded Ray Conniff in Moscow with the help of a local choir. His later albums like Exclusivamente Latino, Amor Amor, and Latinisimo made him very popular in Latin-American countries, even more so after performing in the Viña del Mar International Song Festival. In Brazil and Chile he was treated like a young pop superstar in the 1980s and 1990s when he was in his 70s and 80s. He even played live with his orchestra and eight-person chorus in large football stadiums as well as in Viña del Mar.