Ray Brown was born in Pittsburgh, Pennsylvania, and had piano lessons from the age of eight. After noticing how many pianists attended his high school, he thought of taking up the trombone, but was unable to afford one. With a vacancy in the high school jazz orchestra, he took up the double bass.
A major early influence on Brown's bass playing was the bassist in the Duke Ellington band, Jimmy Blanton. As a young man Ray Brown became steadily more well known in the Pittsburgh jazz scene, with his first experiences playing in bands with the Jimmy Hinsley Sextet and the Snookum Russel band. After graduating from high school, hearing stories about the burgeoning jazz scene on 52nd Street, in New York City, he bought a one way ticket to New York.
Arriving in New York at the age of twenty, he met up with Hank Jones, with whom he had previously worked, and was introduced to Dizzy Gillespie, who was looking for a bass player. Gillespie hired Brown on the spot and he soon played with such established musicians as Art Tatum and Charlie Parker.
From 1946 to 1951 he played in Gillespie's band. Brown, along with the vibraphonist Milt Jackson, drummer Kenny Clarke, and the pianist John Lewis formed the rhythm section of the Gillespie band, and their work together eventually led to the creation of the Modern Jazz Quartet.
Around this time Brown was also appearing in Jazz at the Philharmonic concerts, organised by Norman Granz. It was at these concerts that he met the jazz singer Ella Fitzgerald, whom he married in 1947. Together they adopted a child born to Fitzgerald's half-sister Francis, whom they christened Ray Brown, Jr. Fitzgerald and Brown divorced in 1952.