Ray Brown was born in Pittsburgh, Pennsylvania, and had piano lessons from the age of eight. After noticing how many pianists attended his high school, he thought of taking up the trombone, but was unable to afford one. With a vacancy in the high school jazz orchestra, he took up the double bass.
A major early influence on Brown's bass playing was the bassist in the Duke Ellington band, Jimmy Blanton. As a young man Ray Brown became steadily more well known in the Pittsburgh jazz scene, with his first experiences playing in bands with the Jimmy Hinsley Sextet and the Snookum Russel band. After graduating from high school, hearing stories about the burgeoning jazz scene on 52nd Street, in New York City, he bought a one way ticket to New York.
Arriving in New York at the age of twenty, he met up with Hank Jones, with whom he had previously worked, and was introduced to Dizzy Gillespie, who was looking for a bass player