Richards started playing the xylophone at age six. In high school he performed with the Hartford Symphony Orchestra. He studied with Al Lepak at the Hartt School of Music in Hartford, graduating in 1952. After being drafted, he belonged to an Army band in Japan and played with Toshiko Akiyoshi. He cited Lionel Hampton as his first and biggest influence on vibraphone.
In 1954 Richards moved to New York City, where he played with Charles Mingus, Ed Shaughnessy, and Ed Thigpen while doing studio recordings for Perry Como, the Ray Charles Singers, and Mitchell Ayres. For about three years he was a member of a group led by George Shearing, then moved to Los Angeles and worked with Don Ellis and Paul Horn