Gilmore grew up in Chicago and played clarinet from the age of 14.  He took up the tenor saxophone while serving in the United States Air Force from 1948-1952, then pursued a musical career, playing briefly with pianist Earl Hines before encountering Sun Ra in 1953.
For the next four decades, Gilmore recorded and performed almost exclusively with Sun Ra. This was puzzling to some, who noted Gilmore's talent, and thought he could be a major star like John Coltrane or Sonny Rollins. Coltrane, in fact, was impressed with Gilmore's playing, and took informal lessons from him in the late 1950's. Coltrane's epochal, proto-free jazz "Chasin' the Trane" was inspired partly by Gilmore's sound.
In 1957 he co-led with Clifford Jordan a Blue Note date that is regarded as a hard bop classic: Blowing In from Chicago. Horace Silver, Curly Russell, and Art Blakey provided the rhythm section. In the mid-1960s Gilmore toured with the Jazz Messengers and he participated in recording sessions with Paul Bley, Andrew Hill (Andrew! and Compulsion), Pete La Roca (Turkish Women at the Bath), McCoy Tyner (Today and Tomorrow) and a handful of others. In 1970 he co-led a recording with Jamaican trumpeter Dizzy Reece. His main focus throughout, however, remained with the Sun Ra Arkestra.
Gilmore's devotion to Sun Ra was due, in part, to the latter's use of harmony, which Gilmore considered both unique and a logical extension of bebop. Gilmore had stated that Sun Ra was "more stretched out than Monk"  and that "I'm not gonna run across anybody who's moving as fast as Sun Ra ... So I just stay where I am." 
Gilmore himself made a huge contribution to Sun Ra's recordings and was the Arkestra's leading sideman, being given solos on almost every track on which he appeared. In the Rough Guide to Jazz critic Brian Priestley says:
Gilmore is known for two rather different styles of tenor playing. On performances of a straight ahead post-bop character (which include many of those with Sun Ra), he runs the changes with a fluency and tone halfway between Johnny Griffin and Wardell Gray, and with a rhythmic and motivic approach which he claims influenced Coltrane. On more abstract material, he is capable of long passages based exclusively on high-register squeals. Especially when heard live, Gilmore was one of the few musicians who carried sufficient conviction to encompass both approaches."
Many fans of jazz saxophone consider him to be among the greatest ever, his fame shrouded in the relative anonymity of being a member of Sun Ra's Arkestra. His "straight ahead post-bop" talents are exemplified in his solo on the Arkestra's rendition of "Blue Lou," as seen on Mystery, Mr. Ra.