Albert Thornton Grey was born in Aldie, Virginia on June 6, 1925 to Richard Edward Grey and Lucy Anna Grey. As a boy Al Grey played the baritone horn at the Goodwill Boys Band of Pottstown, Pennsylvania, led by his father. Al Grey joined the Navy in 1942 at the Great Lakes Naval Training Center. Previous to 1942, African Americans served in the Navy only as mess attendants and stewards. Due to a manpower shortage and discrimination during World War II, however, African Americans were recruited as musicians. Some professionals became part of the three resident bands at Great Lakes; others went through basic training and were dispatched to bases across the country in bands composed of twenty-five members. Al Grey was sent to Grosse Isle Naval Air Station in Grosse Point, Michigan. These bands gained fame as “Ambassadors of Goodwill,” in that they helped to break down racial barriers.

Al Grey worked with the orchestras of Benny Carter (1945-1946), Jimmie Lunceford (1946-1947), Lucky Millinder, Lionel Hampton (on and off between 1948 and 1953), and Dizzy Gillespie (1956-1957). Grey gained fame as a member of Count Basie's orchestra, with whom he worked on three separate occasions (1957-1961, 1964-1966, and 1971-1977). Grey led a band with Billy Mitchell in the early 1960s and another with Jimmy Forrest after leaving Basie's orchestra in 1977. In the later decades of his career, Grey became a regular attraction on the international festival circuit as part of All-Star ensembles that included musicians such as Buddy Tate, Harry Edison and Benny Carter. He also performed as a soloist and leader of his own bands, some of which included his son, Mike Grey, also a trombonist.

Al Grey was best known for his unique plunger mute style. To educate future generations about the plunger technique, Al Grey co-wrote with Mike Grey a manual on the subject entitled, Plunger Techniques: The Al Grey Plunger Method for Trombone and Trumpet (New York, c1987). Al Grey died in Phoenix, Arizona, on March 24, 2000.