A saxophonist and percussionist whose bands played a broad range of R&B and dance music, Castor was nicknamed "the Everything Man." The Jimmy Castor Bunch's hits included "Troglodyte (Cave Man)" (which was sampled by N.W.A.), "It's Just Begun" (featured in the breakdance battle scene in the 1983 movie Flashdance) and novelties such as "The Bertha Butt Boogie." Ice Cube and Eric B. and Rakim are among the many hip-hop acts to sample Castor's music over the years; Kanye West sampled one of his records on The College Dropout.
Late in his career, Castor toured with a version of the doo-wop group the Teenagers, whose late singer, Frankie Lymon, had been a boyhood schoolmate of Castor's in New York.