Since it was a development from the African slaves, many rich white folks considered it evil at least until after the Civil War.
When white men began performing in "blackface," they would use the banjo in their act. One player, Joel Sweeney, was often credited with adding the fifth string.
Sweeney was a multi-instrumentalist and traveling minstrel who helped to popularize the playing of the banjo in Engalnd and around the US, as he toured consistently. Although legend has it that he added the fifth string, many believe that isn't true, since a painting created before Sweeney's birth pictured a slave holding a five-string banjo.
He was, however, a champion of the proliferation of the banjo, and encouraged a drum manufacturer to start producing banjos for nation-wide sale.
By the end of the 19th century, white players were gathering for picking contests in urban areas. Metal banjo strings were invented sometime around the middle of the 1800s, and many banjo players would gather for hootenannies during the Civil War. The end of the 19th century also saw the evolution of banjo finger-picking.
From the slave fields to the minstrels, to the parlors, to picking parties, jam nights, and now concert halls and folk venues, the banjo is a staple in most forms of traditional music ... despite the jokes.