Brown grew up in Bond Street in Kingston, living two doors away from Duke Reid's Treasure Isle studio. Heavily influenced by U-Roy, but also citing Big Youth, Dennis Alcapone, Prince Jazzbo and I-Roy as influences, Brown began his career on the Silver Bullet sound system in the early 1970s, moving on to Sound of Music and replacing U-Roy on the King Tubby's Hi-Fi system in 1975. Brown was initially given a chance to record by Winston Edwards and then Yabby You, and had a local hit in 1976 with "Starsky and Hutch", which was followed by a series of albums produced by Bunny Lee. He was signed to Virgin Records in the late 1970s, releasing two albums on their Front Line label. The Virgin contract enabled Brown to travel frequently to the United Kingdom, where he performed with the Unity Hi Power sound system. Brown returned to prominence in 1982, with "Tu Sheng Peng" (a version of Dennis Brown's "If This World Were Mine").
Brown set up his own Hit Sound label in 1977 as an outlet for his work as a producer.