Tommy Brown was born and raised in Sweetwater, Texas. Tommy Brown is the son of Tommy Lee Brown Sr., who worked at Sweetwater Cotton Oil Mill, and Mary Lee Nelson (Brown), who worked three decades as a Licensed Vocational Nurse (LVN). Mary Nelson remarried in 1973 to Jimmie Lee Nelson, who mentored Tommy to adulthood.