Jacob Miller was born in Mandeville, Jamaica to Joan Ashman and Desmond Elliot. At the age of eight he moved to Kingston, Jamaica where he was raised by his maternal grandparents. His move to Kingston would prove to be a seminal event in his life, as Miller began spending much of his time at recording studios around the city, most notably Clement "Sir Coxsone" Dodd's now-famous Studio One.