John Doyle Lee was born on 12 September 1812 in Kaskaskia, Illinois Territory, about one hundred miles south of St. Louis. After the successful American Revolution, John Doyle, Lee's maternal grandfather, had traveled to Illinois from Virginia to claim 400 acres of frontier land allotted under the grant of 1787, earned as part of his military service. Lee's mother was Doyle's first daughter, Elizabeth. She married Ralph Lee, on 26 February 1811.
Ralph Lee was related to the Lees of Virginia and plied the carpenter's trade. When Elizabeth's health failed, Ralph began drinking to excess. In May 181 they executed a deed of trust to care for their children. Elizabeth died in November 1815. John D. spent the next four years in the care of a black nurse, in the home of John Doyle, who died in 1819. At that time John D. went to live with his uncle, James Conner, and his wife, Charlotte. They had six children, and John D. soon felt like an extra child where there were already too many.