Graves was born in Brownsville, PA. His parents were Quakers, and as a young man, he followed in their footsteps, eventually shifting to the Hicksite branch of Quakerism. According to one source, Graves did not attend school for more than three or four months of his life, but another source claims that he had a "academical education" and began teaching at a Richmond school at the age of 19, a career he would continue for more than twenty years. He advocated for Abolitionism, was interested in language reform, and became linked with a number of radical freethinkers within Quakerism. In August 1844, he joined a group of roughly fifty utopian settlers from Wayne County, Indiana. Graves married Lydia Michiner, a Quaker, in July 1845 at Goschen Meeting House in Zanesfield, Logan County, Ohio, and the couple raised five children in Harveysburg, Ohio. They then returned to Richmond and purchased a farm. The Goschen Meeting House was a Congregational Friends center dedicated to Temperance and Peace, health reform, anti-slavery, women's rights, and socialistic utopianism. Graves' Quaker upbringing conditioned him to believe in the idea of the Inner Light, which held that all clergy, creeds, and prescribed liturgy in worship were irrelevant and impediments to God's mission. This was exacerbated by Hicks' version of Quakerism, Quietism, in which an individual's spiritual life was paramount and all outward manifestations were invalid.