Farmington was built between 1815 and 1816 for John (1772-1840) and Lucy Fry (1788-1874) Speed.  Both of them came from wealthy Virginia families that moved to Kentucky in the last decades of the 1700s.  John Speed's father, Captain James Speed, fought in the Revolutionary War and was badly injured.  Like many others, he sought to make his fortune in land speculation in the newly opened territory west of the Appalachian mountains.  In 1782, he brought his young family and his slaves over the Wilderness Road and settled near Danville, Kentucky.

Lucy Gilmer Fry's family moved from the Charlottesville, Virginia area to Danville in 1798.  Lucy's father, Joshua Fry, was a highly respected scholar, who taught at Centre College.  Her maternal grandfather, the noted Kentucky explorer Dr. Thomas Walker, was one of Thomas Jefferson's guardians after Jefferson's father died.  Lucy's aunt and uncle, George and Martha Divers, lived in Charlottesville in a house also called Farmington that had an addition designed by Jefferson about 1802.     

     John Speed was married once before he married Lucy Fry.  He and his first wife, Abby Lemaster, lived at Mann's Lick (now Manslick in southern Jefferson County), and had four children, two of whom died in infancy.  Abby herself died shortly after the birth of her last child.  John took his two little girls, Mary and Eliza, back to Mercer County.  There he met young Lucy Gilmer Fry and in November 1808 the two married.  Probably the next year, John, in partnership with David Ward and William Pope, Jr., bought over 2000 acres of celebrated Jefferson County "Beargrass" land.  Speed's share of that large tract was the 554 acres that made up the original Farmington property.  An August, 1809 letter from John Speed to his friend and partner, William Pope, reports that "we are now living in our cabins," suggesting that by that date the family was living in log cabins on the Farmington property.  In addition to Mary and Eliza, the Speeds had eleven children whom they raised at Farmington.

     There is circumstantial evidence that the plans for Farmington were taken from a plan by Thomas Jefferson.  We know of Lucy's family's connections with Jefferson, and we know that Farmington's floor plan is very similar to one drawn by Jefferson.  The building contract for Farmington mentions Paul Skidmore as having done the plan for the house. Whether Skidmore was provided with a sketch from Jefferson or a verbal description from Lucy of what she envisioned or whether he independently arrived at the original design, we will never know.  In any event, construction, much of it undoubtedly by slaves, began in 1815 and was completed by 1816.

     The primary cash crop at Farmington was hemp which was used to make rope and rough bagging for the cotton trade.  References in the 1840 inventory and  settlement papers related to John Speed's will document that Farmington had  a "rope walk" and weaving house on the plantation where hemp was actually processed into rope and bagging.  Although Louisville's growing resources were, to a degree, available to the Speeds, Farmington was by choice and necessity largely self-sufficient.  The farm produced corn, wheat, apples, cider, vinegar, pork, flax, lamb and mutton, and dairy products.

In 1841, Farmington hosted it's most famous guest, Abraham Lincoln.  Tired and despondent over a break in his relations with Mary Todd of Lexington, and the direction his political career was taking, Lincoln came to Farmington to visit with his great friend Joshua Speed and his family.  Present information suggests that he stayed about three weeks with the Speed family during August and September.  After rest and relaxation, Lincoln returned to Springfield and to his wooing of Mary Todd.  Their subsequent marriage is history.

     Lincoln wrote a famous letter to Mary Speed, eldest of the Speed daughters, following his stay with the Speeds thanking her for the family's hospitality and recounting a disturbing encounter on board the return steamboat to St. Louis.  Here he witnessed the transport down-river of a group of newly sold slaves.  This narrative is thought to have been Lincoln's first known written reference to the horrors of slavery. 

     When Lincoln was elected President of the United States, he invited Joshua to join his Cabinet as Secretary of the Treasury.  Joshua, having no political ambition, declined but suggested his brother James Speed, a successful Louisville lawyer, who, in December 1864, became Lincoln's Attorney General.  James held that position until 1866.