James Smith, a wealthy Baltimore merchant and shipowner, purchased a portion of the Coan Hall estate from a Mottrom heir and erected a brick mansion on a slope there, about 1785. He called his new home Mantua.
Mantua was built along the traditional lines of a Virginia house--a central hall, two rooms on each side and large end chimneys. From the outside it appeared to have three stories but there were really six floor levels. Later, identical wings were added on each side by Smith's sons. Still later, Smith's grandson added an imposing front portico. Before this addition coaches were driven, tradition says, right up to the front steps and ladies could step from coach to marble steps without soiling their dainty slippers. The conveyance could then continue around to the stables on the west side, or follow the driveway on to Coan Stage Road, which ran back of the plantation.
The marble steps, marble window-sills and paneled inside blinds were handsome details of Mantua which may have been the outcome of Smith's residence in Baltimore. Originally he was a native of County Derry, Ireland.
In the rear of the house there were five terraces, planted with flowers and, perhaps, vegetables and herbs. Brick slave quarters were ranged in a semi-circle beyond the terraces.
The second story front windows of Mantua overlooked both the Coan and the Potomac. Before government lighthouses and buoys marked the waterman's course in this section, he had only the stars, landmarks and a lighted window here and there to guide him. Mantua was a help to the watermen for they could always be sure of a lighted window there, a lamp purposely placed by members of the Smith family, and by day the towering poplar trees were familiar landmarks.
In 1791, Light-Horse Harry Lee became Governor of Virginia. While he was in Richmond he had the opportunity to visit the plantations along the James River.
When Harry rolled up before Shirley in all the trappings of a Virginia governor, it is not surprising that the young daughter of the house saw in him her "heart's desire."
But Charles Carter did not see Harry through his daughter's rosy vision. He saw him as a widower who was seventeen years older than Ann, and as a soldier who had been disillusioned by war and had not adjusted to peace.