Meanwhile, Harry's last wild speculations had ended in his complete financial ruin. Ann and the children were now living on a trust fund left to them by her father, Charles Carter, when he died in 1806.
One day, when Robert was not yet four years old, a carriage stood in front of Stratford, waiting to take the family for their last ride down the driveway. Stratford had been left to Matilda's son, Henry, and he had now come of age and was ready to take over the estate. Harry and his family traveled to Alexandria where they moved into a smaller house.
A legend says that when everything was ready for departure little Robert could not be found. He was finally discovered in the nursery saying good-bye to the two cherubs on the fireback.
After this Harry had still greater misfortunes. His body was broken and maimed for life. In 1813, when Robert was six years old, his father left Virginia, bound for the British West Indies, seeking health and a new grip on life. He spent the next five years wandering about among the islands. In 1818, he sailed for home but became so ill that he was put off at one of the islands. There he found the family of his old friend, General Greene. He was tenderly cared for by them during his final illness. He died there and was buried in their family burying ground on Cumberland Island, off the coast of Georgia.
NOTE: In 1913 the body of General Henry (Light-Horse Harry) Lee was brought from Cumberland Island and placed in the Chapel at Lexington, Virginia, beside that of his famous son, Robert E. Lee.
For many years the watermen of the Chesapeake "steered by the stars," by trees, and by a lighted window here and there.
One of the earlier government lighthouses on Chesapeake Bay was the Smith Point Light located at the mouth of the Potomac on Smith Point, Northumberland County.
There seem to be no available records concerning the erection of this lighthouse. In an 1804 issue of _Blunt's American Coast Pilot_ reference is made to a lighthouse having been "erected lately on Smith Point." This establishes the date of its erection as prior to 1804.
In the 1833 issue of the same book there is a small drawing of the lighthouse at Smith Point which shows a tower with a house close by. These structures appeared to be situated on the tip end of a point with a gently sloping hill, or bank, in the rear. The picture shows a lighthouse with the same general appearance as the first government lighthouse at Cape Henry at the entrance to the Chesapeake, built in 1791. The Smith Point tower, however, was round instead of octagonal.