Breckinridge, however, after continuing to menace Baton Rouge for some days, had, by Van Dorn's orders, retired to Port Hudson, and was now engaged in fortifying that position. Ruggles was sent there on the 12th of August. The next day Breckinridge received orders from Van Dorn, then at Jackson, to follow with his whole force. "Port Hudson," Van Dorn said, "must be held if possible." "Port Hudson," remarks Breckinridge, in his report of the battle of Baton Rouge, "is one of the strongest points on the Mississippi, which Baton Rouge is not, and batteries there will command the river more completely than at Vicksburg."
Meanwhile Butler had changed his mind with regard to the evacuation of Baton Rouge, and had directed Paine to hold the place for the present. With an accuracy unusual at this period, Butler estimated Breckinridge's entire force at 5,000 men and fourteen guns. On the 13th the defences were complete, the entrenchments forming two sides of a triangle of which the river was the base and the cemetery mound the apex. The troops stood to arms at three o'clock every morning; one fourth of the force was constantly under arms, day and night, at its station. At two points on each face of the entrenchment flags were planted by day and lights by night, to indicate to the gunboats their line of fire.
On the 16th of August Butler renewed his orders to burn and evacuate Baton Rouge. Its retention up to this time he had avowedly regarded as having political rather than military importance. Now he wrote to Paine: "I am constrained to come to the conclusion that it is necessary to evacuate Baton Rouge. . . . Begin the movement quietly and rapidly; get every thing off except your men, and then see to it that the town is destroyed. After mature deliberation I deem this a military necessity of the highest order."
Against these orders Paine made an earnest appeal, based upon considerations partly humane, partly military. He was so far successful that Butler was induced to countermand the order to burn. The movement was not to be delayed on account of the statue of Washington. However, the statue had been already packed. It is now in the Patent Office at the national capital. All the books and paintings were brought off, "except," to quote from Paine's diary, "the portrait of James Buchanan, which we left hanging in the State House for his friends." Finally, on the 20th, Paine evacuated Baton Rouge, and on the following day reached the lines of Carrollton, known as Camp Parapet, and turned over his command to Phelps.