The Library is furnished in the style of the late Federal period (1800-1820) with most of the pieces attributed to the New York cabinetmaker Duncan Phyfe. It is less formal than the rooms of the State Floor and is often used for teas and meetings. The soft gray and rose tones of the paneling are complemented by a Tabriz carpet of the mid-19th century. The gilded wood chandelier with a painted red band was made about 1800 and belonged to the family of James Fenimore Cooper, author of The Last of the Mohicans and other classics.

On the west wall is a neoclassical mantel that came from a house in Salem, Massachusetts. It dates from the early 19th century and is decorated with grape-leaf swags and bellflower pendants.

On the mantel rests a pair of English silver-plate Argand lamps, a gift of the Marquis de Lafayette to Gen. Henry Knox, Secretary of War in Washington's Cabinet. Such lamps, named after their Swiss inventor, Aimé Argand, were a major innovation; George Washington ordered some in 1790, noting that by report they "consume their own smoke...give more light, and are cheaper than candles".

One of the bookshelves displays an unusual lighthouse clock made by Simon Willard to commemorate the visit of the Marquis de Lafayette to the United States in 1824-25. A likeness of Lafayette appears in a medallion on its base.

Portraits of Native Americans by Charles Bird King flank the east door, and a fifth hangs over the entrance to the corridor.

The Library was completely redecorated in 1962 as a "painted" room typical of the early 1800's and was refurbished again in 1976. The paneling, now a soft gray color, dates from the Truman renovation of 1948-52. Old timber removed when the mansion was stripped to a shell was made into paneling for various Ground Floor rooms.