There are stories that this house was built by Capt. John Swift who was a Revolutionary War Soldier who built this house as a fortress from Indian attacks but the Native Americans had left the area by the 1760s and the early period of cobblestone construction did not begin until 1825 so this does not make sense.

From Sodus Past and Present Page 29:

Raymer (Klint) Houses (also known as the Kolbe House):

Cobblestones in the front wall are of the round and oval lake-washed variety, of various sizes and colors; these are laid seven courses tp two quoin heights. The courses vary from three to four inches high from center to center of horizontal joints, but the stone heights in the different courses are maintained throughout their length. Under the quoins thin filler stones were inserted. Horizontal joints are wide flat "V's", and the vertical joints, from one-half to one inch wide, are finished with triangular pyramids.

The quoins are dark red sandstones with a crude attempt at tooling edging and with vertical scratches in the panels.

From Festival 1812 Tour of Homes (July 12, 1980)

The Kolbe House

The house is two stories high, and the walls are one and one-half feet thick. Two huge fireplaces warmed the two main rooms downstairs, and a flock of tiny "parlor bedrooms". Upstairs were a half dozen more bedrooms, also heated by fireplaces.

The house was in the Raymer family after 1875, for many years. It was not occupied during the 1950's and 1960's until it was purchased by Mr. and Mrs. Klint. They commissioned Mr. Glenn Young to extensively restore the house. The present owners have added a kitchen and two bedrooms in the carriage house, which is now part of the main structure. 

 As of 2019, this is the Kolby residence.