There is an article on the Alfrey family in the 1956 Centennial Edition of the Rowan Co. News8 . It reads, in part,

“Back in 1775 two Alfrey boys, John and Sam, came from the Potomac Valley, along with a group of Myers', Crouch's and Craig's. John and Sam settled on Licking River just this side of what is now Baldwin McKenzie's place. Sam built a log house in what is now the corner of Clell Jones' field.

Two of Sam's children were Fielding and Adam.”

Birthplaces listed in census records for John’s known children suggest that he and Sam really arrived in Kentucky about 1798 or 1799, and that, moreover, Sam was not John’s brother but his oldest son. Fielding and Adam may also have been sons of John, and thus Sam’s younger brothers. I am not aware of any documentary evidence either way, and family traditions vary depending on who you are talking to. Alfred, John and Minerva are almost certainly Samuel’s, though, since his widow Rachel is living with them at various times.

1850 census of Morgan Co., Ky, records on page 103, as family #232 (right before Alfred Alfrey at #233):
Rachael Alfrey, 59, b. Va; John, 25 (idiotic), b. Ky.