I have not found this Harrison Gardner/Garner in 1850 or 1860, doing a statewide search. (There was a Harrison B. Garner age 32 born in Ky listed in the 1860 census of Hendricks Co., Indiana, with a wife Frances and several children including a daughter Sarah C. age 1, but he was still there in 1870, and all of his chldren were born in Indiana.) I suspect he may have gone by a different name when younger. E.g., there was a William Gardner age 22 with mother Catherine in the 1850 census of Fleming Co.

1870 census of Fleming Co., Ky, Hillsboro precinct, enumerated 1 Aug, lists on p. 7 as family #46/45:
Garner, Harrison, 46, farm laborer; Nancy, 40; Sarah, 9; Story, Newton, 24. All born in Ky.

1880 census of Nicholas Co., Upper Blue Lick precinct, enumerated 3 June, lists on p. 5 as family #41/42:
Gardner, Harrison, 51, farmer; Nancy, 40, wife. Both born Ky Ky Ky.

1900 census of Fleming Co., Ky, district 2, enumerated 12 June, lists on p. 6A as family #107:
Gardener, Harrison, 70, b. June 1829, widowed, farmer; Petitt, Sallie M., granddaughter, 18, b. Mar 1882, single. Both born Ky Ky Ky.