Morgan County marriage records list marriage on 1 Oct 1860 of A. J. Brown, age 20, born Morgan Co., to Malvina Adkins, age 19, born Morgan Co., by R. Humphrey, witnesses James Fultz and Phillip Fraley. (Judging by census records, though, Malvina was really only about 15 or 16.)

1870 census of Elliott Co., Ky, precinct #5 (Mocassin) lists as family #58:
Brown, Andrew, 29; Malissa, 26; Melvina, 8; William, 7; Henry, 5; Isaac 7/12 born Dec.
All born in Kentucky. Neighbors were John Brown age 33 with wife Anna, and William Brown age 27, single, possibly brothers.

1880 census of Elliott Co., Ky, precinct #2 (Devil’s Fork) lists as family #41:
Brown, Andrew J., 40; Melvina, 35; Jasper, 17; Henry, 14; Isaac, 10; Frances, 6; Eveline, 4; William, 2. All born in Kentucky.
Jasper age 17 must be the same as William age 7 in 1870.

1890 tax list of Elliott Co., Ky, lists Andrew J. Brown in Devils Fork, precinct 2, neighbor G.W. Stamper, 1 adult male, 1 voter, 1 militia member, no children 6-20. (Those are the only categories. Women and children under 6 were not counted.)

Since in 1890 Mary Evaline would have been 15, William 11, and Lizzie about 6, this suggests that the children may have been living in a different household after Melvina died.

I have not found Andrew in the 1900 census.