1850 census of Morgan Co., Ky, lists as family #271:
Coleman Brown, 28; Joanna, 20; James, 6/12. All born in Kentucky.

1860 census of Morgan Co., Ky, district 1, enumerated 8 June, lists on pp. 10-11 as family #61/59:
Coleman Brown, 42, farmer; Joanah, 29; James W., 10; Sesa E., 8; Mary G., 6; Rebecca A., 4; Diadama, 2; Louisa, 3/12; James Lewis, 24, laborer. All born in Kentucky.

The preceding household is headed by Thomas J. Ferrand, who is married to Coleman’s sister Martha.

1870 census of Morgan Co., Ky, Blair’s Mill precinct, enumerated 21 July, lists on p. 2 as family #10:
Brown, Colemon, 52, farmer; Joanna, 39; James W., 20; Rebecca, 14; Dinah, 13; Lizzie, 11; Josephine, 9; Jesse C., 7; Martha, 5; John W., 7/12. All born in Kentucky.

Lizzie is apparently the daughter recorded as Louisa age 3 months in 1860.

1880 census of Morgan Co., Ky, district 87, p. 17 (handwritten), p. 538A (stamped), lists as household #147:
Brown, Coleman, 63, farmer; Louiza, 49, wife; Damie, 22, dau; Josie, 18, dau; Jesse, 16, son; John, 10, son; Charles, 7, son. All born Ky Ky Ky.

This is the only record I’ve seen that shows the name of Coleman’s wife as anything other than Joanna or some variant spelling. I’m not entering Louisa as a middle name for her, because it is possible that the census taker made a data entry error, and combined the name of Coleman’s daughter Louisa with the age for his wife Joanna. (Very often census takers took down notes in cheap lightweight notebooks when making their rounds, and transferred those notes into the official ledgers at home, rather than carrying the heavy ledgers around with them. Errors then sometimes crept in during the copying process.)

1900 census of Morgan Co., Ky, Panthers Branch precinct, enumerated 2 June, lists on p. 2A as family #23/24:
Brown, Coleman, 82, farmer, b. Ky Va Va; Joana, wife, 69, b. Ky Va Va, married 52 years, mother of 10 children, 7 living; Maggie, dau-in-law, 29, b. Sep 1870, Ky Ky Ky, widowed, mother of 2 children, 2 living; Lizzie E., granddau, 8, b. June 1891, Ky Ky Ky.

Two households above is the family of Coleman and Joanna’s son James W. Brown and his wife Nancy. Maggie’s other child is very likely to be the Ezra W. Brown, nephew, age 6, listed with James and Nancy.