[KCD2] Question about The Hermit quest?

I got all the books and the cape thing from the grave so I should have special dialogue when I find the hermit right? When I talk to him though even after reading the books he just tells me to fuck off and there's no dialogue. What am I doing wrong?