"Down the Chantry" certainly means 'at the Chantry'. What the Chantry is, who knows ? It could be an open space, it could be a nightclub or dance hall or a pub, it could be a part of town where a chantry, or something called 'the chantry', once stood, it could be a cinema, it could be anywhere where, or outside of which, this activity took place ( Chantry is not a word in current English; I only hear it in connection with church architecture, where it belongs: "
chantry; an endowment for a priest or priests to sing Masses for the soul of the founder, the chapel etc so endowed", says the Oxford Dictionary

)
In British we say, for example, 'down the pub' meaning 'at the pub' or 'in the pub' : " I met him down the pub". There is no suggestion of descending a hill or slope to get there

. Of course we also say 'up' for the same : " Are you coming up the pub?" could, logically (?) get the answer " I wasn't going , but I'll come down the pub with you". It wouldn't , in practice, get that reply because 'up' suggests that the speaker sees the visit as of some importance to him, it's a visit to the pub for a pleasant evening, for social reasons, whereas 'down the pub' suggests that the visit is not special and may even be seen as a duty or chore. So to reply 'I'll come
down the pub' would seem a little grudging or impolite when the first speaker has rated the visit worth an 'up'
