They perform every Sunday in a church first chartered more than 120 years ago, but the music itself goes back much, much further.