Koyama-ji está associado a Yakushi Nyorai, o Buda da Medicina, que é venerado aqui como a divindade principal do templo.

Koyama-ji está em Zentsuji, Prefeitura de Kagawa, ligando-o a uma das cidades de peregrinação mais conhecidas de Shikoku.

Young Kobo Daishi played in this area as a child, building mud Buddha statues and plant temples. The field where he played was called Sen-yugahara (Field Where Hermits Play).

A saint appeared from a cave on Mt. Koyama and told Kobo Daishi to build a temple there. That cave, located to the left of the Daishido Hall, became the birthplace of Koyamaji Temple.