Kobo Daishi prayed at the Bishamonten cave for the safe completion of Manno-ike Reservoir renovations. The project finished in just three months, and this temple was built to commemorate that achievement.

Koyama-ji está associado a Yakushi Nyorai, o Buda da Medicina, que é venerado aqui como a divindade principal do templo.