Kobo Daishi prayed at the Bishamonten cave for the safe completion of Manno-ike Reservoir renovations. The project finished in just three months, and this temple was built to commemorate that achievement.

Koyama-ji is associated with Yakushi Nyorai, the Medicine Buddha, who is venerated here as the temple's principal deity.