Kobo Daishi personally carved the temple's main image, a Thousand Armed Kannon, when he converted Yashima-ji to Shingon Buddhism in 815 and relocated it to its current spot on the plateau's south side.

The temple sits atop a lava plateau that once served as an ancient fortress connected to Japan's 7th-century defeat at the Battle of Hakusukinoe in Korea, when they allied with the Kingdom of Baekje.