The temple's name 'Stone Hand' comes from a legend where a priest opened a baby's clenched fist to reveal a stone inscribed 'Emon Saburō is reborn,' proving the child was a reincarnated pilgrim.

Founded by Gyōki as Annoyō-ji, the temple was converted from Hossō to Shingon Buddhism by Kūkai himself, the founder of the Shikoku pilgrimage tradition.