One-time key based phase scrambling is proposed for privacy-protected image matching. The image matching is performed using invisible templates that are protected by phase scrambling. A key for phase scrambling is not required for the image matching, and the key can even be discarded after scrambling if the template does not need to be reconstructed. Theoretical analyses are presented to provide guidelines for designing key parameters that affect the visual effect and image matching. Experimental results demonstrate the effectiveness and appropriateness of the proposed method.