This study explores the hierarchical order in the directionality of synaesthetic metaphors by means of a psychological experiment. Previous studies present varied models of hierarchy among different sense modalities for synaesthetic metaphors. Among the unsettled questions, we address the issues of the separability and the positions of “temperature”, “shape”, and “taste” in synaesthetic metaphors in Japanese, as well as their directions of extension. We carried out a questionnaire-based comprehensibility test with 245 Japanese noun phrases. Each phrase consists of an adjective and a noun from seven modalities (temperature, touch, smell, taste, shape, color, sound). Our results show the following three points concerning synaesthetic phrases in Japanese, (i) the modality of temperature is distinguished from that of touch, and temperature is located at the left end of the scale in the model of directionality; (ii) the modality of shape is separable from color, and its position in the model of directionality is closer to touch than to color; (iii) the modality of taste does not become the target of extension from higher modalities like sound or color, and color does not become the source of extension to lower modalities like touch, taste, and smell. Cross-linguistic variations as well as similarities are found by comparing these results with previously hypothesized directionality models.