Human communication of emotions is achieved through both facial and vocal information. The purpose of this study is to investigate the dominant sensory modality in recognition of emotions to the multi-modal expression. In Experiment 1, expressions of happiness, surprise, sadness, or aversion was presented vocally, facially, or in both modalities through the expression of an interjectory word “eh”. Participants were required to judge the emotion that was expressed. In Experiment 2, recognition of conflicted emotions between modalities was investigated by combining different emotions between facial and vocal expression. Results of the two experiments indicated that the observers predominantly recognized happiness and surprise that was expressed facially rather than vocally. Furthermore, the expression of happiness was often mistaken as surprise, and the expression of sadness was often mistaken as aversion. Importantly, however, the reverse of these mistakes was little observed. Such the asymmetries of confusion were consistently obtained in every modality including bimodal presentations. This evidence is suggestive that an amodal processing system exists in multi-modal recognition of emotions.