The sense of agency is the sense that one is causing an action. The predictive account of the sense of agency proposes that sensory prediction based on efferent information plays a critical role in generating the sense of agency. Alternatively, the inferential account of the sense of agency proposes that we experience the sense of agency when we infer that one's own thoughts are the cause of an action. According to this account, the inference occurs when a thought appears in consciousness prior to an action, is consistent with the action, and is not accompanied by conspicuous other causes of the action. Recent study showed that both of these factors did contribute to the sense of agency. In this paper, multi-layered model of sense of agency was presented. Within this framework, the basic level consists in sensori-motor processes, while the higher level comprises conceptual process. At the basic level, the non-conceptual feeling of agency is produced by sensori-motor integration process of efferent with afferent information. In case of incongruence between these indicators, the pre-conceptual feeling of agency is further processed by inference mechanism to form an attribution of agency (judgment of agency). This paper proposes that what is self-specific is not judgment of agency but rather non-conceptual feeling of agency. That is, the intertwining of action and its sensory consequence in the world makes self so special.