Living in complex social structures, humans have evolved a unique aptitude for mentalizing: trying to understand and predict the behaviour of others. To date, little is known about how mentalizing interacts with other cognitive processes. “Sense of agency” refers to the feeling of control over the outcomes of one’s actions, providing a precursor of responsibility. Here, we test a model of how social context influences this key feature of human action, even when action outcomes are not specifically social. We propose that in social contexts, sense of agency is affected by the requirement to mentalize, increasing the complexity of individual decision-making. We test this hypothesis by comparing two situations, in which participants could either consider potential actions of another person (another participant acting to influence the task), or potential failures of a causal mechanism (a mechanical device breaking down and thereby influencing the task). For relatively good outcomes, we find an agency-reducing effect of external influence only in the social condition, suggesting that the presence of another intentional agent has a unique influence on the cognitive processes underlying one’s own voluntary action. In a second experiment, we show that the presence of another potential agent reduces sense of agency both in a context of varying financial gains or of losses. This clearly dissociates social modulation of sense of agency from classical self-serving bias. Previous work primarily focused on social facilitation of human cognition. However, when people must incorporate potential actions of others into their decision-making, we show that the resulting socio-cognitive processes reduce the individuals’ feelings of control.