Sustainability outcomes are influenced by the laws and configurations of natural and engineered systems as well as activities in socio-economic systems. An important subset of human activity is the creation and implementation of institutions, formal and informal rules shaping a wide range of human behavior. Understanding these rules and codifying them in computational models can provide important missing insights into why systems function the way they do (static) as well as the pace and structure of transitions required to improve sustainability (dynamic). Here, we conduct a comparative synthesis of three modeling approaches— integrated assessment modeling, engineering–economic optimization, and agent-based modeling—with underexplored potential to represent institutions. We first perform modeling experiments on climate mitigation systems that represent specific aspects of heterogeneous institutions, including formal policies and institutional coordination, and informal attitudes and norms. We find measurable but uneven aggregate impacts, while more politically meaningful distributional impacts are large across various actors. Our results show that omitting institutions can influence the costs of climate mitigation and miss opportunities to leverage institutional forces to speed up emissions reduction. These experiments allow us to explore the capacity of each modeling approach to represent insitutions and to lay out a vision for the next frontier of endogenizing institutional change in sustainability science models. To bridge the gap between modeling, theories, and empirical evidence on social institutions, this research agenda calls for joint efforts between sustainability modelers who wish to explore and incorporate institutional detail, and social scientists studying the socio-political and economic foundations for sustainability transitions.