The Science of Better Governance

Turning Policy Failures into Progress with Data, Design, and Discovery

Introduction

Jim Manzi, an American technology entrepreneur, has written extensively about the philosophy of science, politics, and policymaking. His book, Uncontrolled: The Surprising Payoff of Trial and Error for Business, Politics, and Society (2012), addresses a fundamental question that has long perplexed policymakers, economists, and social scientists. How do we effectively measure which social and economic policies yield the desired outcomes? Manzi's argument is a call to action, offering a potentially more effective policy-making process. He suggests that the current policy-making framework's failure to leverage the potential benefits of trial-and-error experimentation has led to persistent uncertainty about the efficacy of various strategies. This essay examines the importance of measuring policy effectiveness and the implications of this uncertainty, while also presenting a potential approach to enhance the measurement of government performance. This approach offers hope for a more effective policy-making process in addressing societal issues.

Challenges of Evaluating Policy Effectiveness

Evaluating the effectiveness of social and economic policies is crucial for informed decision-making. The government must know when a failed policy should be abandoned and replaced. But they don’t. Currently, the effectiveness of policies has remained unmeasured, so most policies, once implemented, stay in place for a long time. Without evaluation, we can’t know whether a policy achieves its intended outcome or causes harm. Implementing governmental policies occurs in a dynamic environment where many unmeasured variables, such as social attitudes (which can influence public acceptance and compliance with policies), individual and group behavior (which can affect the implementation and impact of policies), or technological advancements (which can change the context and potential of policies) interact, making it difficult to isolate the effects of a specific policy initiative. While helpful, traditional tools of economic analysis may fall short in capturing the complexity of social systems.

Manzi's work underscores the need for a more experimental approach to policy-making. Drawing on the success of methodologies used in the business sector, particularly A/B testing —a method where two versions (A and B) of a policy are compared to see which one performs better in achieving the desired outcomes —he advocates for a systematic process of trial and error. This approach, treating policies as hypotheses to be tested, offers a new perspective. The government can learn from failures and successes, inspiring a shift away from the more dogmatic approaches that dominate policy planning and discussions today.

The Social and Economic Implications

The consequences of ineffective policy evaluation extend beyond inefficiency; they impact trust, equity, and civic engagement. We waste resources when ineffective policies persist due to a lack of evaluation. This 'government waste' refers to the inefficient use of public funds, where public funds allocated for the policy initiative are wasted on programs that do not yield significant benefits, diverting attention and resources from more effective initiatives.

The lack of systemic evaluation can erode public trust in government institutions. The inability to discern effective social policies leads to social inequalities, leaving vulnerable populations without the necessary support. When policies fail to deliver measurable results, citizens become disillusioned with the political process, and as a consequence, there is less personal civic engagement. A transparent, evidence-based approach to policy evaluation can foster greater public trust and involvement, reassuring the audience and instilling confidence in the process. Without rigorous analysis, the same ineffective policies are frequently recycled across administrations and political cycles, contributing to stagnation in innovation and reform. Continued reliance on outdated and ineffective strategies creates an environment that prevents society from evolving and adapting to new challenges.

The Absence of Systemic Measurement

The absence of systemic and scientific measures to evaluate policy effectiveness contributes to government waste. Current methods often rely on correlational studies, a type of research that explores the relationship between two or more variables. However, they mostly rely on anecdotal evidence, which does not adequately account for the multiple variables in social contexts. Governments may selectively fund studies that affirm their existing policies, creating political bias and distorting evaluations. This can lead to a biased understanding of their effectiveness. Policymakers tend to be reluctant to acknowledge the failures of popular initiatives that are known to have been unsuccessful in achieving the ultimate goal.

The political culture is critical in creating obstacles to practical and objective evaluation. Frequently, decisions are made based on ideological beliefs and bias rather than empirical evidence. For example, investments in social welfare programs may be driven by a political agenda rather than careful analysis of their effectiveness in improving societal outcomes. This approach to making decisions based on an agenda and beliefs creates a political environment where politicians resist adopting a scientific and rigorous approach to evaluation, fearing that they will have to terminate popular policies that are not proven effective based on data.

Building a Culture of Evidence-Based Policymaking

We must take several steps to enhance the effectiveness of the government’s evaluation of social and economic policies.

First, our behavior must drive a cultural shift toward accepting and demanding experimentation and accountability in policy-making. We must embrace trial and error, moving away from the belief that government policies must always follow established norms. This shift towards accountability should inspire us to feel more responsible and engaged in the policy-making process.

Second, education and training for policymakers in statistical analysis, experimental design, and data interpretation must be prioritized. By empowering government officials with the necessary skills to conduct and analyze experiments, the quality of policy evaluation improves significantly. Policymakers should advocate for establishing independent entities dedicated to objectively assessing the effectiveness of government programs. These entities must operate free from political pressure, providing unbiased evaluations and guidance that informs policy decision-making. Collaboration with academic institutions and objective think tanks can enrich the policy evaluation. By leveraging academic expertise in the social and behavioral sciences, governments can better understand the impacts of their policies. Applying randomized controlled trials, longitudinal studies, and other rigorous methodologies fosters a culture that seeks out scientific approaches to policy making, which includes measurement and analysis. This emphasis on education and training should reassure the audience and instill confidence in the policy-making process.

Finally, technology offers powerful tools to support this shift. Data analysis and machine learning can uncover patterns and insights previously hidden by traditional analytical methods. Governments can harness big data to make more informed decisions while monitoring the real-time effectiveness of implemented policies.

Conclusion

Jim Manzi’s Uncontrolled (2012) delivers a powerful message about the importance of trial and error in evaluating social and economic policies. The key question he raises about determining effective policy, the implications of persistent uncertainty, and the absence of systemic measurement highlights a significant gap in contemporary governance. By embracing experimentation, fostering a greater understanding of data analysis among policymakers, and promoting collaboration with independent evaluators and academic institutions, society can improve the effectiveness of its policies.

By embracing transparency and data-driven decision-making, guided by collective prosocial values, we can optimize resource use and restore public trust in government. In an ever-evolving world, policymakers must reassess and refine their approaches to ensure that social and economic interventions are evidence-based and meet the needs of their constituents.

Thank you, and please pass it on!

Francisco & Faris