A new research approach to the "but-for" question

| Posted in analysis, data, economic development, effectiveness, evaluation | | No Comments

A new report from the Urban Institute assesses how the synthetic control method can be used for economic development policy evaluation.

The synthetic control method (SCM) creates “a synthetic control region that simulates what the outcome path of a region would be if it did not undergo a particular policy intervention.” The method involves creating a “hypothetical counterfactual region” based on combined predictor variables from specified “donor regions” – such as other states – and comparing outcome variables. It is considered a useful quantitative supplement to qualitative case studies because it creates a control group that can help analysts separate policy effects from other variables. In this way, the SCM method has the potential to assess the ‘but for’ test in economic development evaluations.

The procedure gained attention from an analysis of California’s Proposition 99, which raised cigarette excise taxes and established an antitobacco media campaign. The analysis used the SCM method to consider the effect on smoking in the state and found that per capita sales of cigarette packs fell because of the program, which was unique to California. The study did so by creating a “synthetic California” based on a pool of other states that had similar predictor variables (such as per capita GDP, population characteristics, average price of cigarettes) but did not pursue a similar antismoking policy.

The report authors describe how SCM has been used for economic development analyses. For example, a review of a tourism development policy in a province of Argentina was found to raise employment in the tourism industry. Another examined whether the siting of nuclear power plants in Japan created economic benefits as measured by change in real per capita taxable income. In the latter example, the results were mixed among the locations studied and did not yield clear guidance on the economic benefits of the investments.

The SCM is promising for economic development evaluation but has limits. The policy change must be limited to the region being studied and cannot have affected the region before going into effect. However, most economic development initiatives are not entirely new, but instead build on past efforts and are frequently replicated in other locations. At the same time, the region being studied cannot be an outlier, which might eliminate distressed locations that might be more willing to undertake unique or untested policy interventions. On the plus side, the authors describe it as an improvement over other quantitative methods in use and it is considered straightforward to use.

Bottom line: the SCM “can be a useful addition to an analyst’s toolkit” but should supplement rather than replace the case study approach.

For more information, see The Synthetic Control Method as a Tool to Understand State Policy, Robert McClelland and Sarah Gault, Urban Institute. March 2017.

No Comments

Leave a comment




Comments have to be approved before showing up..