Augusto Cerqua and Marco Letta are both assistant professors at the Department of Social and Economic Sciences of Sapienza, University of Rome. They were our guests at the D2 Seminar series, presenting their talk on “Losing control (group)? The Machine Learning Control Method for counterfactual forecasting” a joint work with Fiammetta Menchetti, our fellow.
The standard way of estimating treatment effects relies on the availability of a similar group of untreated units. Without it, the most widespread counterfactual methodologies cannot be applied. Augusto and Marco tackle this limitation by presenting the Machine Learning Control Method (MLCM), a new causal inference technique for aggregate data based on counterfactual forecasting via machine learning. The MLCM is suitable for the estimation of individual, average, and conditional average treatment effects in evaluation settings with short panels and no controls. The method is formalized within the Rubin’s Potential Outcomes Model and comes with a full set of diagnostic, performance, and placebo tests. They illustratde our methodology with an empirical application on the short-run impacts of the COVID-19 crisis on income inequality in Italy, which reveals a striking heterogeneity in the inequality effects of the pandemic across the Italian local labor markets.
What are the research questions behind your work? and could you explain briefly what is the Machine Learning Control Method you are proposing?
This research effort was born at the onset of the COVID-19 pandemic. When this unprecedented event hit Italy (which was also the first Western country to be severely hit and went into the international spotlight), we – like countless other researchers – became immediately interested in understanding and quantifying its socioeconomic impacts and their distribution across the country.
However, we soon realized we were facing a methodological problem we had not encountered before in our experience with impact evaluation: the pandemic and the lockdown represented an ubiquitous simultaneous shock which hit the whole of Italy, leaving no areas unaffected. In causal inference jargon, this meant we did not have a control group to build the counterfactual scenario in which the pandemic never hit Italy and, in turn, estimate treatment effects. The problem is that without a control group, most impact evaluation techniques cannot be applied. We were thus wondering: “If we cannot use the methods we are familiar with, then how can we accurately assess the impacts of COVID-19?”
This is when we came up with the Machine Learning Control Method, a methodology specifically suited for evaluation settings without a control group. It adds a new technique to the toolbox of empirical social scientists that can be used for the evaluation of nationwide policies, large-scale shocks, or situations with widespread spatial spillovers between treated and control units.
Using the Machine Learning Control Method, we were able to estimate the granular impact of the COVID-19 pandemic in Italy on several outcomes: excess mortality, employment, firm entries and firm exits. Given the ‘emergency’ situation in which we came up with this method, the first applications were purely intuitive, but now we are formalizing the methodology and embedding it within the Rubin’s potential outcome framework, as it is standard practice for quantitative impact evaluation methods.
What are the major challenges in addressing this kind of study?
There are two sets of challenges: empirical and methodological ones. Empirically, the challenges we face relate to the need to accurately estimate the counterfactual scenario at a very granular level, using areas with different sizes, and a host of machine learning routines which need to be fine-tuned on the data. There are a lot of choices and trade-offs involved in this empirical task, one for which we are leveraging both our knowledge of traditional causal inference tools as well as more recent insights from data science and applied predictive modelling.
Methodologically, it is not easy to conceptualize and formalize a new technique for such a peculiar evaluation setting. Rubin’s potential outcome framework offers us an important starting point, but it was specifically developed for settings in which the researcher also observes a set of untreated units. The assumptions and the causal estimands underlying the Machine Learning Control Method thus need to be adapted to this different setting. For instance, we are thinking a lot about how to properly adapt a key assumption of the Rubin’s potential outcome framework – the Stable Unit Treatment Value Assumption – to our framework.
Another methodological challenge has to do with our hybrid approach. Machine learning tools were born to deal with predictive tasks, not with the estimation of causal effects. The field of Causal Machine Learning is a recent and active one which lies at the intersection between traditional econometrics and machine learning. There are a lot of unanswered questions, such as, for instance, how to adapt machine learning techniques for forecasting tasks in a panel data setting, on which we are currently reflecting a lot. This is very exciting and stimulating for us, but also quite challenging!
What are the unexpected or most interesting results you encountered during this research?
From the applied viewpoint, the unexpected result was the very good performance of machine learning tools in forecasting counterfactual outcomes at a very granular level and using only a handful of pre-treatment periods. After all, at the beginning we just tried to use these methods without knowing a priori whether they would do a good job. We were surprised to see that in most cases they tend to be quite accurate, and much more so than other simpler techniques from the more traditional time series domain. This is especially noteworthy if you take into account that traditional forecasting tools cannot be used if you have a very short panel – a common occurrence in many economic settings, including our own.
In terms of actual results, the biggest surprise was the total lack of correspondence between the spatial distribution of the epidemiological impacts of the pandemic and the economic and inequality effects of the crisis. If you compare the different maps, you will see there is no overlap whatsoever: the mortality impacts in 2020 were mostly concentrated in Northern Italy, whereas the economic and inequality impacts are scattered throughout the country, and appear to be driven by pre-existing territorial characteristics of local economies which are unrelated to health aspects. Unfortunately, the widespread heterogeneity of the economic repercussions is something you do not hear talking about often in the media and – more importantly – in the policy debate, whereas we are of the opinion that it should be front and center of the resource allocation and implementation of the National Resilience and Recovery Plan.
What are the overall practical conclusions of your study on the impacts of COVID-19 on income inequality?
The application to the effects of the pandemic on income inequality in Italy is a new one, which features the latest version of our technique. The insights are therefore provisional and also focus on short-term effects. Yet, they are quite concerning: we find that the COVID-19 crisis led, in 2020, to a sudden and wide increase in income inequality in the country. At the same time, this average impact masks a striking degree of heterogeneity across the Italian territory: some areas, such as parts of Tuscany and Sardinia, saw large increases in inequality, whereas others were less affected or even experienced a reduction in inequality. We find that these difference are mostly due to intrinsic characteristics of local economies: the areas which suffered the largest increase in income inequality are those specialized in tourism, more isolated, and with a lower level of education and social capital. Overall, we reckon that these results should be of interest to policymakers and lead to the adoption of place-based policies to address these unfolding issues.