ML framework overview
Our end-to-end ML framework operates as follows (Fig. 1b; see the “Methods” and the Supplementary Information for full technical details). For each company–year (i.e., each company observed in a given year), we collect the corresponding annual and sustainability reports (as PDFs), extract and clean the text from the PDFs, split the text into overlapping chunks, and index the chunks in a vector database. Within each company–year, we then query the indexed text separately for each of the 501 ESG indicators, retrieve and re-rank candidate chunks, and use a state-of-the-art LLM to extract the requested numeric value (or abstain if absent). Finally, we standardize units and currencies and store the resulting output in a structured format (Supplementary Fig. S4). We apply this framework to STOXX Europe 600 index constituents (as of 2023) over the 2014–2023 period. Key methodological limitations are discussed in the Supplementary Information S4.
We validated the outputs against (i) a proprietary benchmark dataset and (ii) expert human annotations on a subset. The agreement is strong against both comparisons (Supplementary Fig. S1–S2), which implies that the generated dataset is reliable for the descriptive analyses below. The full validation design and error analysis are described in Methods section under Validation.
While our framework enables large-scale, automated ESG indicator extraction, the reliability of its outputs depends on the quality of extraction at each stage of the pipeline. We therefore quantify how observations progress through each stage of the ML pipeline and report the results at both the company–year level and the indicator–company–year level, which allows us to identify at which stage information enters—or drops out of—the process (Supplementary Fig. S6). Building on this decomposition, an indicator may fail to be retrieved for two reasons: (i) it may not be disclosed (e.g., due to non-materiality or because the company reports a different indicator or uses a different reporting variant), or (ii) the extraction may fail (e.g., due to ambiguous reporting). To assess whether the extraction may further vary across indicators, we additionally report per-indicator disclosure detection and standardization rates (Supplementary Table S6).
ESG-related transparency
We use our ML framework to track ESG indicators from both the annual and sustainability reports of the 600 major European companies listed in the STOXX Europe 600 (as of 2023) for the time period 2014 through 2023 (Fig. 1b). These companies cover nearly 90% of the investable equity market in Europe and span 16 European countries, including the UK (139 companies), France (78), and Germany (71) (Fig. 1c). The full list of all companies is provided in Supplementary Table S1. From each report, we extract reported ESG indicators from among the 501 indicators defined by European Sustainability Reporting Standards (ESRS)35 (see Supplementary Information S1 for background). ESRS are structured along over-arching topics (e.g., climate change, own workforce, business conduct), each mandating a granular set of quantitative ESG indicators that collectively constitute the 501 indicators. We track these indicators to determine: (1) whether the indicator is present or absent from the corporate report (transparency), and, if present, (2) its numerical value (performance). Accordingly, this enables systematic benchmarking across firms, sectors, and time, which helps generate descriptive evidence, but without causal claims, to inform hypothesis development and demonstrate the scientific value of the dataset.
Our analysis shows an overall trend toward increased transparency (Fig. 2a). The average number of disclosed ESG indicators increased from 117.8 (2014) to 179.7 (2023; + 52.5%), with more pronounced increases in specific topics such as climate change ( + 125.0%), water ( + 83.0%), and circular economy ( + 73.0%). Notably, the number of reported ESG indicators varies across industry sectors due to differences in business model, resource intensity, and regulatory exposure. But even beyond structural sector differences, our analysis points to a notable transparency gap (Fig. 2b, c). In 2023, companies with a top − 10% ESG rating (based on external data providers38) disclosed on average 186.5 ESG indicators, compared to an average of 174.7 indicators for companies that are lagging behind, corresponding to a 6.8% difference. This gap was substantially wider in 2014, when companies in the upper decile reported on average 144.7 indicators versus just 103.8 for the bottom decile (difference: 39.4%). This narrowing of the transparency gap over the last decade coincides with a general increase in average ESG ratings over this period38, suggesting an overall strengthening and alignment of data availability about corporate sustainability.
Fig. 2: Transparency gap between average and top-in-class companies.
a Shown is the average number of ESG indicators disclosed annually, grouped by ESRS topic. Of note, the number of mandated ESG indicators varies across sectors due to differences in business model, resource use, and regulatory exposure. To assess the transparency gap, we compare the average against companies that are b higher-rated and c lower-rated in terms of sustainability performance. Specifically, we compare against companies for which the ESG rating (based on lagged MSCI ESG ratings38) ranks in the top − 10% and bottom − 10%, respectively. Deciles are calculated on a yearly basis. This analysis shows how top- and bottom-rated companies compare in ESG disclosure relative to the full sample.
We find substantial disparities in ESG-related transparency across topics and industries, as measured by a transparency score defined as the relative number of disclosed indicators out of all 501 indicators (Fig. 3a, b). This could be explained by differences in the materiality of topics, that is, certain sectors or entities may not be affected by a given topic and therefore do not disclose detailed information, or different scope and depth of disclosure frameworks prior to the ESRS. Overall, transparency scores are unequally distributed across topics (Fig. 3a). In 2023, topics with particularly high transparency scores were own workforce (54.7%), governance (48.9%), and circular economy (42.3%), whereas transparency scores were particularly low for the topics pollution (5.6%) and biodiversity (8.3%). Overall, several topics show increasing transparency scores between 2014 and 2023, such as circular economy (change in transparency score: + 17.8 percentage points [p.p.]), climate change ( + 15.5 p.p.), and own workforce ( + 13.1 p.p.). There is also some heterogeneous variation in overall as well as topic-specific transparency scores across industries (Fig. 3b). Variation in overall transparency may reflect sector-specific disclosure traditions and peer benchmarking effects, whereby firms adopt similar practices as their industry peers to meet investor and stakeholder expectations. Topic-specific variation might reflect their relative importance across industries. For example, whereas topics like own workforce and governance consistently score highest, arguably due to being universally material and often regulated, the relative importance of other topics varies. One striking example is the financial industry, which displays comparatively lower transparency across environmental topics in contrast with social and governance, consistent with its lower direct ecological footprint but greater exposure to organizational and ethical issues. Still, the relative ranking of topics shows broadly similar characteristics across industries, with one possible explanation for that being a preference of companies to report on commonly accepted, measurable metrics requested by data providers and rating agencies. Further context on ESG transparency trends and the role of rating agencies is provided in Supplementary Information S2.
Fig. 3: Transparency across ESRS topics and industries.
Here, transparency score refers to the relative number of disclosed indicators out of all 501 indicators. a reports the transparency scores disclosed by ESRS topic between 2014 and 2023. b shows the transparency scores by industry. Industries are categorized according to the Sustainable Industry Classification System (SICS), developed by the Sustainability Accounting Standards Board (SASB) to classify companies into sectors with comparable exposure to sustainability-related risks and opportunities68. Industries in (b) are sorted by their overall transparency score (descending).
To identify potential drivers behind ESG transparency, we analyze transparency scores across different company characteristics. For this analysis, we compare companies in the top 10% and bottom 10% by market capitalization and ESG rating. In general, larger companies are expected to be more transparent, as they have more resources and stronger incentives (e.g., public exposure) for ESG reporting. Companies with higher ESG ratings are typically more advanced in their sustainability practices and should thus tend to disclose more. Overall, transparency is indeed lower among smaller companies and those with lower ESG ratings (Fig. 4a,b).
Fig. 4: Heterogeneity in ESG-related transparency across company characteristics.
Here, we analyze transparency scores (i.e., the relative number of disclosed indicators out of all 501 indicators) by (a) market capitalization, b ESG rating (based on MSCI ESG ratings38 measured annually by company), and c ESG controversies score (by Refinitiv60; measured annually by company). The ESG controversies score captures exposure to negative ESG-related events and news coverage39. All independent variables are lagged by one year to mitigate potential reverse causality. For each, we compare companies in the bottom 10% and top 10% against the middle 80%, calculated on a yearly basis. We inverse-code the ESG controversies score such that companies with more controversies appear in the top 10%. Shown are violin plots, which represent the kernel probability density of the data, together with boxplots displaying the median and interquartile range. Reported below each group is the number of firm-year observations (n). Reported above the brackets is the difference (diff) in percentage points (p.p.). Statistical comparisons are based on two-sided t-tests (with *** corresponding to the 0.1% significance level). Whiskers indicate the range of non-outlier values, extending to 1.5 times the interquartile range beyond the first and third quartiles.
Specifically, companies in the top 10% by market capitalization and ESG rating have significantly higher transparency scores than the middle 80% by + 3.1 p.p. (p < 0.001) and + 3.7 p.p. (p < 0.001), respectively. In contrast, transparency scores are significantly lower for companies in the bottom 10% of market capitalization ( − 3.9 p.p.; p < 0.001) and ESG rating ( − 2.5 p.p.; p < 0.001) compared to the middle 80%.
Further, we analyze the ESG controversies score, a proprietary metric capturing exposure to negative ESG-related events and news coverage39 (Fig. 4c). Companies with higher (i.e., worse) ESG controversies scores are those whose ESG performance has been more contentious in the past. Here, we find that companies in the bottom 10% of the ESG controversies score (i.e., the least controversial) have significantly lower transparency scores than the middle 80% ( − 2.4 p.p.; p < 0.001), while the difference for companies in the top 10% is small and not statistically significant ( − 0.6 p.p.). This suggests that companies with low controversy exposure also tend to disclose less, possibly because they face less external pressure to report comprehensively on ESG matters. Overall, these differences translate into substantial transparency gaps between companies, as confirmed in a regression analysis (see Table 1). For example, companies in the top − 10% of ESG ratings have, on average, 22% higher transparency scores than companies in the bottom − 10%, highlighting the substantial transparency gap between the two groups.
Table 1 Determinants of transparency
In addition to assessing corporate ESG transparency, our framework enables large-scale, granular analysis of ESG performance—that is, the numerical values of disclosed ESG indicators over time. We focus our analysis on a selected set of indicators that are particularly relevant to current public debates and recent policy frameworks (e.g., the Corporate Sustainability Due Diligence Directive40, and the Zero Pollution Action Plan41). The complete set of indicators is included in our dataset (see Data availability statement) and is accessible through an interactive dashboard at (https://lookerstudio.google.com/s/jUlW6L1X8u8).
Environmental performance
Overall, progress toward reducing corporate emissions is mixed (Fig. 5a–d). The graphs show the development of selected indicators over time and provide information on percentiles as well as reporting intensity (i.e., the number of firms reporting on a particular indicator in a given year). For example, median scope 1 emissions (i.e., direct GHG emissions from sources owned or controlled by the company such as production facilities or company vehicles) have declined by 66.8% since 2014 (Fig. 5a), suggesting a gradual shift toward lower direct emissions. Similarly, median scope 2 emissions (i.e., indirect emissions resulting from purchased electricity, steam, heating, or cooling) have declined by 76.4% (Fig. 5b). In contrast, total scope 3 emissions (i.e., indirect emissions across the companies’ upstream and downstream value chain), remained largely stagnant between 2014 and 2020, before increasing by a factor of 5.6 through 2023 (Fig. 5c).
Fig. 5: Environmental performance over time.
Here, we analyze key environmental indicators between 2014 and 2023. Shown are (a) total scope 1 emissions, b total scope 2 emissions, c total scope 3 emissions, d scope 3 emissions (travel), e energy consumption, f the percentage of renewable sources in total energy consumption, g water consumption, h total waste generation, and i the percentage of non-recycled waste. For each indicator and year, we present the median value across all reporting companies (solid line, 50th percentile), the interquartile range (shaded band, bounded by the 25th and 75th percentiles, inner dashed lines), and the 10th and 90th percentiles (outer dashed lines) to capture the distribution and highlight variation between top and bottom-performing companies. For each panel, dot sizes indicate reporting intensity (i.e., the number of companies disclosing values for each indicator in a given year), n denotes the total number of company–year observations for the indicator, and Δ denotes the percentage change in indicator value between 2014 and 2023. Note that logarithmic axes are used to better visualize ESG indicators that span several orders of magnitude. For indicators recorded in percent, we use linear scales. In d we also indicate the time period of the COVID-19 pandemic, referring to the World Health Organization’s definition of COVID-19 as a public health emergency of international concern91.
In the specific case of scope 3 emissions, however, we argue that this sharp increase is mainly attributable to higher transparency levels: companies have started to report emissions data for a wider range of previously untracked categories, which mechanically raises total scope 3 emissions. To substantiate this, we inspect the 15 scope 3 categories as defined in the Corporate Value Chain Accounting and Reporting Standard by the GHG Protocol. These exhibit no clear upward trends in individual categories (see Fig. 6) but a stark increase in the number of categories companies report emissions for (see Supplementary Fig. S21)42. Additionally, we confirm through regression analysis that the increase in total scope 3 emissions is partly driven by the increase in reported categories (see Supplementary Table S4). As expected, we observe a sharp drop in scope 3 emissions from travel during the COVID-19 pandemic (Fig. 5d). Although travel-related emissions have rebounded by a factor of 2.8 since 2021, they remain below pre-pandemic levels in 2023, which suggests that many companies have either revised their travel policies or continue to embrace virtual meetings as part of a post-pandemic shift in workplace practices. This analysis highlights that the perceptions of corporate ESG performance are shaped by data availability, that is, ESG transparency. By enabling the tracking of ESG indicators, our ML approach can contribute to the ability of the public to monitor corporate ESG performance.
Fig. 6: Scope 3 emissions by categories.
Here, we analyze subcategories of companies’ scope 3 emissions. Shown are scope 3 emissions for (a) purchased goods and services, b capital goods, c fuel and energy-related activities, d upstream transportation and distribution, e waste generated in operations, f travel, g commuting, h upstream leased assets, (i) downstream transportation and distribution, j processing of sold products, k use of sold products, l end-of-life treatment of sold products, m downstream leased assets, n franchises, and o investments. For each indicator and year, we present the median value across all reporting companies (solid line, 50th percentile), the interquartile range (shaded band, bounded by the 25th and 75th percentiles, inner dashed lines), and the 10th and 90th percentiles (outer dashed lines) to capture the distribution and highlight variation between top- and bottom-performing companies. For each panel, dot sizes indicate reporting intensity (i.e., the number of companies disclosing values for each indicator in a given year), and n denotes the total number of company–year observations for the indicator. Note that logarithmic axes are used to better visualize different orders of magnitude.
Beyond emissions, progress across other environmental indicators is uneven, with some showing considerable changes, while others have stagnated (Fig. 5e–i). For example, the increasing share of renewable energy sources (Fig. 5f), accompanied by a decline in fossil-based energy (not reported here), indicates growing alignment with global decarbonization efforts and a gradual transition toward more sustainable energy sources. However, companies in the highest decile of energy intensity—measured as energy consumed per EUR of revenue—continue to rely predominantly on fossil fuels. Furthermore, energy (Fig. 5e) and water consumption (Fig. 5g) have declined by 37.7% and 38.1%, respectively, over the observation period. Reductions in total waste generated (Fig. 5h), and non-recycled waste (Fig. 5i) have been modest.
To account for the underlying trends in economic activity, we analyze inflation-adjusted median revenues across the sample period from the Worldscope database43. Overall, we find only a modest increase of 16.7% over the full sample period (see Supplementary Table S3), suggesting that macroeconomic growth alone is unlikely to fully explain the trends in ESG performance. By analyzing intensity ratios, we find that the companies in our sample seem to have made real adjustments in some areas (e.g., reducing scope 1 and 2 emission intensities), while, in other areas, the intensity ratios remain stagnant (e.g., energy consumption) or increase (e.g., indirect emissions) (see Supplementary Fig. S13).
To further explore heterogeneity across companies, we stratify our analysis of corporate ESG performance by different company characteristics, namely, market capitalization, ESG ratings, and ESG controversies scores, comparing companies in the top and bottom 10% of each characteristic (see Supplementary Fig. S22–S24). Here, scope 1 emissions are substantially lower among smaller companies, companies with lower ESG ratings, and those associated with fewer ESG controversies. We observe similar results for most of the other indicators.
The observed differences suggest that, in line with expectations, environmental performance is lower for larger companies and companies with more controversies around ESG practices. Notably, companies with higher ESG ratings also exhibit lower performance, and this pattern holds when analyzing intensities instead of absolute performance levels (as larger companies tend to have higher ESG ratings), highlighting the need to differentiate between sustainability transparency and performance (see Supplementary Fig. S14). There are two possible explanations for why companies with higher ratings exhibit lower environmental performance. First, ESG ratings as those by MSCI do not solely assess risks arising from negative impacts but also emphasize how these risks and opportunities are managed, which can favor larger firms with more advanced governance structures. Second, the MSCI rating is determined relative to peers within the same industry, meaning that companies operating in resource-intensive sectors (e.g., extractives & minerals processing) may still receive comparatively high ratings despite sizable absolute impacts if they outperform their sector counterparts38.
Finally, to account for changes in the number and composition of reporting companies over time, we further examine whether trends differ between early and late adopters of ESG-related reporting. Specifically, we conduct a separate analysis comparing firms that began ESG reporting early in the sample period to those that started later (see Supplementary Fig. S15). We find that overall trends in performance remain consistent across both groups, indicating that the main findings are not driven by sample composition. Restricting the sample to companies that report a given metric in all sample years yields closely similar trends (see Supplementary Fig. S18–S20).
Social performance
Performance across social indicators is mixed (Fig. 7). Notably, for instance, employee turnover has increased by 2.6 p.p. since 2014 (Fig. 7a), suggesting growing challenges in workforce retention. At the same time, the share of female employees in top management has increased steadily by 9.2 p.p. (Fig. 7e), which reflects ongoing efforts to promote gender equality in corporate leadership. The gender pay gap has narrowed by 4.7 p.p. since 2014 but has widened again by 0.5 p.p. since 2021 (Fig. 7f), suggesting potential setbacks in corporate efforts to promote equal pay. In 2023, the amount of fines, penalties, and compensation for damages as a result of incidents and complaints has decreased by 69.5% compared to 2014 levels. In contrast, the number of training hours per employee (Fig. 7c) remains stagnant over the observation period.
Fig. 7: Social performance over time.
Here, we analyze key social indicators between 2014 and 2023. Shown are (a) employee turnover, b the percentage of employees covered by collective bargaining agreements, c the number of training hours per employee, d the number of days lost to work-related injuries, ill health, and fatalities among employees, e the percentage of female employees in top management, f the gender pay gap, g the annual remuneration ratio (i.e., defined as the ratio of total annual compensation for top executives compared to the median employee), h the number of complaints filed by own workforce, and i, the number of fines, penalties, and compensation for damages as a result of incidents and complaints. For each indicator and year, we present the median value across all reporting companies (solid line, 50th percentile), the interquartile range (shaded band, bounded by the 25th and 75th percentiles, inner dashed lines), and the 10th and 90th percentiles (outer dashed lines) to capture the distribution and highlight variation between top and bottom-performing companies. For each panel, dot sizes indicate reporting intensity (i.e., the number of companies disclosing values for each indicator in a given year), n denotes the total number of company–year observations for the indicator, and Δ denotes the percentage change in indicator value between 2014 and 2023. Note that logarithmic axes are used to better visualize different orders of magnitude. For indicators recorded in percent, we use linear scales.
However, other indicators even exhibit decreases over time. The share of employees covered by collective bargaining agreements has decreased by 7.1 p.p., (Fig. 7b), indicating a weakening of employees’ bargaining power. Similarly, the annual remuneration ratio (i.e., the ratio of total annual compensation of the highest-paid individual to the median annual total remuneration for all employees, excluding the highest-paid individual) has increased by 1325.9% since 2014 (Fig. 7g), pointing to a widening gap between executive compensation relative to employee pay. The number of days lost to work-related injuries, ill health, and fatalities among employees has increased by 40.4% (Fig. 7d), mirroring an increase in the number of complaints filed by own workforce by 39.1% (Fig. 7h).
Consistent with our findings for environmental performance, companies that began ESG reporting earlier do not exhibit substantially different trends in social performance (see Supplementary Fig. S16), and trends among constant reporters are closely similar (see Supplementary Fig. S19). As expected, social indicators measured in absolute terms exhibit lower levels for smaller companies while relative metrics display comparable performance, with the exception of those pertaining to gender equality, where larger companies seem to perform better, possibly due to higher public exposure (see Supplementary Fig. S25). For ESG ratings and controversies, the results are mixed (see Supplementary Fig. S26–S27).
Governance performance
To understand governance practices (Fig. 8), we examine the share of independent board members, which remains consistently high, reaching 75.0% in 2023 (Fig. 8a), reflecting previous governance reforms aimed at strengthening board autonomy. In contrast, lobbying expenses have increased by 747.6% since 2019 (Fig. 8b). In the case of lobbying expenses, the earlier drop around 2014/15 could be attributed to limited or strategic disclosure by early adopters of ESG-related reporting (see Supplementary Fig. S17); trends among constant reporters are broadly consistent (see Supplementary Fig. S20). However, transparency on lobbying expenses remains generally low, resulting in fewer observations compared to other indicators shown here. Lobbying expenses are also higher for larger companies and companies with higher ratings, as well as more controversies around their ESG practices. For other governance indicators, the results are mixed, while larger companies tend to have a higher degree of independence within their board structure (see Supplementary Fig. S28–S30).
Fig. 8: Governance performance over time.
Here, we analyze key governance indicators between 2014 and 2023. Shown are (a) the percentage of independent board members, b lobbying expenses, and c the number of days to pay invoice. For each indicator and year, we present the median value across all reporting companies (solid line, 50th percentile), the interquartile range (shaded band, bounded by the 25th and 75th percentiles, inner dashed lines), and the 10th and 90th percentiles (outer dashed lines) to capture the distribution and highlight variation between top- and bottom-performing companies. For each panel, dot sizes indicate reporting intensity (i.e., the number of companies disclosing values for each indicator in a given year), n denotes the total number of company–year observations for the indicator, and Δ denotes the percentage change in indicator value between 2014 and 2023. Note that logarithmic axes are used to better visualize different orders of magnitude. For indicators recorded in percent, we use linear scales.