Variance among assessments of a company’s environmental, social, and governance (ESG) activities is not necessarily the product of error or bias, as is sometimes presumed in the literature. The presumption that there is a singular ESG value at which assessments must converge is counter-productive to the very nature of ESG reforms, which stem from the recognition that we need alternative measures of company value. Instead, we must develop a framework that can be used to identify and foster healthy forms of variance among ESG assessments. In the ESG context, ensuring intra-assessor consistency through disclosure may thus be a more reliable measure of integrity and accuracy of assessments rather than seeking to achieve inter-assessor consistency.
Growing Pains in ESG Data Provision Markets
As more attention is given to firms’ ESG considerations, ESG data and ratings providers serve an increasingly important function in the corporate discourse. It is reported that there were more than 160 ESG data and ratings providers in 2020 and more than 600 ESG ratings and rankings products available globally as of 2018.
Even as the ESG provider and product markets have grown exponentially, the lack of ESG data has been cited as an impediment to a broader embrace of the ESG movement. For example, a 2017 BNP Paribas survey of institutional investors revealed that more than half (55%) of respondents regarded the lack of robust ESG data as the most significant barrier to greater adoption of ESG strategies. One source of this perception of inadequacy originates from the widely reported variance among ESG assessments.
A Summary of the Literature on the Variance among ESG Assessments
Variance among ESG assessments has been a key focus of academic research on ESG data providers. Most recently, a 2022 study by Florian Berg, Julian Kölbel, and Roberto Rigobon investigates the variance among ESG ratings based on data from six ESG ratings providers and 709 underlying indicators.1 The authors report correlations between ESG ratings ranging from 0.38 to 0.71 and attribute the lack of correlations primarily to measurement divergence.
In another widely cited study, Aaron Chatterji and co-authors focus on substantial divergence among the ratings of six ESG raters, which they evaluate as a validation problem.2 A 2010 paper by Robert Daines, Ian Gow, and David Larcker studies the quality of four commercial rating services that rate the corporate governance practices of firms, which falls under the “G” pillar of ESG.3 The authors find that variance among these rating services is a potential indication of significant measurement error.
Another recent study by Ventura Charlin, Arturo Cifuentes and Jorge Alfaro investigates the ratings provided by four leading rating agencies and reports a low level of reliability (18.3%) and agreement (5.4%).4 Noting that the reliability and consistency across ESG raters is much lower compared to other contexts (such as bond ratings and wine ratings), the authors view the state of ESG assessments to be in disarray.
A 2015 review of the ESG scores of three sustainability ratings providers finds a lack of convergence among ESG measurement concepts, including distribution and risk.5 Similarly, a study of six sustainability indices and ten ESG agencies finds that the methods currently used by ESG agencies and sustainability indices show a lack of standardization.6 Another examination of four ESG ratings providers on the basis of their dimensionality, reliability, and validity finds significant differences across measurement constructs, despite some commonalities.7
The emerging consensus from this literature is that variance among ESG assessments is a reason to doubt their accuracy and validity. It has led to a legitimacy crisis for ESG gatekeepers and, furthermore, the ESG movement.
The Duality of Variance among ESG Assessments
In my recent paper, “The Duality of Variance Among ESG Assessments,” I explain that convergence is not necessarily a proxy for reliability and may itself be the product of inflation, laxity, groupthink, or monopolistic market conditions. For example, the credit ratings of structured finance products during the 2007–2008 period were highly convergent, yet were later found to have been inflated and believed to have been the catalyst of one of the most devastating financial recessions in recent history. Inflated ratings were also at the heart of the dotcom bubble, the East Asian Financial Crisis, and the accounting scandals of 2001 and 2002 that led to the collapse of Enron and other landmark corporations.
Divergence among assessments is neither categorically harmful nor categorically optimal. In the ESG context, divergence among ESG products could be the product of error or bias but could also be the product of relativism, complexity, and subjectivity of the subject matter being assessed.
Fostering Optimal Variance among ESG Assessments
Acknowledging this duality of variance among ESG assessments, what are some market or regulatory efforts that can be used to cultivate optimal variance and mitigate harmful variance among ESG assessments? The common thread is transparency.
First, we need transparency about ESG’s meaning to each stakeholder. Some investors view ESG as a pathway to generate greater financial returns (the “financial value of ESG”). Others view ESG as a standalone measure of firm value that is independent of, and sometimes even counter to, financial returns (the “values-based view of ESG”). Under the values-based view of ESG, ESG’s significance primarily lies in imbuing corporations with a social responsibility to uphold ESG priorities which may mean different things to different stakeholders. An ESG framework that embraces both the financial and values-based views of ESG must be premised on the disclosure of the methodologies of each ESG assessor and internal consistency in the methodologies’ use and application.
The International Organization of Securities Commissions (“IOSCO”) has outlined one disclosure framework that could provide a useful template. IOSCO proposes that ESG assessors should be required to disclose: (1) the measurement objective of the ESG rating or data product; (2) the criteria used to assess the entity or company; (3) the key performance indicators used to assess the entity against each criterion; (4) the relative weighting of these criteria to that assessment; (5) the scope of business activities and group entities included in the assessment; (6) the principal sources of qualitative and quantitative information used in the assessment as well as information on how the absence of information was treated; (7) the time horizon of the assessment; and (8) the meaning of each assessment category (where applicable).8
Second, we need transparency in ESG data collection and dissemination practices. Currently, there is no common market practice for the method by which ESG information is gathered from companies, leading to redundancies and gaps in data collection practices. One recommendation from IOSCO that has been favorably received by market participants is for providers and firms to work together to develop one master report that is used to gather ESG data.9 Such an effort would ensure consistent inputs among assessors and a more streamlined and predictable data-gathering process for the assessed firms.
Third, we need transparency about ESG assessors’ funding models. In the credit ratings context, the main source of conflicts of interest was attributed to how the credit rating agencies were compensated. At a minimum, ESG assessors should not tie fees to ratings, and further, assessors should also be required to disclose their pricing practices and schedules and any subsequent changes thereto. In addition, an assessor’s reliance on a rated firm for other businesses as well as overlapping ownership stakes may also give rise to conflicts of interest. In response to these concerns, IOSCO recommends separating staff responsible for data/ratings products from those providing such ancillary services. 10
Conclusion
For the ESG movement to reach its desired scale, it needs investor participation. At the same time, for the ESG movement to reach its desired peaks, it needs to transcend the bounds of short-term, profit-oriented thinking. The very impetus of the ESG movement was the recognition that shareholder profit alone is an incomplete measure of firm value.
The strength of the ESG movement lies in its ability to encompass different views about why and how ESG activities of firms matter to different stakeholders. Some users care about investment returns and risk management, while others care about sustainability or compliance, among many other possibilities. The common thread among them is their interest in environmental, social, and governance factors. As varied as the views about ESG are, assessments of a firm’s ESG activities will also necessarily diverge.
The current trend in the ESG literature calls for ESG assessments to converge, specifically toward the financial view of ESG. While the financial view of ESG may help persuade investors to turn their attention to other stakeholder interests, the idea that ESG matters only if it can be translated into financial gain undermines one of the underlying goals of ESG, which is to move away from short-term, profit-oriented thinking toward long-term sustainability and resilience.
Sung Eun (Summer) Kim is a Professor of Law at the University of California, Irvine.
This post is adapted from her paper, “The Duality of Variance Among ESG Assessments”, which was recently published in the Missouri Law Review and is available on SSRN.
