On Balance: Using Congressional Hearings to Identify Policy Impacts

July 10, 2019

By: Deven Carlson

The field of benefit-cost analysis is replete with guidance on theoretical and empirical valuation of impacts. Many fewer resources, however, are available for analysts to consult when determining the set of policy impacts to value in the first place. This post describes a new article published with co-authors Joseph Ripberger, Wesley Wehde, Hank Jenkins-Smith, Carol Silva, Kuhika Gupta, Benjamin Jones, and Robert Berrens, in the Spring 2019 Issue of the Journal of Benefit-Cost Analysis. The article, entitled, ‘Benefit-Cost Analysis, Policy Impacts, and Congressional Hearings,’ proposes a method for identifying policy impacts that contributes a degree of systematism, transparency, and replicability to the process.

Our proposed method for identifying policy impacts relies upon information provided during Congressional hearings, which are the most systematic and prominent method that the U.S. Congress uses to acquire and process policy-relevant information. And even though there is evidence that hearings have become more partisan over time, research also shows that the range of policy impacts discussed in hearings has actually increased during this period. In highlighting the value of Congressional hearings, we are not discounting the value of other information sources—such as comments on proposed regulations, stakeholder input, and others—for analysts working to identify the full range of policy impacts.

Leading textbooks on benefit-cost analysis instruct practitioners to identify all policy impacts for inclusion in the analysis, but offer little practical guidance for systematically doing so. Rather, they offer generalities such as “think broadly about possible costs and benefits” (Cellini & Kee 2015), or “identify a cause-and-effect relationship between some physical outcome of the project and the utility of human beings with standing” (Boardman et al. 2011). And, at least in the United States, federal guidance documents typically note the importance of including all valued policy impacts in the benefit-cost analysis, but are often vague and ultimately provide little meaningful guidance.

Not surprisingly, in practice, applied benefit-cost analyses are typically very clear about their approach to valuing policy impacts, but silent on the approach used to identify the set of valued impacts. For example, our review of the 24 applied benefit-cost analyses published in the Journal of Benefit-Cost Analysis since its inception, we only identified 4 that engaged this issue in some form. Of course, this sample is not representative of the broader universe of benefit-cost analyses that were conducted over this time period, but it certainly suggests a relative lack of attention to this important issue.

The article proposes a six-step approach to identifying important societal impacts that should be included in the benefit-cost analysis. The process draws on the witness statements provided during Congressional hearings on the particular policy topic of interest and analyzes their relevance to the specific policy under consideration. The approach, which we address in more detail in the article, consists of the following steps:

  1. Identify the set of hearings potentially relevant to the policy
  2. Obtain hearing transcripts and written witness statements for all relevant hearings
  3. Develop an impact classification scheme using an iterative process
  4. Classify transcripts and witness statements using scheme developed in Step 3
  5. Compile results into a dataset and describe the resulting categories
  6. Determine whether each impact in the dataset will manifest in the specific policy serving as the subject of the benefit-cost analysis.

We acknowledge that this approach does not eliminate subjectivity from the process of identifying benefit and cost categories. There is subjectivity in determining the set of relevant hearings. Developing an impact classification scheme is an inherently subjective endeavor. We address this subjectivity by being as transparent as possible about our decisions throughout the process. For example, we list the keywords we use to identify the set of relevant hearings. We detail the contents of our impact classification scheme. Such transparency allows for potential replication as well as constructive debate over the set of identified impacts.

The potential utility of this approach can be illustrated for a proposal to alter operations of the Glen Canyon Dam. After identifying 34 Congressional hearings relevant to this proposal, we created a sentence-level dataset containing 25,916 sentences from 409 statements made by 344 unique witnesses. We then developed an impact classification scheme, which consisted of 26 specific impact categories nested within 6 broader categories. Each sentence was then classified according to that scheme. We provide a comprehensive description of this process in the article.

The figure illustrates the results of the classification, giving the proportion of sentences with any substantive content classified into each impact category. It identifies 26 separate impacts that should be considered for inclusion in a benefit-cost analysis for a policy altering operations of the Glen Canyon Dam. Analysts then need to draw upon content-area expertise—either their own or that of others—to determine how, if at all, each of these impacts will manifest in the specific context under consideration.

Although undeniably resource intensive, this application demonstrates the promise of drawing on Congressional hearings to identify policy impacts for valuation in a benefit-cost analysis. In addition to identifying a wide range of potential impacts, it illustrates how the process can be conducted in a systematic, transparent, and replicable manner. More generally, we view our paper as a first step in an effort to develop a set of resources that analysts can draw upon when faced with the task of identifying the full set of policy impacts, with the end goal being a literature on impact identification with as much breadth and depth as the literature on impact valuation.

Deven Carlson is Presidential Research Professor, Associate Professor of Political Science, and Associate Director at the National Institute for Risk and Resilience at the University of Oklahoma. His research examines the role of evidence in the policymaking process, with a particular focus on education policy.

References

  • Boardman, Anthony E., David H. Greenberg, Aidan R. Vining, and David L. Weimer. 2011. Cost-Benefit Analysis: Concepts and Practice. Vol. 4. Upper Saddle River, NJ: Prentice Hall.
  • Cellini, Stephanie Riegg and Kee, James Edwin. 2015. “Chapter 24: Cost-Effectiveness and Cost-Benefit Analysis.” In Handbook of Practical Program Evaluation: 636–672. Hoboken: John Wiley & Sons.