2017 Conference - Session 1

Session 1 - Thursday, March 16, 9:00 - 10:30am

A.1: Valuing Mortality Risk Reductions in Low- and Middle-Income Countries: Addressing Data Gaps and Inconsistencies (Roundtable)

Chair: Lisa A. Robinson, Harvard University

In benefit-cost analysis, reductions in mortality risks are typically valued using estimates of the value per statistical life (VSL). These estimates are derived from studies of individuals’ willingness to pay for small changes in their own mortality risks. However, few such studies address low- and middle-income countries, raising issues related to how to best estimate VSL in these contexts. While some issues are positive and could be resolved through more primary research, others are normative and require applying value judgments. This panel brings together several researchers who have developed approaches for estimating VSL in these countries. Each will first discuss the approach they recommend and the rationale for the approach, including how they have adjusted for factors such as income and age or life expectancy as well as how they have addressed concerns related to estimating social rather than individual values. We will then discuss options for harmonizing these approaches as well as priorities for future research.


Nils Axel Braathen, OECD
Maureen Cropper, University of Maryland
James K. Hammitt, Harvard University
Alan Krupnick, Resources for the Future
Urvashi Narain, World Bank
W. Kip Viscusi, Vanderbilt University

B.1: Estimating Economic Values: Methods and Applications

Chair: Frits Bos, CPB Netherlands Bureau for Economic Policy Analysis


1. Sensitivity to Scope in Contingent Valuation and Discrete Choice Experiments: Results Based on Valuing Health Risk Reductions; Henrik Andersson*, Toulouse School of Economics

There is a large stated preference literature estimating willingness to pay (WTP) for health risk reductions using the contingent valuation (CVM) approach, and more recently often based on the discrete choice experiment (DCE) approach. Irrespective of method, these studies often fail to show adequate sensitivity to scope, i.e. WTP does not increase as the quantity of the good or the number of goods increases. In this paper we compare the sensitivity to scope with the CVM and DCE approach based on respondents’ WTP for mortality and morbidity risk reductions. We analyze scope sensitivity using between-subject tests, which is a novelty in the DCE setting. The results show that we can reject adequate sensitivity to scope in both the CVM and DCE design, and the degrees of bias and welfare estimates are very similar in the two approaches. Thus, using a more stringent scope sensitivity test than the standard approach in the DCE literature indicates that sensitivity to scope is an equal pressing issue in DCE as well as in CVM studies.

2. Hedonic Vices: Fixing Inferences about Willingness-to-Pay in Recent House-Value Studies; John Yinger*, Syracuse University

A key tool for studying the demand for neighborhood amenities and estimating the benefits from amenity improvements is a regression of house value on amenity levels, controlling for housing characteristics. Several scholars have developed methods to address the methodological challenges, such as endogeneity, faced by these “hedonic” regressions. Unfortunately, however, some recent studies neglect basic principles of hedonic estimation in Rosen (1974). After providing conceptual background, this article explains these hedonic “vices” and how to avoid them. We focus on inappropriate functional forms, inappropriate control variables, and misinterpretation of hedonic regression results. Our analysis is supported using data from the Cleveland area in 2000 and a simulation model.

3. Including the Perspective of the Incarcerated Person in Jail CBAs; Chris Mai*, Vera Institute of Justice

Most cost-benefit analyses of incarceration focus on crime victims and taxpayers, ignoring the perspective of incarcerated people. Yet jail incarceration clearly causes harm, and an analysis that ignores this perspective does not capture the full cost of incarceration. This session will introduce an original framework to calculate the human cost of jail or prison incarceration which can be used in benefit-cost analyses of jail and prison policy. Using the extant cost of crime literature and economic data, this analysis models the costs of jail incarceration to the incarcerated individual. These costs include lost liberty, lost wage earnings, fees paid during incarceration, the cost of substitute child care, and the harm–translated into financial cost–of the additional risk of sexual and physical assault and suicide while incarcerated. The session will discuss a methodology to average the cost of harm over a diverse population. It will also investigate several additional harms of incarceration that do not yet have reliable shadow prices.

This research is based on a Vera Institute of Justice project to analyze the true cost of criminal justice fines and fees and financial bail practices in New Orleans. Combined with the taxpayer costs to run the jail, the human harm of jail incarceration far exceeds the revenue collected from fines and fees. While the estimates from this project are specific to New Orleans, this session will provide a framework for calculating the harm of jail that can be applied to any jurisdiction. 

C.1: Non-Market Valuation of Environmental Goods Using Revealed and Stated Preference Methods

Chair: Steve Newbold, U.S. Environmental Protection Agency

This panel of presentations focuses on non-market valuation of environmental goods using either revealed or stated preference methods. Topics include: water quality, offshore wind projects, environmental quality of lakes and bird biodiversity. The first paper focuses on using stated preference data to look at a general approach to examine sensitivity to scope and test the adding-up condition using choice-experiment stated preference data. The second paper measures the effect of the presence offshore wind projects on beach goers on the east coast of the United States using a travel cost model combining stated and revealed preference data from an internet-based survey of residents from 20 east-coast states. The third paper studies the theoretical and empirical foundations for the use of social media data (FLICKR) as a novel means to measure the welfare effects of changes in environmental quality of lakes to whether the non-random generation of these data rules out their meaningful use for welfare estimation. The last paper analyzes the preferences of bird watchers who are involved in the citizen science project called eBird, run by the Cornell Ornithology lab. The trip data for each member is combined with an external survey to analyze how the characteristics of the birder affect their preferences for site attributes.


1. External Scope and Adding-Up Tests for Stated Preference Choice-Experiment Surveys; Chris Moore*, U.S. Environmental Protection Agency

We propose a flexible approach to test the adding-up condition using choice experiment stated preference data. The approach involves estimating a model that is sufficiently general that it would comply with the adding-up condition only if certain parameter restrictions are not rejected. If the parameter restrictions are rejected, then a willingness-to-pay function that does not comply with the adding-up condition would provide a better explanation of the survey responses than one that does. We illustrate the approach using a series of numerical experiments and sensitivity tests using simulated data. We also show how to examine the quantitative deviation from the adding-condition when the test of statistical significance is failed. We discuss some features of choice experiment survey design that influence the power of the proposed test, and make suggestions for the design of future surveys to allow stronger validity tests to be performed. In particular, this strategy is facilitated by a survey design that varies the baseline levels of environmental quality across two or more versions of the survey instrument. 

2. Measuring the Effects of Offshore Wind Projects on Beach Use and Tourism on the East Coast of the United States; George Parsons*, University of Delaware

We measure the effect of the presence offshore wind projects on beach goers on the east coast of the United States using a travel cost model combining stated and revealed preference data from an internet-based survey of residents from 20 east-coast states. The data were gathered by GfK (formerly Knowledge Networks) using their probabilistic-based, pre-screened KnowledgePanel. Respondents (n=2051) were shown photo simulations of hypothetical wind projects at distances ranging from 2.5 to 20 miles offshore and were asked if the projects would have affected their beach experience and/or caused them to change in their trips plans. In the context of a random utility model we predict changes in trip patterns and measure welfare effects. The east coast, in our case, includes beaches from states as far north as Massachusetts and as far south as South Carolina. We have 275 beaches in our model. In addition, we consider a number of auxiliary models to predict (i) effect of wind projects on enjoyment, (ii) likelihood of cancelling a trip, and (iii) likelihood of making a special trip to see a new wind project. The results of the models are used to simulate impacts on selected beaches in each state on the east coast.

3. The Use of Social Media Data for Nonmarket Valuation; Yongjie Ji*, Iowa State University

This paper studies the theoretical and empirical foundations for the use of social media data as a novel means to measure the welfare effects of changes in environmental quality. A key question for non-market valuation is whether the non-random generation of these data rules out their meaningful use for welfare estimation. Several outcomes are possible. First, it may be that analysts will be satisfied that the preferences of some subgroups of the population can be adequately represented through the use of a social media generated sample, but that it will not be possible to make inferences about the values of all households in the general population. The sub population for which inference is possible may be quite limited. Alternatively, with additional structural assumptions and/or comparisons to external data sources, it may be possible to conclude that preferences of a broader set of users can reasonably be captured. In this paper, we address these questions in the context of recreation demand modeling for the specific case of FLICKR data associated with 135 major lakes in Iowa. We focus on this set of FLICKR data because we have an independently collected random household sample to these same lakes, providing a unique opportunity for comparison between a social media data set subject to numerous selection problems and a sample that is relatively free from such concerns. In addition to standard visitation and socioeconomic data, information on whether the household used FLICKR or other social media was also elicited from the random household survey. This information provides another unique comparator on which to assess the representativeness of the FLICKR sample.

4. The Value of Bird Biodiversity to eBirders: Exploring Citizen Science Data using a Recreational Site Choice Model; Sonja Kolstoe*, Salisbury University

Environmental citizen science projects provide the researcher with access to unique data that would not otherwise exist. These projects engage people to report their natural-resource-related activities in real time, for example, their bird watching trips. In this study, we analyze the preferences of bird watchers who are involved in the citizen science project called eBird, run by the Cornell Ornithology lab. We know the member’s home address and their birding destination alternatives. This allows us to set up a random utility model (RUM) of site choice based on site attributes which can be used to generate willingness to pay (WTP) estimates. We explore both fixed and random parameter specifications with these unbalanced panel data on recreational choices. We build on an existing analytical framework we developed in earlier papers using a smaller data set. We now have a much larger and different sample of eBird members and their birding trips than used for these prior studies. We combine these trips with information learned from a separate survey distributed to eBird members in Oregon and Washington, to collect more variables that allow us to address questions our earlier work did not, including the frequency with which eBirders report their trips to eBird. Armed with this additional information, we unpack the preference of eBirders, to include understanding the preferences of those who travel and those who do not. Additional questions we address include: 1) what is the appropriate site-choice consideration set; 2) how does a birder's perceived ability level affect their preferences for site attributes; 3) how does being a competitive birder (also known as a "lister" ), versus a casual birder, affect their preferences for site attributes, etc.

D.1: Strengthening Benefit-Cost Analysis of Early Childhood Interventions Through High-Quality, High-Utility Cost Analysis

Chair: Jonathan Belford, Child Trends

Benefit-cost analyses (BCA) documenting the investments needed to implement effective social programs and the returns that can be anticipated are increasingly sought to inform the allocation of scarce public and private resources. Yet a 2016 consensus study, Advancing the Power of Economic Information to Inform Investments in Children Youth, and Families, by the National Academies of Science, Engineering, and Medicine concluded that the utility of BCA and economic evaluation more broadly is hindered when analyses do not follow best practices or adequately consider the context in which findings will be used. In support of high-quality, high-utility economic evaluation, this panel will present cost analyses of three early childhood interventions targeting families and children at increased risk for behavioral health problems: (a) REDI, an enhanced version of Head Start designed to increase school readiness by developing social-emotional and emergent literacy skills, (b) the Family Check-Up, a family-oriented intervention that addresses child and adolescent adjustment problems, and (c) Promoting First Relationships, a brief home-visiting intervention for strengthening parent-child relationships and reducing child welfare system involvement. The papers illustrate how attention to quality methods and stakeholder context strengthens social program cost estimates, the foundation for subsequent benefit-cost analyses, and the utility of standalone cost information. They also show how decisions made in the research design, data collection, analysis, and reporting phases have implications for the utility of cost analysis findings. Together the three papers help provide a roadmap for increasing the impact of cost and benefit-cost analyses of social programs. After the three presentations, the discussant will comment on the papers from the perspective of best-practices and relevance to a broad set of stakeholders.

Discussant: Phaedra Corso, University of Georgia


1. Issues in Estimating Costs of Early Childhood Interventions: An Example from the REDI Intervention; Damon Jones*, Pennsylvania State University

Cost analyses of early childhood educational interventions can provide important information for improving efficiency of program delivery as well as setting the stage for considering the potential for return on investment. However, the nature of such efforts varies widely, given those leading the cost analyses in this field may be less familiar or experienced with best practices. Such differences limit the ability to make comparisons across different programs in order to inform policy. This research reviews the key issues in cost analysis approaches for interventions directed toward young children in preschool settings. We discuss how efforts to increase consistency, transparency, communication, and standards can help the field better leverage cost analysis findings going forward. We also identify a set of key elements that are especially salient for assessment of resources with these types of programs. To illustrate how attention to these elements increases both quality and value, we present a cost analysis of the REDI project—an enhanced version of Head Start designed to increase school readiness through broader development of social-emotional and emergent literacy skills. We examine per-family costs for children receiving a classroom only program versus children additionally receiving home-visiting services. Results indicate a per-child cost of $183 to deliver one year of the program in pre-school. Additional costs to implement the home visiting component were estimated as $2,823 per family for the same time frame. We present a range of costs based on a sensitivity analysis, and include a focus on areas expected to drive variation in total and per-family costs in different settings and timeframes (e.g., training costs, home visiting travel cost, coaching needs). Finally, we discuss implications from our cost estimates for the REDI program in general as well as plans for assessing REDI’s cost-effectiveness based on past indication of program effectiveness.

2. Dynamic Cost Analysis of Evidence-Based Family Services in a Randomized Controlled Trial; D. Max Crowley*, Pennsylvania State University

The rise of tailored interventions in a number of policy and practice environments requires thoughtful and detailed cost analyses that model the dynamics of resource flow across time at the individual level. The Family Check-up (FCU) is a family-oriented intervention that promotes family management, strengthens relationships between parents and children, and addresses child and adolescent adjustment problems in a manner that is brief, timely, and adapted to the specific needs of each child and family. The FCU has demonstrated significant reductions in substance abuse, delinquent behavior and developmental psychopathology across multiple randomized trials offered to indicated populations in diverse service settings (e.g., WIC families, schools, homes). In a comprehensive cost analysis of the FCU as implemented in the Early Steps trial involving 731 WIC-eligible families with children between the ages of 2 and 5 at three geographically distinct sites, we explore the resources needed to invest in this effective family-centered intervention. Our analysis benefits from individual-level time tracking of intervention resources deployed to each family, which clinicians maintained throughout the intervention. In addition to estimating the total ($621,045 in constant 2015 dollars for 4 years of intervention with 367 families randomly assigned to receive the FCU, SD across sites = $40,168), average ($1,692 per family for 4 years, SD = $64), and marginal ($759 per family for 4 years, SD = $85) costs of the FCU intervention, we describe time spent and associated costs of key intervention components (e.g., training, contact with families, ongoing support and technical assistance, time spent driving to family homes). This uniquely detailed dataset allows us to explore the variation in family resource needs and related costs across time and across different intervention sites. Key cost drivers will be identified. Implications for the development of an optimized FCU model will be discussed.

3. Efficient and Effective Early Childhood Home Visiting Programs for High Risk Families: A Cost Analysis of Promoting First Relationships; Margaret Kuklinski*, University of Washington

High-quality cost analyses provide a crucial foundation for return on investment analyses of social programs targeting children and families, yet they also can guide the development of more efficient, effective interventions. Home visiting programs for parents of young children at elevated risk for behavioral health problems provide a case in point. Though they can be cost-effective, they also are quite costly per family to the public and private funders of these services, which may limit their reach. Alternative approaches that improve outcomes at reduced cost could increase overall public health impact. In this paper, we present a cost analysis of Promoting First Relationships (PFR), a 10-week, manualized home visiting program for families with children ages 0 to 3 at increased risk of child welfare system (CWS) involvement and removal from the home. In contrast to more resource intensive approaches, PFR assumes that brief attachment-based intervention at a time of heightened family need can have an enduring positive impact on the emotional bond between parents and children, children’s developmental outcomes, and system impacts including CWS involvement. Findings from a randomized controlled trial involving a high-risk sample of 247 parents and their 10-24 month old children randomly assigned to PFR or an alternative resource and referral condition showed significant improvements in social-emotional outcomes and out-of-home placements in PFR compared to control families. Effects were sustained for 18-months post-intervention, the latest funded follow-up. This presentation highlights the total and per family average (preliminary point estimates: $190,539 total, $1,537 per family), marginal, and incremental costs of delivering PFR. It also shows how the utility of cost analyses is enhanced when costs are disaggregated by key intervention components, resource use, unit price, and cost information are delineated separately, possible sources of efficiency gains are identified, and findings are communicated in a transparent but credible manner.

E.1: Estimating Benefits and Costs of Risk Interventions in Face of Uncertainty

Chair: Elisabeth Gilmore, Clark University

Discussant: Laura Stanley, U.S. Environmental Protection Agency


1. Uncertainty Evaluation in DOT and EPA RIAs of Lifesaving Regulations; David Good* and Kerry Krutilla, Indiana University

Benefits and costs are naturally subject to several uncertainties. On the theoretical level, these can be incorporated into benefit cost analysis using a variety of mechanisms, e.g., Monte Carlo analyses, bounding analysis, etc. In our paper we compare the treatment of uncertainty in the regulatory impact analyses (RIAs) for two Federal agencies: the US Department of Transportation and the US Environmental Protection Agency. Our attention focuses on one of the most contentious areas in the impact analysis, the lifesaving potential of these regulations. With the central value of VSL approaching 10 million per life, the value of lives saved tend to overwhelm other benefits. Together, these two agencies comprise the majority of Federal lifesaving regulations over the four year period of our study from 2011 through 2014, a total of 18 regulations. We eliminate from our analysis studies whose primary purpose is other than saving lives such as Corporate Average Fuel Efficiency standards. 

Unfortunately, often uncertainty analysis focuses on the aspects of the problem where it can be most easily applied, rather than the aspects of the problem that would be most illuminated by its use. For example, we recognize the importance of uncertainty in VSL, but note that it affects all regulations equally. Instead, we diagnose several sources of uncertainty for each of the regulations in the study, sometimes using one regulation as a better practice than others. Often the primary source of uncertainty is with the input-output process (i.e., translating actions into lives saved). For each of our regulations, we develop a mechanism for qualitative, and in cases quantitative, assessment of how much the performed uncertainty analysis in the RIA has captured, and how much it has missed.

2. Risks and Benefits of Reducing Ozone Exposure after Re-Evaluating Most Recent Chamber Study Data; R. Jeffrey Lewis*, ExxonMobil Biomedical Sciences, Inc.; and Richard Belzer, Regulatory Checkbook

Our previous work has shown that conventional spirometric methods understate maximum forced expiratory volume in 1 second (FEV1) by about 7% (interquartile range: -7.9% to -6.4%). This measurement error or bias exceeds the consensus threshold for within-day differences generally considered biologically or clinically significant (>5%). Thus, some reported FEV1 differences are interpreted as significant but actually are test protocol artifacts.

Conventional methods also fail to account for within-person inter- and intra-test variability. That is, they treat each test values as a fixed result instead of as a realization of random variable with known (or estimable) properties. Differences in FEV1 across tests are thus incorrectly assumed to be real when they may in fact be statistically indistinguishable.

The National Ambient Air Quality Standard (NAAQS) for ozone is based in part on chamber studies in which volunteers are exposed under varying conditions. These studies follow the conventional spirometric protocol. Reported comparisons across exposure conditions do not account for inter- or intra-test variability and thus are statistically unreliable.

This presentation adds to this body of knowledge by disclosing alternative estimates derived from the most recent chamber study modified by us to incorporate inter- and intra-test variability. Results are especially sensitive to within-person intra-test variability. It is shown that under robust conditions differences reported as statistically significant are actually well within the range of test variability. Reported statistical inferences concerning small differences in ozone exposure are thus unreliable. This has implications for published estimates of the health benefits likely to result from reducing the ozone NAAQS. Developing statistically valid inferences requires first obtaining reliable and objective estimates of the intra-test coefficient of variation for the population and for numerous subpopulations of interest. 

3. Adaptive Benefit-Cost Analysis for Changing a Few of Many Causes; Tony Cox*, Cox Associates

Identifying which action or policy from a small set of alternatives creates the greatest net social benefit is a key task for benefit-cost analysis (BCA), but this criterion must be refined when benefits and costs are so uncertain, e.g., due to the effects of uncontrolled factors, at the time a choice must be made (i.e., ex ante) that future information is likely to change the comparison of the alternatives (ex interim or ex post). Under these conditions, different stake-holders with access to different information may also disagree in their assessments of the probability distributions for costs and benefits, and hence disagree about the best alternative to embark on even if they have the same preferences for outcomes. Although stochastic dynamic programming can in principle solve such dynamic decision problems under uncertainty for a single decision-maker, no analogously complete and conceptually satisfactory framework exists for multiple stakeholders with divergent preferences for the initial choices to be taken. To overcome this challenge, we propose to apply techniques developed in machine-learning and artificial intelligence to help cooperating agents learn to improve their collective performance over time in an uncertain, changing environment. We synthesize key ideas including actor-critic algorithms for multiagent reinforcement learning, cooperative planning architectures and algorithms, and methods for adaptively estimating net benefits for different actions and policies while hedging against the possibility that initial estimates could be mistaken and misleading. We illustrate the practical value of these techniques for adaptive BCA and decision-making under substantial uncertainty with tow practical applications: adaptive regulation of air pollution when the public health effects of alternative regulations are uncertain, and deciding whether to ban animal antibiotics when the effects of a ban on human health and on the evolution of multi-resistant “superbugs” are initially highly uncertain.

F.1: Incorporating Economic Evaluation into the Portfolio of Private Foundations (Roundtable)

Chair: Lynn A. Karoly, RAND Corporation

The growing demand for evidence-based policy applies to both the public and private sectors. The aim of this session is to bring together representatives from private foundations for a roundtable discussion of the various ways that cost analysis, cost-effectiveness analysis, benefit-cost analysis, and related methods currently shape the philanthropic agenda and how that may evolve over time. This may take the form of (1) supporting the inclusion of economic evaluation as part of research funded to do program impact evaluation, (2) using anticipated return-on-investment (ROI) calculations to determine which programs to invest in, (3) drawing on these methods as part of pay-for-performance initiatives, (4) incorporating findings from economic evaluations into advocacy efforts, as well as other applications. Members of the panel will describe the ways that their foundation incorporates economic evaluation methods, the challenges faced in doing so, broader lessons for philanthropy, and the ways that research can support these efforts.


Thomas Forissier, Bill and Melinda Gates Foundation
Cindy Esposito Lamy, Robin Hood Foundation
Woody McCutchen, Edna McConnell Clark Foundation
Jon Baron, Laura and John Arnold Foundation