Evaluation, Measurement and Verification for ESRPP
Why Was ESRPP Developed?
Traditional widget-based rebate programs have been the mainstay of utility energy efficiency portfolios for decades, but these mature programs are running into headwinds as efficiency baselines increase and cost-effectiveness declines.
In response, ESRPP was developed to move away from the traditional approach of heavy reliance on a patchwork of non-standardized, oftentimes complex utility programs. Instead, ESRPP sponsors work in coordination and directly with participating retailers to transform fragmented markets across the U.S. into a cohesive national marketplace that puts an emphasis on more efficient product choices and increasing stocks of ENERGY STAR certified products.
What Challenge Is ESRPP Facing?
ESRPP is a new program model that seeks to transform entire markets instead of focusing on an individual consumer’s behavior. Because ESRPP operates under a market transformation model that is fundamentally different than traditional consumer rebate programs, a tailored approach to evaluating program results is required.
Given that the ESRPP introduces a market transformation program approach in areas that still run traditional consumer rebate programs, the evaluation community recognizes the need for evaluation methods to be developed that are appropriate to the program theory underlying ESRPP; however, they also report that they will need more time and experience with ESRPP before they can arrive at consensus on a prescriptive evaluation methodology for programs like ESRPP.
What Policies and Actions Could be Advanced to Accelerate Market Transformation Based Efficiency Programs?
National coordination with retailers through ESRPP offers the efficiency community the opportunity to increase cost effective energy efficiency going forward. Early indicators are positive. In little more than a year of ESRPP program experience, participating retailers are reporting actual increases in gross sales of qualified ENERGY STAR products, which validates the program model.
Additional results from initial program sponsor evaluations include:
- Increased sales for program-qualified models leading to short-term savings in certain product categories
- Initial evidence that ESRPP has begun to influence the market in a manner consistent with the near-term program theory for ESRPP. For example, the outputs and activities documented in ESRPP logic models, such as increased collaboration, ability to place in-store signage, and program activity decisions being based on program data, are occurring.
- Cost-effectiveness research, while requiring additional evaluation to account for market transformation impacts, shows significant prospects for most product categories to be cost effective.
Despite the bright promise of this market transformation program, the slow pace of evolution of evaluation techniques may undermine the progress and growth of ESRPP. While we await the evolution of evaluation techniques to catch up with the emerging ESRPP program model, current program sponsors whose investments launched ESRPP are left uncertain as to whether they will be able to receive credit for the savings the program promises to create. Continued uncertainty presents challenges that could threaten the ability of ESRPP program sponsors and prospective new program sponsors to participate. If participation remains limited, national retailers who are counting on program sponsors’ investment to justify their own investment may revisit their commitment to this national platform for market transformation.
State policies and actions can play an important role today by addressing these issues. Here are three steps policymakers and regulators might consider taking right away.
- Evolve Evaluation: Consider Using SEEAction’s Guidance When Developing Evaluation Criteria for ESRPP and Have Staff Participate in ESRPP Conference Calls to Learn Even More—Evaluation techniques and protocols designed to evaluate widget-based rebate programs are ill-fitted for evaluating ESRPP. This market transformation program model requires a new approach to evaluation. To address this need, the State and Local Government Energy Efficiency Action Network (SEEAction) convened evaluators from across the country to participate in the SEEAction Evaluation, Measurement and Verification working group and provide input into guidelines and key considerations for designing an appropriate evaluation of ESRPP. These guidelines were published in a report entitled ENERGY STAR® Retail Products Platform (RPP): Conditions and Considerations in Evaluating Market Transformation Programs and Evaluation Guidance for RPP.[i]
- Provide Program Sponsors with the Certainty Needed to Support Continuation/Expansion of ESRPP—There are many ways to increase certainty and each jurisdiction’s regulatory bodies should, of course, arrive at their own conclusions for appropriate ways to address this issue. Some options for consideration are:
- Guidance to Evaluators—The long-term fix for uncertainty related to the need to evolve evaluation is for the evaluation development process to be allowed to take place and run its course to an appropriate resolution.
- Protect the Program—Since market transformation takes time, early years are expected to have lower savings and higher costs, while later years deliver higher savings at lower costs. As a result, first year savings and similar tests do not accurately measure progress for market transformation-based programs. During the period of time before evaluation uncertainty is resolved, regulators might adopt measures and policies that temporarily protect the ESRPP platform from the near-term cost effectiveness policy pressures placed on traditional and more mature programs. This could be done any number of ways, including treating the program as a pilot or as a non-resource program, thereby exempting it from cost effectiveness standards until evaluation issues are resolved.
- Provide a Bridge to Measured Savings—Another potential approach for temporary protection of the ESRPP program is to adopt very conservative assumptions about the level of net savings being created by ESRPP in the short-term and use those assumptions to provide a base level of deemed near-term savings credit to the program sponsors which can be adjusted upwards if later-completed evaluation studies support that result. ESRPP’s unique ability to provide actual, full-category sales data provides a reasonable near-term basis for making temporary conservative estimates of program impacts until such time as a body of evaluation results can replace them.
- Ensure that Cost Effectiveness Tests Appropriately Address the Nature of ESRPP’s Incentive Strategy & Market Transformation Approach—Currently, almost all jurisdictions would apply different treatment to an incentive provided to an individual consumer under a traditional rebate program (i.e., treat it as a pass-through) and an incentive provided to retailers to “push” sales of ENERGY STAR products under ESRPP (i.e., treat it as a program cost). Treating ESRPP incentives provided to retailers as a program cost sends a false signal about the cost-effectiveness of the program. Similarly, ignoring mid-term and long-term savings impacts by assessing ESRPP solely on short-term impacts creates a counterproductive bias against investment in longer-term, more cost-effective and permanent savings from market transformation approaches.
Contact EPA for more information about EM&V for ESRPP.
Updated July 23, 2019
[i] See, ENERGY STAR® Retail Products Platform (RPP): Conditions and Considerations in Evaluating Market Transformation Programs and Evaluation Guidance for RPP (SEE Action Measurement, Evaluation and Verification Working Group, January 2018)