One of the most important but often overlooked components of a program implementation cycle is learning from the program’s performance. Building an evaluation mechanism from the very beginning of a program lifecycle allows administrators to truly understand overall performance, pivoting as needed during implementation as well as improving future design after the close of a program.
Implementers and administrators have finite resources for program implementation and program wind down. However, as tempting as it may be to transition all resources on to the next project, program, or portfolio, a full evaluation uncovers valuable, actionable insights for use in future program designs.
Monitoring Outcomes During Program
To perform a program evaluation, there must first be a mechanism in place to track and report upon the journey toward program goals. An evaluation plan established early on in a program, during goal setting and portfolio development, can help program administrators visualize program performance throughout the entire cycle.
Any goals and targets, even those established at the beginning of a program, are not required to stay static. At some point during a portfolio’s progress, new requirements may be established, or a program activity may veer off course. Perhaps a program activity doesn’t demonstrate the intended outcome, or what was once emerging technology instead becomes standard practice in which case the measure is no longer eligible for program incentives. Understanding why a specific goal was not met in the initial program design can inform current (or future) program designs.
Many portfolio administrators and regulators do not plan for program flexibility and changes. However, with an early-established evaluation mechanism, program managers can identify potential solutions for any gaps in performance, such as specific market activities such as targeted solicitation, or adopting changes to the program, implementation strategy, or incentive.
For example, a utility rolling out a residential energy efficiency program may find during an evaluation that it only reached large multi-family complexes with budget to spend on projects, and omitted smaller, low-income apartment complexes. Understanding this trend could trigger a reevaluation of program implementation, so that future redesigned programs can reach the omitted customer and achieve portfolio goals. Potential solutions could be identified, such as developing a social assistance solicitation, or creating a subprogram for an incremental incentive, direct install approach, or financing solution. Equally important, these metrics also may point to new program issues that cannot be circumvented. In this case, it provides important insight to the implementer to potentially halt program offerings and redirect resources.
Identifying these problems and being able to dive in to find a solution, is a critical role for the portfolio administrator. This begins with having the ability to identify gaps early and the flexibility to adjust a program when needed, using milestones and metric monitoring. An early effort to track program activity outcomes, with adaptability within the portfolio, can help better meet portfolio budgets or goals.
Building Evaluations into Program Wind Down
Every program will ramp down eventually, when the contract term is completed or the program cycle is ended. Many program implementers will provide a final report and then use this ramp down to shift resources (such as staff hours) into the next program cycle or project.
However, the end of a program is the best time for an evaluation of the program’s performance. Resources and time should be allocated in advance for a dedicated review during the close down of a program, to ensure projects have met goals and inform program design in the future.
Long before a program winds down, the program implementer should have a plan for the evaluation, including meeting with stakeholders and reviewing expectations and feedback. A program implementer’s evaluation priorities will be specific to the customer, such as optimizing energy efficiency savings and incentive payments.
For a portfolio administrator, the evaluation priorities will cover the opportunities for different sectors, for example residential or industrial customers. The critical task is ensuring that customers receive the correct amount of funding for each segment to maximize energy efficiency, and understanding how regulations and program designs can increase the impact of specific programs.
To complete these priorities, before winding down a program, all stakeholders should have a strategy in place to learn from program performance.
Completing Project Evaluation
As discussed, metrics established during the portfolio planning stage encourage program and portfolio progress tracking against goals during the entire program implementation cycle. For an energy efficiency program, key metrics may include tracking savings achieved, on-bill savings, cost effectiveness, number of customers served in different segments, and more metrics around various benefits.
Key metrics should be consistently forecasted and reported to actual goals, quarterly and annually. From the gaps, a program implementer can compile a list of barriers and start proposing solutions to share with the portfolio administrator.
For example, program managers can use evaluation data to identify areas which measures had the smallest market penetration, which might indicate the need to refine incentives and market strategies. Or, a portfolio administrator may be able to spot which projects had the greatest impact on goals and cost-effectiveness, offering insight on how to develop more impactful projects or transform less impactful projects into significant contributors.
As energy efficiency program regulations change over time, program implementers may also consider expanding the scope of data tracked. In California, for instance, energy efficiency program implementers are now required to report on cost effectiveness through the Total System Benefit (TSB), which combines multiple benefits such as optimizing energy and peak demand savings goals and greenhouse gas benefits of energy efficiency into one key metric. Having more data in hand can be helpful for program managers and portfolio administrators to adapt to these changes and still be able to compare program performance over time, even as metrics evolve.
Informing Future Program Success
Every portfolio contains years of insight, and a portfolio’s success or failure can guide future program development. To identify solutions that will contribute to future portfolio budgets and goals, this evaluation needs to be carried out during implementation and immediately upon closing.
The program implementer can improve program design for a specific audience, and administrators and portfolio managers can refine fund allocation or designs that allow specific programs to be more impactful. For administrators, evaluating and reporting upon completed program findings can help prepare future portfolios for success.
A program’s end is a good time to present findings to management, discuss pitfalls and lessons learned, and determine how this insight can be integrated into an organization’s overall objectives. These lessons can make future program offerings more effective, but only if they are understood across the entire organization.
An experienced program implementer like Lincus brings a background of technical knowledge as well as lessons learned across various programs and stakeholder relationships. Lincus’ program management experience spans from initial program development all the way through evaluation and ramp down, with an ability to adapt and streamline solutions for energy efficiency programs every step of the way.
For help in reaching your portfolio goals, contact Lincus today.