We saw a potential game changer in behavioral demand-side management (DSM) evolve over the last few years: the online audit.
The audit is the unsung hero of behavioral programs—silently contributing savings to DSM programs by bringing customers from email home energy reports with a single click, offering customers insights on their energy use, and leading them to the single next best action they can take to save energy.
But the online audit can do so much more.
In late 2017, DNV-GL released a study on the impacts of online audit completions, finding that PG&E customers who completed the Opower online audit saved 1.2% more electricity and 1.5% more gas than controls.
Measuring the impact of opt-in actions (like a customer choosing to complete an online audit) can be difficult because audits are not typically set up as randomized controlled trials (RCTs). In their study, DNV-GL used a matched control group approach, a popular non-experimental method of inferring impact. In simple terms, they looked at the customer households that completed the audit, then identified a second set of households that didn’t but were similar to the first group in terms of historical energy usage and demographic characteristics. They used this second group as a matched (albeit non-randomized) control group and then calculated the energy usage difference between these two groups to find the effect of the audit on energy usage.
With this data in hand, our team set to work last year applying a comparable method to our client data. We found that, across clients, the variation in energy usage and other customer characteristics was too great to isolate audit savings from overall behavioral program savings with confidence. Our team arrived at the conclusion that we need RCT-measured results in order to forecast the incremental savings impact of more customers completing the online audit.
And we can do just that. After all, were working with some of the largest behavioral DSM programs in the world. So, we’ve got the data to make that happen. We actually could directly measure the impact of the audit using an RCT, either by temporarily hiding the audit on the website for a randomized set of customers or by randomizing the recipients of an email promotion of the audit.
We see a ripe experimental opportunity here, and we’re actively working with our clients to make the opportunity a reality. Those results could allow us to far more confidently estimate just how much energy customers save when they complete online audits.
This article is part of our Beyond Paper Opower blog series. In our next post, we’ll wrap up this series with some thoughts on what all this means and where it’s going.
Read the rest of the posts in this blog series:
Editor’s notes: (1.) Lead art features the authors Sanem Kabaca, Data Scientist; Ryan Irwin, Opower Group Product Manager; and Matt Frades, Senior Manager, Analytics with variations of the Opower mascot Pluggie. (2.) In 2016, Opower joined Oracle Utilities. Learn more here.