Where now with benchmarking?

In broad terms, they rejected the benchmarking approach that the AER had adopted to assess efficient operating costs for the NSW and ACT distributors. This article examines the implications of the Tribunal’s decision.

The context

The AER is in a difficult position because it lacks the information on and understanding of the network businesses possessed by the businesses themselves. This information asymmetry means that the AER is less well placed than the businesses they regulate to determine whether an individual business is efficient. The regulator therefore has a choice:

• to expend greater effort on reducing that information asymmetry; and/or
• developing incentive structures that encourage efficiency improvements by governance and management (as opposed to regulatory impost), the benefits of which are shared between owners and customers in a manner that encourages continual improvement.

The information asymmetry of the regulators was starkly exposed in the regulatory decisions on NSPs around 2012, which were widely criticised for being too lenient.

The regulatory asset base (RAB) comprising the past investments by the NSPs is essentially fixed under Australia’s regulatory regime. That is, the AER cannot, except under some quite exceptional circumstances, prevent NSPs from earning an adequate return of and on these past investments. From a price and revenue perspective, then, the AER must focus on reducing inefficiency in operating costs and future capital expenditure.

The AER has developed repex (replacement capital expenditure) and augex (augmentation capital expenditure) models to help reduce information asymmetry in respect of capital investment. The RIN data initiative and the AER’s benchmarking of operating costs are similarly aimed at reducing information asymmetry.

The AER will be seeking to continue to rely upon benchmarking. It will do so because of its obligations under the NER to publish annual benchmarking reports and to take account of them in its decision making. But it should also do so because benchmarking is a useful tool for understanding performance. Given this, we can draw the following implications for the future use of benchmarking from the Tribunal’s reasoning.

The importance of data

The data used by the AER was inadequate for benchmarking. The AER will not be able to use benchmarking as a central component of its regulatory decision making until it has a sufficient duration of RIN data which is not based on estimates and which has been appropriately verified. We expect that this will be a core objective of the AER over the duration of the current regulatory period (once the NSW and ACT decisions have been remade). We expect the AER will place greater emphasis on ensuring DNSP compliance in assembling robust and reliable RIN data.

Broadening the database and benchmarking approaches

The AER’s use of international data in its operating costs benchmarking was heavily criticised in the Tribunal. Some of these were valid and accepted as such by the Tribunal. But benchmarking is fundamentally pragmatic. The benchmark models that are adopted are inevitably a compromise between the precision with which the production technologies that firms adopt to supply their services are represented and the availability and robustness of (i.e. cost of acquiring) data. Hence, while there is some merit in the Tribunal’s observation that benchmarking was determined by what was available rather than what was best, such criticism should not be taken too far; it is important not to ‘throw out the baby with the bathwater’.

All benchmarking has limitations; wherever benchmarking has been used as a pivotal component in regulatory decision making, opponents have usually been able to supply a cluster of expert reports that criticise the approach. The fault, in so far as there is fault, lies less in international benchmarking itself — although we are sure that more time and more consultation and collaboration between the AER and the DNSPs will improve on the AER’s first efforts — but in excessive reliance on the results to determine efficient (and hence allowable) operating costs.

We believe that expanding the database used for the benchmarking to include more international counterparts while addressing some of the more glaring shortcomings recognised by the Tribunal will add considerably to the utility of benchmarking. Indeed, we would have reservations if the AER were to rely on this type of benchmarking (parametric or index number approaches) based solely on Australian RIN data, given the small number of Australian DNSPs and their diversity.

In expanding the database beyond Australia, New Zealand and Ontario, the AER should consider adopting a broader range of approaches such as DEA (which is less reliant on financial measures, where differences in accounting conventions and cost classifications can have a large impact, but which can be sensitive to data outliers), alternative SFA formulations some of which were described in submissions to the Tribunal, total expenditure (totex) benchmarks and measures that assess in more detail the impact of legacy asset bases (which are in turn a function of characteristics such as size and density) on costs. It is not clear augmenting the database will address the inherent deficiencies in seeking to benchmark businesses that will be outliers in virtually any sample of DNSPs.

These different approaches have their advantages and disadvantages and would, no doubt, give different results from the SFA that the Tribunal rejected. However, augmenting the databases and being open to a wider range of techniques and approaches will enable a richer analysis to be undertaken, and with it, a better understanding of the opportunities to enhance efficiency in service delivery.

The value of an annual benchmarking report that contained a diverse set of measures would lie as much in the commentary that explained the reasons for the diversity and its implications for efficiency as in the efficiency scores themselves. That would be a better approach than selecting a single measure as preferred.

Accordingly, we would expect the AER to focus over the next three to five years on improving its annual benchmarking reports to reflect this, perhaps in collaboration with overseas entities such as the Ontario Energy Board, the Commerce Commission and Ofgem who would also benefit from improved benchmarking. The DNSPs should remain vigilant to the changes and improvements in benchmarking as they emerge in these reports, in order to understand the implications for future revenue proposals.

Understanding the overall regulatory paradigm

The NER is framed so as to require satisfaction that total forecast opex of a DNSP reasonably reflects the efficient costs of achieving the operating expenditure objectives and the costs that a prudent operator would require to achieve the operating expenditure objectives. The prospect that any demonstrably inefficient firm will have its revenues cut will give strong incentives to lower costs, but at the same time will result in inadequate profits for a period of time, particularly if the DNSPs struggle to meet the opex targets that the AER has identified. That, in turn, will lower the effective rate of return that DNSPs earn on their assets, undermining incentives for timely investment. Not only this, at some stage the costs of reducing information asymmetry, so that the AER can more accurately judge whether forecast operating costs reflect efficient and prudent costs, will themselves give rise to inefficient outcomes.

The use of formal benchmarking as the predominant measure of the efficient level of DNSP costs, and then to set revenues to a level sufficient to cover those estimated efficient cost, is therefore a flawed approach. Benchmarking itself is not flawed; it should and can provide useful guidance to the regulatory process. What is flawed is the AER’s interpretation of requirements under the NER to, in effect, translate a benchmark measure of inefficiency into a revenue ceiling, and given that interpretation, the incentive it presents to the regulator to implement ever more intrusive and costly processes to minimise information asymmetry (and to determine what is prudent). On this basis, the Tribunal decision is welcome, even though the decision was not framed in these terms.

What is required is an adaptation of the NER that focuses less on the precise level of forecast operating costs, and more on the process and incentives by which a DNSP improves its efficiency, back in a sense to the roots of incentive-based regulation embodied in yardstick regulation and CPI-X. The purpose of benchmarking in such a model would not be to accurately or precisely set revenues to the level of efficient costs, because that is a hugely costly exercise, but to identify a reasonable but fixed medium-term revenue target that gives strong incentives to improve efficiency and invest, even at the expense of short periods of higher than expected profits, cognisant in the knowledge that the regulator can use benchmarking in the future to establish new fixed medium-term revenue targets. Under such a framework, benchmarking would retain its utility, but its limitations, and there are many, would be far less consequential.