Coordination: A Discussion About the Operating System for a Smart Grid

Jack Ellis | Oct 06, 2010


Boiled down to its essence, the Smart Grid concept is all about coordination. Smart Grid proponents note that it will spur the development of microgrids, distributed generation, demand response, electric vehicles, storage and renewables by allowing these technologies to operate seamlessly and efficiently alongside central station generation. Indeed, all of these things are possible, but it’s also necessary to sort out exactly how that coordination will take place. Just as the iPhone (or any other “smart phone” for that matter) has an operating system that coordinates operation of all the useful applications it can provide, a Smart Grid needs an operating system that coordinates the actions of market actors, large and small, from central station generating plants to the gadgets that many expect will someday help consumers manage energy usage in their homes and businesses.

In all of the many discussions I’ve read about, listened to and participated in, there’s been no real consensus about what this operating system -- which allows disparate technologies connected at the transmission and distribution levels work together in a seamless fashion -- would look like, though there are three schools of thought on the matter:

  • A centralized control paradigm in which the utility or the grid operator exercises direct control over suitably equipped devices (supply and demand) that are made available on a voluntary basis. A/C switches are the best known example of this idea.

  • A centralized market in which loads and resources, either individually or via an intermediary, are required to offer supply and/or bid demand into a retail or wholesale market. Prices and quantities sold by generators and purchased by loads are determined by whether their bids and offers clear the market. In real time as energy is actually delivered and consumed, they are also expected to comply with operating instructions (dispatches) from the grid operator. This is the mode under which most RTOs and ISOs operate today.
  • Decentralized decision making, in which loads and resources react to transparent prices broadcast by a central market operator and are charged/paid for what they consume/produce. The grid operator exercises little or no control, directly or indirectly except in certain emergency situations. Although this idea is not widely employed in electricity markets1, it is the basis for most other commodity markets.
  • The choice of coordination mechanism is important, especially at the retail level, because manufacturers need to know how their grid-enabled end-use devices will have to interact with it. For example, A/C switches controlled directly by the grid operator don’t need to know anything about prices, while an electric vehicle charger needs to be capable of receiving and interpreting price data if it operates around a dynamic pricing tariff. The purpose of this paper is to help policymakers, regulators, and stakeholders more clearly understand the advantages and disadvantages of each approach. In each case, it is assumed customers can decide whether to participate and they will have off-ramps that enable them to opt out.

    Centralized Control

    This is the way vertically integrated utilities that are not part of a power pool or an organized market continue to operate. Utilities have directly controlled end-use devices for many years. The most notable example is air conditioning cycling programs in which the grid operator periodically switches off banks of air conditioning compressors for a portion of each hour on a rotating basis to reduce demand. Programmable communicating thermostats (PCTs) and EPRI’s Vehicle to Grid (V2G) concept are other examples. In a Smart Grid context, grid operators and/or qualified intermediaries2 would exercise control over all grid-enabled devices -- both supply and demand -- where customers allow such control and comply with other applicable eligibility requirements. Two-way communications, presumably through some type of gateway that supports multiple devices, might be required in some situations while one-way communications might be acceptable in others. Consumers would have the ability to temporarily disable utility control in accordance with applicable tariffs. Customers who allow the utility to control their grid-enabled devices would receive Incentives in the form of either rebates or price discounts based on how well they perform3.

    Perhaps the most compelling advantage of centralized control is that it is a known quantity. A/C cycling programs have been around for a long time. In many places, customers accept them and grid operators have learned how to utilize them. Grid operators also like the apparent certainty of being able to exercise control over the grid compared with the perception that devices not under their control are less likely to respond, or won’t respond in a predictable manner. Centralized control doesn’t rely on a market mechanism or dynamic pricing, both of which are controversial in some regulatory jurisdictions.

    However, these advantages have to be balanced against a number of difficult challenges, including:

    Customer Acceptance. At least some customers are likely to object to the idea that anyone, including their local utility, can reach inside their homes and businesses4 to control energy use.

    Availability of Smart Devices. Many manufacturers of the smart devices that would interact with a smart grid are not enthusiastic about the idea of a command-and-control world. Service providers are similarly skeptical. This could limit the availability of suitable devices, limit both competition among the manufacturers that would provide them, and more importantly, discourage innovation. Moreover, some would argue that a switch controlled by the grid operator needs no real intelligence and consequently, doesn’t need to be very smart.

    Lack of Flexibility. By their very nature, A/C cycling and similar initiatives are inflexible and cannot easily be tailored to address variations in customer preferences or, more importantly, grid conditions that are likely to be less predictable as penetration levels of renewable generation increase. One-size-fits-all programs also can’t easily deal with combinations of devices -- for example a PV array coupled with batteries at one home, and a fuel cell coupled with thermal storage at another.

    Suboptimal Outcomes. Computer systems that attempt to “optimize” the use of individual devices will not be able to handle all of the possible combinations of supply and end-use devices, each with different efficiencies, duty cycle requirements and customer preferences as to their use. As thing stand today, the computer systems used by grid operators in organized markets to optimize the dispatch of generation occasionally fail to solve in the available time frames, even though they have to deal with a relatively modest number of grid-connected resources5. Adding millions of end-use and distributed generation devices means that either the amount of available computing power must be increased by many orders of magnitude at enormous cost in order to guarantee acceptably complete solutions in the required time frames, or the optimization problem must be greatly simplified which in turn affects the optimality of the solution.

    Customer Privacy and Data Collection. Privacy advocacy groups have written extensively about the risks of collecting device-specific consumption data. Distributed generation exacerbates this situation to the extent the grid operator must also know how much each customer pays for fuel, the efficiency of each customer device, how much each customer pays for maintenance, and whether and how generation and storage are co-located, among other parameters, in order to incorporate distributed generation in its optimization6. Moreover, grid operators and intermediaries take on an additional burden by having to ensure that all of this data is up-to-date and correct in order to guarantee that their optimal solution does not inflict harm on individual customers even though customers as a group are better off.

    Liability. Centralized control of the Smart Grid creates new legal liabilities of a type and scale that grid operators have never faced before (though intermediaries do deal with some of them today). First, they will likely have to assume a greater degree of liability for grid-enabled device under their direct control that fail to operate properly. Second, they will be held liable for unauthorized release of device-specific data, whether on the supply (e.g. distributed generation inputs) or the demand (meter data) side, no matter what the reason. Third, the scale and complexity of a central optimization means it will occasionally produce faulty outcomes. Customers as a whole may realize meaningful savings, but individual customers could see higher costs, whether it is because the optimization was unable to solve in the allotted time, the optimization algorithm was faulty, or certain customer-specific data was incorrect or out-of-date.

    Centralized Market

    All RTOs and ISOs in the US and the Alberta Electric system Operator (AESO) operate in this fashion. The Centralized Market differs from Centralized Control in several respects. First, buyers and sellers must submit bids (to buy) and offers (to sell) consisting of price-quantity pairs, but the prices need not be strictly cost-based7. Second, an algorithm clears the market by matching up buyers and sellers subject to a variety of operational and power flow constraints, including transmission congestion. This typically takes place once around mid-day for the following delivery day (Day-Ahead Market, or DAM), and again periodically within each delivery hour (Real-Time Market or RTM) to deal with new information and residual changes in supply, demand, available transmission capacity and other system conditions. The Day-Ahead Market is generally financially binding whereas the Real-Time market is physical. Third, variations in amounts delivered or consumed between the financial Day-Ahead Market and the physical Real-Time Market are settled in cash. Fourth, while suppliers are discouraged from producing amounts that are inconsistent with their financial commitments unless otherwise instructed by the grid operator, typically these kinds of behavior are not prohibited or penalized as they would likely be under centralized control.

    One of the important advantages of the Centralized Market approach is that it transfers some of the decision-making authority away from the grid operator and into the hands of market participants. Another is transparent prices that help form expectations about market participant behavior provide investment signals, and guide operating decisions.

    The Centralized Market approach relies on optimization algorithms that are largely identical to the ones used for Centralized Control. To the extent market participants are allowed to submit multi-part cost functions rather than simple price quantity pairs, the optimization must be capable of handling them8. This means the Centralized Market approach is still prone to algorithmic flaws and it may still not reach a suitable solution quickly enough, especially as the number of participating devices and/or entities grows from a few thousand to a few million. More significantly, customers that wish to participate must understand and operate under an extensive, complex set of market rules and procedures that typically runs into the thousands of pages and imposes burdensome obligations and eligibility requirements, which makes participation difficult, time-consuming, and expensive. The high cost of participating also limits liquidity, which can lead to price volatility unrelated to market fundamentals.

    Centralized Markets typically only provide limited forward price visibility, which means customers have little ability to act independently of the grid operator to address changes in grid conditions that could otherwise become apparent through changes in forward market prices. This is particularly important for integrating variable generation, which is subject to a relatively high degree of forecast uncertainty and some production variability, and for storage, which could very effectively self-manage its own state of charge given a stream of forward prices that reacts to changes in market and grid conditions, including the impacts of changes in forecast amounts of variable generation.

    Decentralized Market

    This idea builds on the centralized market approach by adding several elements and slightly modifying several others.

    The first point of departure is providing more opportunities for market participants to make adjustments by clearing the market more frequently than once for all hours of the upcoming delivery day (Day-Ahead Market) and then again in real-time (Real-Time Market). A series of Hour-Ahead Markets (HAM) would be added to the Day-Ahead and Real-Time Markets that can include as many or as few forward-looking hourly delivery intervals as desired, though 24 would likely be the right number, and these could be cleared as frequently or infrequently as desired. Electricity retailers could take advantage of these more frequent update intervals9 to manage changes in weather and other hard-to-forecast conditions that affect their procurement decisions on behalf of retail customers. Large central station generators could better manage around unexpected outages. Variable generators could adjust their financially binding delivery commitments based on forecast updates. Grid-level storage and electric vehicles would benefit by being able to respond more flexibly without having to wait for guidance from the grid operator. In each case, providing more frequent opportunities to make adjustments reduces risks and costs.

    The second point of departure is to provide a simplified method for retail customers to participate directly in the market. The availability of forward prices that adjust regularly over the upcoming 24 hours means smaller market participants could simply react to these prices (by acting as price takers) rather than having to formulate bids or offers, though they would have the option to do so and the forward prices would help them10. To the extent parties wish to submit bids or offers, they could do so via the same kinds of intermediaries that currently provide this function in many RTOs. Conceptually, this is similar to the way retail customers participate in equity markets today: larger parties make the market by submitting bids and offers directly to ECNs11, while smaller parties operate through retail intermediaries12 that aggregate bids and offers and submit them to the ECNs on a customer’s behalf. Although market participants (via their intelligent, autonomous devices) could simply plan and execute their operation guided by the forward price stream but without submitting bids or offers (price/quantity or quantity-only), doing so would provide price certainty compared with the less certain and more volatile Real-Time prices that would be used to cash out deviations from forward commitments.

    The third point of departure would be to simplify the algorithms that are used to clear the market, if necessary, so that they can be run more quickly and more often without requiring large investments in computing power. These algorithms would still have to consider transmission and system security constraints, but they could limit bids and offers to simple price/quantity pairs rather than allowing more complex bid curves.

    If one of the key goals of a Smart Grid is to better integrate renewable resources, then one of the most important benefits of this approach is that it allows market prices to more readily reflect changes in system conditions that result from changes in variable generation forecasts and other factors, which in turn allows other elements of the grid to adjust accordingly. A forecast increase in wind production during the middle of the night would lead to a reduction in market prices, which might in turn induce the operator of a storage device to buy that low cost energy and it might also prompt other, flexible generators to reduce their output by purchasing back energy sold earlier at a higher price. This scenario is not possible today because once the Day-Ahead Market is cleared, the hourly energy markets tend to operate for no more than the upcoming delivery hour.

    Another advantage is that it would facilitate the economically beneficial use of distributed generation and storage, electric vehicles and price-responsive demand. Consumption and production decisions, such as charging an electric vehicle now at a higher price rather than in four hours at a lower price, would be greatly simplified by knowing what those prices are and being able to act on them. Having opportunities to enter the market to make adjustments more frequently also simplifies decision-making and participation. Distributed generation, demand response, storage and electric vehicles would not require specific, proscriptive tariff provisions in order to be able to buy or sell electricity, or to beneficially take advantage of market information.

    Unlike either the Centralized Control approach or the Centralized Market approach, the grid operator would not have to know anything about the economic details of the myriad supply and end-use technologies that participate in the market. It would not be responsible for gathering data, securing that data, and making sure it is up-to-date. Decisions taken by market participants based on forward prices would be the responsibility of the participants rather than the responsibility of the grid operator. To the extent some type of optimization algorithm is required to clear the market, it could be somewhat simpler and the required degree of computational effort to clear the market for a given hour would not have to expand to accommodate increases in the number of intelligent devices.

    The Decentralized Market approach does have some drawbacks. First, the forward hourly markets may not attract enough participation to provide sufficient liquidity, which could lead to unacceptable levels of price volatility. Second, it would require changes in existing RTO and ISO market systems that could be costly and time-consuming to implement13. Third, there would likely be many prices for an individual delivery period14 rather than the single price that is inherent in today’s RTO and ISO markets. Although multiple prices for a single delivery period that depend on the timing of the transaction are a common feature of most financial markets, they are still relatively rare in US electricity markets. Finally, some customer groups are uncomfortable with the idea of passing locational marginal prices directly to customers.

    Transitional Issues

    It’s relatively easy to describe an end state. Defining a path toward that state that doesn’t needlessly scare customers; disrupt their lifestyles, budgets and businesses; or raise unfounded objections is much more difficult. Following are some of the transitional issues that need to be addressed before the soul of a Smart Grid can emerge:

    • Renewable resources have high fixed costs and low or no variable costs. At high penetration levels, they depress average market prices by displacing other generation with higher variable costs. Their presence also leads to increased price volatility as conventional resources are dispatched around them. Some price volatility is good for a Smart Grid, but how will volatility be perceived alongside market prices that are low on average?

  • If customers pay for energy based solely on market prices that reflect marginal cost bidding, what mechanism allows resource owners to recover their fixed costs, particularly if market prices are depressed by the presence of low or zero variable cost renewable energy resources?
  • How will changes in the predictability and timing of customer demand affect utilities? Does the underlying regulatory and institutional structure have to change as well?
  • No RTO currently passes locational prices directly through to retail customers. Some customer groups are concerned about the impacts of doing so, yet the Smart Grid won’t work well unless the coordination mechanism reflects system conditions. How should this dilemma be resolved?
  • Will customers accept and be able to understand a bill that includes prices that change every hour?
  • To address customer concerns over locational pricing, would it be sufficient to pass through locational prices to intelligent devices for coordination purposes while continuing to charge customers flat rates? Would it be problematic to pass through locational prices to intelligent devices while charging customers an average hourly price that has been averaged over all locations in a utility service territory? What gaming and adverse selection issues does this option raise?
  • Conclusion

    Smart Meters and general concepts about what a Smart Grid might do are not enough. Regulators, policymakers, industry experts and stakeholders need to turn their attention to the operating system that will enable a Smart Grid to actually work.


    1 A concept along these lines was proposed for California’s wholesale electricity market in “Simplified Bidding for WEPEX”, Cazalet, E.G. and J. Ellis, 1996.

    2 Most RTOs and ISOs only deal with qualified intermediaries, which in turn deal with end-use customers. In California, the intermediaries are known as Scheduling Coordinators, while in ERCOT, they are known as Qualified Scheduling Entities.

    3 In other words, customers would not be rewarded for simply allowing the control device to be installed. Disabling the device, even when allowed to do so, would affect the level of compensation.

    4 Conspiracy theorists are already alleging that Smart Meters open to the door to government control over electricity use. For example,

    5 Typically in the low thousands of individual generating units, if that many.

    6 A system operator would prefer to know this information so that it can treat DG on an equal basis with central station plants. FERC has already done something similar by requiring RTOs and ISOs to ensure that demand resources can provide some of the same operational attributes so DR can be treated on an equal basis with central station generation.

    7 Market participants can also be price takers by submitting quantity-only bids and offers, in which case they pay or receive the prevailing market price for their entire bid or offer volume. In most RTOs and ISOs, generators submit offer curves that are not unlike the price curves they would provide to the grid operator in the centralized control arrangement.

    8 More precisely, a set of price/quantity pairs that must be monotonically increasing with respect to price, plus a value for starting and running at zero output.

    9 It is likely, though, that they would.

    10 Customer devices, including PEVS, could be pre-programmed to respond at certain prices, e,g, through auto-DR.

    11 Electronic Communications Networks, which have largely replaced floor brokers and specialists (pit trading) in most financial markets.

    12 Examples include Charles Schwab and E*Trade Financial.

    13 Presumably such changes would be undertaken only if they produce benefits that outweigh the costs.

    14 Potentially as many as 26 different prices -- one each for the Day-Ahead and Real-Time Markets for that hour, and an additional price every time the market for that hour is cleared between the close of the Day-Ahead Market and the time power starts to flow.

    Related Topics


    Marvelously written article Jack! It brings together in one paper most or all of the structural and financial problems facing the adoption of a Smart Grid.

    Personally I don't believe the Centralized Control or Centralized Market systems will work very well at all with a Smart Grid of any kind, given the main purpose of Smart Grid is to coordinate everything in the grid to optimize generation, transmission, and especially optimize customers' consumption habits. Being in electronics engineering throughout my career I can say with some confidence that the computational effort required to achieve this optimization on a grand scale is far too overwhelming to be cost effective in ANY central location such at an RTO or ISO.

    The ONLY economically feasible system would be the Decentralized Market approach. It would parallel the revolution in my industry created by the personal computer having reached every desktop and home, and by embedded microcontroller electronics appearing in every consumer microwave oven and refrigerator etc or industrial control system. We live in a world of distributed markets, where distributed technology is everywhere today. It would be unthinkable to imagine centralized control and computational effort in achieving any of this, and the same economic logic will eventually apply to a Smart Grid if it is ever to become successfully implemented on a grand scale, in my humble opinion.

    On this note I will repeat what I have written about in past on this website. Len Gould's proposals for an Independent Market for Every Utility Customer (IMEUC) detailed on this website is the closest idea I have seen to realizing any sort of Decentralized Market system. But more importantly is decentralizes control of the Smart Grid technologies when it deals with consumption of electricity since for example it places smart meters and any other customer Smart Grid technology under the ownership and control and programming of the customers themselves. The latter is what makes every other decentralized technology or market work effectively in our world today.

    Chris King at eMeter just posted this note to EEI's AMI list server:

    "In viewing various EV rates that have been adopted, I haven’t seen one that seems to work particularly well. The goals would be:
    • to allow EV owners to save money by charging at low off-peak rates,
    • to provide an economic incentive to avoid charging during high-cost peak hours, and
    • to provide an economic incentive to minimize rapid charging, because it creates very high demand that ultimately could threaten distribution transformers.

    Hourly pricing – opt-in – offers the best solution to the first two goals. However, hourly pricing is determined in the generation and transmission markets. Therefore, there is no price signal to avoid rapid charging. Time-of-use pricing provides a good, continuing incentive as well, but also without the distribution incentive.

    The classic solution is a monthly demand charge, but this provides no continuing incentive once the first high demand peak is reached during the month. As one solution, this could be modified into a daily demand charge to make the incentive continuing.

    Another possibility would be to impose a demand charge when a threshold is exceeded. However, because there is great variation in individual home peak demands, such a threshold would have to be individually tailored relative to something like total monthly usage. This is complex and probably not a good idea.

    Another would be to impose a surcharge any time rapid charging is used. This could be determined by monitoring the charger or using an algorithm to determine when rapid charging is used.

    Given that any rate must be simple for consumers to understand and practical for utilities to bill, what do folks think?"

    For folks with web access, the fact that the price is different every hour is not a big deal. Just get on the web and look at the detail. We did this at APX in a way that allowed users to see the hourly detail for any of dozens of Cal ISO charge codes. Price times quantity. Simple, easy math. If you want to buy a hedge, there will be people standing ready to sell you one.


    "Given that any rate must be simple for consumers to understand and practical for utilities to bill, what do folks think? For folks with web access, the fact that the price is different every hour is not a big deal. Just get on the web and look at the detail. .... Simple, easy math."

    The web does indeed make it relatively easy to manually monitor real-time rates or rates that change frequently. But this is not the issue for consumers under variable rates, the REAL issues are a) how to reconcile their bills with variable rates, and b) how to monitor their running bills AUTOMATICALLY so they won't be 'shocked' by a high bill when it arrives.

    Under flat-rate billing reconciling was pretty simple for consumers (just multiply one, or possibly two rates under bi-level schemes, times kwhrs). Under variable rates this becomes much more time consuming to make it worthwhile to practice all the time, assuming the consumer can easily match up rates with interval consumptions for each rate period.

    Here in Ontario most of the 5 million utility customers are now on Time-Of-Use billing with smart metering. The province regulates fixed TOU rates for three daily rate levels as off-peak, mid-peak, and on-peak, with the ratio of highest on-peak to lowest off-peak rates in the ballpark of 2-to-1. To make it easier for consumers, the Ontario government has forced every distribution-only utility company to provide on-line customer presentment tools that provides hourly consumption profiles up to the previous day, but not every utility provides a view of their running total bills. Final bills in all cases however must show total kwhrs separately for each TOU period.

    The initial impact of TOU rates has not been very well received by the Ontario public as most consumers' bills have gone up under them, substantially. The are multiple reasons for it, the biggest one being the lowest off-peak rates are now set virtually the same as the province's flat rates were just a year ago, So without drastic load shifting, most consumers' bills have increased overall.

    There are many businesses and consumers who simply cannot drastically load shift the bulk of their consumptions to off-peak for many varied reasons, but for those consumers that are able to practice much load shifting, the paltry 2-to-1 rate ratio is not large enough incentive to change their habits much. Coincidentally the Ontario government has now seen the load profile data under TOU billing and recently admitted that the on-peak-to-off-peak ratio should be re-examined to promote more incentive to load shift for those consumers that could do more.

    In my opinion, and I’m sure it is shared by others, the solution to getting consumers to be much more active in changing their consumption habits goes beyond simply making the TOU rate ratio larger. It is to make those web presentment tools Ontario’s utilities are providing on-line more readily available in the consumer's face in real time, without having to go on-line every day to view them. The data needs to be up-to-the-minute information and not just up to the past day’s consumption, since the latter is viewed as “too late to do anything about” by most consumers, if they even remember what they did the day before.

    Real-time in-home displays or personal computers can easily do these functions, and lead to progressive adoption of other home automation for demand responses using standardized interfaces, e.g. load shedding or handling home electric vehicle charging management etc. Although some progress has been made in recent years, the private sector can only go so far in developing this stuff without the intimate involvement, read financial support and commercialization support, of the utility companies (and/or governments), because this stuff must ultimately communicate with the rest of their Smart Grids.

    Bob. I don;t think I agree with your proposition that "the computational effort required to achieve this optimization on a grand scale is far too overwhelming to be cost effective in ANY central location such at an RTO or ISO. " What assumptions did you use to arrive there?


    I included all the computational effort in an ideal widely implemented Smart Grid required to do all or most of the following tasks:

    - manage grid resources for all large central and growing numbers of distributed generation, and a growing number of variable transmission paths
    - control customer-programmed demand responses to either price signals or grid operator requests
    - control customer-programmed variably-timed loads such as vehicle charging
    - communicate to all customers real-time prices, their instantaneous power demand, and cumulative energy consumption up to the minute.
    - calculate customers' running bill and display it in real time based on metered energy from a smart meter.
    - calculate, update and display in real-time each customers' historical energy consumption load profiles
    - manage a true distributed retail market where each individual customer or groups of customers can buy energy from individual generators, and manage/change their energy supplier retail contracts at will (such as in IMEUC).

    Bob, there's a very simple way to solve the problem that could have been built into interval meters with two-way communication. Add a display, wired or wireless, that shows the running total of price times quantity. This can be done with flat rates, TOU rates and dynamic prices, but it cannot be done with schemes like California's that use tiers, or with the two part (demand and energy) rates paid by most commercial and industrial customers.

    The key reason for using dynamic prices is to provide coordination. I'd much rather see dynamic prices combined with a hedge so customers get the best of both worlds - a predictable price for most of their consumption plus the right incentives to move demand around, either manually or through the use of automation, when it makes sense. TOU prices will do that when system conditions are predictable, but in a world with lots of renewables where production is variable and uncertain, TOU prices are too inflexible.

    Len, I agree completely with Bob's assertion regarding the amount of computational effort that would be required if this was all centralized. Simplify the problem or buy trainloads of hardware and hope.


    You are very correct that much of the problem stems from smart meters that were not installed with two-way communications built in for future uses in Smart Grid.

    I was made acutely aware of this several years ago when I learned our utility companies in Ontario, and most elsewhere, refused to bear the incremental costs of equipping their smart meters with the added options of two-way communications, unless forced to by regulators or government legislation. Meter manufacturers bend over backwards to make several meter options available but cannot sell them to most utilities in large numbers.

    Hence there is indeed some advantages in Len's IMEUC proposals that the smart meters need to be placed in the ownership of customers. Like a home PC, customers could add accessory options by simply buying them from a retailer and installing them, including software upgrades. As it is now, customers are prevented from adding any extra functionality to meters EVEN IF THEY ARE WILLING TO PAY FOR THE METER UPGRADES THEMSELVES! A common response from a local distribution utility is that these options for meters are simply not available to their customers, period. And if you want them made available, they tell you talk to the government and the utility regulators about changing it.

    I didn't mean to imply that meters had to have two-way communications. There are other ways to disseminate the prices. In California, PG&E equipped its meters with a home area network gateway (HAN) that's supposed to allow the utility to reach into homes so it can capture end-use consumption data and control individual appliances. That's not necessary and in fact, it has caused a lot of justifiable agitation among privacy advocates. A method for broadcasting prices would be useful, but some form of low frequency encrypted radio signal might be equally effective and much cheaper.


    I’ll grant you there are other ways to broadcast prices to customers that likely cost less, but having the smart meters do it simultaneously allows other useful information to be transmitted to customers such as their instantaneous power demand, up-to-the-minute cumulative energy consumption and bill tracking, and even power quality. The meters are also a natural gateway into the grid for all customers to communicate back to the utility, or generators in something like a decentralized retail market, since every customer will have a smart meter.

    I suspect t and predict the internet will ultimately be used for much of the anticipated customer communications with Smart Grids since everything else in technology these days is getting equipped with internet connectivity. Smart meters will end up being used only for interval metering and TOU billing, and for utility benefits like outage management and power quality monitoring. Sadly there will be few if any benefits in this case to consumers.

    Furthermore, the justifiable agitation about PG&E's HAN gateway equipped smart meters is understandable because PG&E has made the mistake of planning on controlling customer load shedding with them. The last thing most of the public and privacy advocates want is big brother reaching into homes and controlling their energy uses. PG&E should have made plans and made it clear to the public that individual customers can program and control their own load shedding in response to price signals sent through the HAN gateway, and not controlled directly by the utility company as many utility-controlled air-conditioner cycling systems work now.

    I get it now. "I agree completely with Bob's assertion regarding the amount of computational effort that would be required if this was all centralized." when the assumption is that all control decisions are made by the utility's central processing resources. Utilities and regulators need to learn from the effects of the networked micro-processor-based computers kiling off the large mainframes in the 1980's. (though even what are still called "mainframes" these daya are just many microprocessors working in parallel inside a single box.)

    Exactly Len. There are far more embedded microcontrollers i.e. small microcomputer chips in use in the world now than there are desktop or laptop PCs and mainframes. Every home appliance has one like microwaves, fridges, dishwashers, laundry machines, furnaces, workshop power tools, thousands of kids games from baby crib toys to toy vehicles to video games, road vehicles of all kinds, you name it, the list is literally mind boggling.

    I was more than once told by some of Ontario's utility executives and by Ontario's Power Authority that smart meters should indeed be used for a lot more than just interval metering and TOU billing. But the issue time and time again is money - who would pay for upgrading the millions of meters in the field to do so FOR EVERY CUSTOMER without foisting extra charges onto everyone's bills under existing regulatory frameworks. There are only two solutions to this in my opinion a) place the ownership of the meters in the hands of customers like in your IMEUC proposals such that customers can buy the upgrades if they choose, or b) get the government to force the regulators to allow utilities to change their business models where they don't have to uniformly bill every customer, and as such add charges to only those customers that want extra goodies for their meters or homes or participate in different market schemes like an IMEUC.

    The utility industry and the government need more brilliant out-of-the-box thinkers and inventors like you Len. Maybe we would then see the utility industry progress much faster than the snail’s pace it does now.

    Before anyone should try to modify the excisting grid to become *smart*, I would recommend to analyze the excisting analgog AC grid.

    Here are some figures from Germany, a relative small country, of the size of Californai and Oregon combined:
    Usually, these voltage lines run into thousands of kilometres, in Germany the network consist of almost 1,67 million kilometres, that equals 1.000.000 miles... (94 percent for the low voltage lines alone). 566,300 substations/transformers are used to step up and down the voltage (98 percent for the base voltage network).
    At the power plants the electricity is generated at a relatively low voltage of up to max. 40 kV, then stepped up by the power station transformers to a voltage of 220 kV to 380 kV (extreme voltage), because transmitting electricity at high voltage reduces the fraction of energy lost. The electricity is carried through a transmission network of high voltage lines to substations, where the voltage is stepped-down to 110 kV (high voltage) for load points (localities, industrial areas, villages, etc.) or to 20 kV (base voltage) for industries or further distribution to low voltage lines. Here transformers are again used to step the voltage down to 0,4 kV for distribution to commercial and residential users. All those transformers are fillied with poisioning cooling liquids...

    The figures for the US and other countries are similar alarming. Just try to find out the facts for the excisiting power grid in your state or your country.
    Than you know, that is does not make much sense
    to invest a penny into such archaic networks.

    We should be much smarter than just to update a dinonsaur into another one.
    A quantum leap is required for our energy system, that can be said by just using common sense - and the laws of physics, if you like.

    More here:

    Arno, as you point out, 98% of those grid miles and transformers etc. are still going to be required just to share power among neighbours, an absolute requirement for any rational electricity system no matter what source of generation is used.

    The transitional issues are more problematic for regulators than the public. In California, with high cost electricity, my water bill is nearly as high as my electric bill, and the cell phone bill dwarfs both. And cell phone bills are per call, more detailed than per hour. The problem described in Ontario seems to be that the TOU rates were not revenue neutral. The description appears to be that customers are upset about a rate increase that was implemented along with TOU rates.

    Significant volatility in prices is necessary to get customers to pay attention to differences in rates when the total bill is not all that large on a relative scale. So once we get past regulators customers will get over the fear of change. Once again the regulators are the problem.

    The more interesting question is who should set the rates at various times, and how. Some entity needs to be the retailer for all but the biggest loads. Perhaps unregulated retailers can answer these questions better than regulated utilities. To begin with the prices incurred by the retailer are not passed on unmodified in any industry. So the starting point here is probably that different retailers may offer different pricing plans, just like every other industry.

    The Dean of Electrical Engineering and Computer Science at UC Berkeley (wish I could remember his name) made a point during the disastrous design of the California re-regulation of the 1990s (erroneously labeled deregulation) that markets have solved problems that cannot even be formulated mathematically. The engineers and lawyers designing a "market" that lacked the essential ingredients of a market did not listen to such wisdom. They implemented command and control re-regulation.. I am suggesting that the best answer for retail pricing in a smart grid is to listen to the Dean and allow markets to offer as many answers as competitors want to try. Then we can get the best answer(s) for pricing without having to have perfect foresight.

    "Then we can get the best answer(s) for pricing without having to have perfect foresight."

    This is a huge comment Dean. A predominant culture exists in governments that foresight and planning must be perfect, with no room for errors. Politicians in particular are loathe to "experiment" with solutions that have any chance of being flawed when it comes to energy policies.

    So I agree completely with your assertion that much or our electricity system problems can be traced back to regulators and their mandates set by governments to begin with as being flawed. They worked in the past when the system was static, but they do not facilitate technology changes or pricing changes very efficiently or in many cases very effectively.

    I've just finished writing a paper that I had intended to file with the CPUC (until I discovered that I had filed for party status on behalf of a client) that talks about some of the impediments created by poorly considered regulatory and legislative policy. For example, regulators went from one extreme (requiring the utilities to procure all of their net short position from the spot market) to the other (requiring them to be 115% hedged), which has played havoc with the spot market. If there's no spot market volatility, there's no point in building a Smart Grid unless you think I'm wrong about command-and-control.

    Lately I've been reading a few pages of Robert Caro's biography of Robert Moses. Even if Caro's claims are wildly exaggerated, it's a fascinating description of the means by which "public servants" can act contrary to the public interest. More importantly, it's a fascinating study of human nature and human weaknesses. Like regulators and politicians.

    Another problem created by regulation as it is implemented in our electricity systems is that it stifles private investment in new technology research and developments. The risks are considered too great by many private companies when the only customers are the utility companies themselves, and given regulators determine how and when utility companies can fund the commercialization of new products. And everyone knows regulators are usually at the mercy of unpredictable energy policies set by governments, the latter having a historical habit of changing whenever a new political party is elected into power.

    Utility companies must be very frustrated by this. Time and again I have heard of companies in my electronics industry be led down the garden path when their latest technology is pilot tested by a utility company but never sees commercial production beyond a pilot program. Often the utility company can fund small-scale pilot test projects out of their own pocket, that go on for months or years with very limited numbers of product samples. But as often happens the product never becomes commercially successful when it never goes beyond the pilot project – because regulation often prevents the utility company from recovering their costs to later deploying it on a large scale.

    The result is many companies simply stay out of the business altogether even though they are quite capable of developing innovative solutions.

    Bob, I agree that regulation is partially responsible stifling innovation, but you're presuming a utility has no choice other than to have consumers fund R&D. Maybe that's fair when utility returns are limited on the upside.

    But then I see Ice Energy pursuing sales of its residential scale thermal storage through utilities and wonder why they just don't sell directly to consumers?


    You have a point, utilities can and often do get funding from other sources besides consumers. Government handouts are a big alternate source in recent times, which are really just consumers' tax money anyway.

    Ice Energy is typical of companies who may indeed by in a position to sell their new products directly to consumers, but, they perceive their sales as being much higher if they sell their products to consumers through the local utility company. The reason is that consumers have for so many years been accustomed to utility company monopolies on energy, and anything that might help save consumers money on their energy bills should be SANCTIONED by their local utility company. Anything not sanctioned by their utility is suspect as being questionable, or at worst being a scan. Hence the best way to sell new energy products that are good for lowering your energy bills is to have your utility company market them.

    ... typo "scan" should read "scam."

    Here's a point to chew on Jack regarding my pet subject.

    If consumers had real-time energy display monitors at their disposal, they could buy an Ice Energy system and test it. They could first accurately log their energy bills without using Ice Energy's thermal storage product, and then compare their bills using it. As such they could accurately see the economics of buying and using the system, perhaps even sharing the information on the internet with millions of other consumers as is often done today for hundreds of other consumer products.

    If the Ice Energy system saved substantially on energy bills, and was reported in customer reviews with actual data on-line, such advertising would be priceless as it would be a huge boost for Ice Energy selling the product. It would be a prime example of one technology, in-home displays, leading to the widespread commercialization of another innovative technology.

    The problem ice storage faces is that utility rates do not reflect the cost of providing service. They are highly averaged. Rates during the afternoon air conditioning peak are subsidized by charging surprisingly high demand (capacity) charges during the night when ice is made. But the marginal capacity cost at night is zero, except when wind generation overwhelms night time demand. Then the marginal cost falls below zero. But retail customers have no such information in their rates. So again, the problem is the regulators.

    Ice energy goes to regulators and utilities to get funding because there is no retail market with incentives to operate the grid efficiently.


    Your revelations about rates not reflecting cost of services is very true, I'm sure most including me agree with you. Most consumers however don't care what the costs to the utilities are of providing services, they only care about consumer rates and their own bills.

    I'll grant you the public SHOULD care more about costs of providing services, but most of the public already knows that rates DO NOT reflect true costs of services. Indeed much of the public believes that their rates are set by political forces manifesting themselves through regulators, hence the public largely believes that consumer rates are not entirely determined or influenced much by costs of services as much as they should be.

    If marketed directly to consumers, the decision to buy new technology products like Ice Energy's thermal storage system would depend entirely on energy bill savings. In areas that implement TOU billing, all consumers will pay non-zero rates at night at something that is lower than daytime rates, so there will be some potential for saving money with such products as Ice Energy’s. The only question is how much of a savings is it over a given period of time for a given consumption lifestyle of the consumer versus what the product's initial investment cost is. Energy display technology that can monitor up-to-the-minute consumption patterns and track bills with high granularity could resolve this question very easily for consumers.

    The principal reasons we have these esoteric conversations about rates and whether they reflect costs are because a) electricity has become an instrument of social policy and b) the regulatory compact. If politicians could contain themselves and we could put in place a more competitive structure for utilities, pricing would take care of itself.

    In fact, I suspect most consumers know how much they pay in aggregate but they are clueless about the price per kWh or the "incentives" that are built into their pricing (rate) plan.

    Policy initiatives like the Renewable Portfolio Standards may depend on customer engagement (aka, Smart Grid), but they way they have been implemented makes a mess out anything resembling real-time pricing which, as I tried to point out in this article, is about the only sensible way to coordinate demand and supply.

    "If politicians could contain themselves and we could put in place a more competitive structure for utilities, pricing would take care of itself."

    How true Jack. Just try and make politicians contain themselves on a subject that raises ire in most consumers.

    Here in Ontario where TOU billing has been forced onto everyone, and, more importantly, TOU rates have been set by our regulators to have the lowest off-peak rate almost on par with what our flat rates were just a year ago, much of the public is now furious seeing their overall bills rise substantially. Many on-the-fence voters have been angered so much by it that they have already decided to vote against the incumbent Liberal party in our next provincial election slated for October 2011.

    Whether we like it or not, the regulatory setting of electricity rates is a very hot political potatoe likely in just about all of Canada and the US. Politicians are acutely aware of this, and know they would commit political suicide to ignore it as an issue and not oversee price regulation. This is the unfortunate reality of our electricity system, sadly, so finding a solution is not as simple as saying the politicians to take their hands off.

    However I’m glad that I am not the only writer on this website that recognizes the problem. Let’s hope some politicians read this good article of yours Jack and actually do something about it to change things.

    "the politicians NEED to take their hands off." is what I meant to say.

    But of course not until they set in place changes to regulation that would allow the coordination between demand and supply, and that my friends means much more exposure of customers to real-time prices, which by extension will mean much more (electronic) communication between customers and the grid.

    It promises to be an interesting future for Smart Grid and our politicians, and especially we consumers.

    One of the dilemmas governments and regulators are historically faced with is the long-held belief hat the general public could not HANDLE their electricity rates being exposed to real-time prices. The large variation from daytime to nighttime prices, and potentially even larger variations over shorter periods during the days when supply cannot satisfy peak demand has been a principle reason behind having price regulation in the first place for consumers.

    Among the biggest concerns about real-time prices would be consumers' ability to reconcile their utility bills. If some portion of the public have trouble now reconciling their bills under flat-rate billing or under TOU billing, the problem would explode in magnitude under real-time prices.

    Technology by today's standards however could EASILY handle real-time prices and bill reconciliation, and moreover also handle demand responses, or any sort of distributed electricity markets that we have been talking about on this website. And it could handle all these in most cases in an AUTOMATED fashion with software and with the necessary communications in place between customers and the grid.

    Because the software and the technology have not been developed to do all these things comprehensively, it becomes a chicken-and-egg problem for politicians to even consider exposing consumers to real-time prices. The same can be said for utility industry people. They would all rather wait for the technology to be developed first and pilot test it before changing regulation to allow its use.

    Bob, the technology to reconcile a bill that contains over 100 different charges in each hour is already available. It just has to be adapted for retail use. Web based.


    Thanks for the update, this sounds interesting. I suppose such tools do exist for industrial or commercial customers that are already signed up for real-time pricing. I would be interested in learning who has developed this web-based tool if you would like to email me any details that you can at

    Being web based I wonder f it can present a consumer's running bill in real time like an in-home display would/could, or does it do it after the utility bill has been created every billing period. The former would be much more desirable for consumers. If it did I also wonder if it could be interfaced with demand response software that could control residential loads in a home-area-network, using price thresholds programmed by the consumer. This feature would likely make a nice comprehensive package for consumers.

    Personally I prefer a more direct form of communications than the internet, where prices and consumption data can come from a utility smart meter directly into a customer's in-home equipment. Having this data from the utility or from the meter first go outside to the internet and then back to the consumer seems less efficient, and depends on the customer having internet access.

    The bigger problem I still believe is with getting regulators and politicians to allow real-time price billing for residential consumers.