In the summer of 1999, the United States was struck by a series of major utility outages as aging T&D systems proved vulnerable to the I twin challenges of extreme heat and extreme demand. New York City, New York, was hit by its worst blackout in more than 20 years. Half a million customers lost power in New Orleans, Louisiana. And in Chicago, Illinois, as a month of recurring outages crippled vital parts of a sweltering city, an angry Mayor Richard M. Daley expressed the voice of many when he demanded immediate action. ComEd, he declared, needed to rebuild its system and do it now, starting at “ground zero.”
In response, ComEd's new chairman, John Rowe, launched a comprehensive overhaul of its T&D system, a US$1.5 billion reliability improvement plan that industry experts called “unprecedented.” Rowe demanded a fundamental core change aimed at producing a T&D system that met or exceeded industry standards. His message was simple: “Nothing matters if we don't keep the lights on.”
The Long Hot Summer of 1999
In 1999, the first major blackout of Chicago's late summer heat wave began beneath the manholes located along California Ave. In the early morning hours of Friday, July 30, the 12-kV line feeding Cortland Substation short circuited. Within hours, a series of falling T&D dominos had two major substations down, with the power gone and the air conditioning out in nearly 100,000 homes. It was the hottest day of the summer, in what the Chicago Tribune later calculated was the fourth hottest week of the century.
Public anger rose along with the temperature as other T&D components failed over the next five weeks. Manhole fires occurred on August 9 and 10. Chicago's central business district, the Loop, went dark on August 12. Later, power failed at four Chicago icons: Meigs Field, Lake Shore Drive, the Field Museum and the downtown courthouse named for the mayor's father.
These highly visible back-to-back service interruptions dramatically exposed the true depth of problems that had troubled customers, ComEd and public officials for several years. Rowe, stated plainly, “I will not tolerate it, and you will not have to.”
The ComEd Response
ComEd hit the ground running. By the time the last service was restored on August 12, ComEd had dispatched more than 700 men and women to open manholes and inspect substations across the city. All told, during the first six weeks, ComEd devoted an estimated 250,000 additional man-hours and more than US$20 million to the round-the-clock emergency-response effort, above and beyond normal operations.
Two days before the August 12 Loop outage, Rowe had assigned David Helwig to head up a task force to review the July outages and to develop a plan to achieve fundamental improvements. Helwig, who was formerly the senior vice president of ComEd's Nuclear Generation Group and has a background in both T&D and nuclear power programs, had already been recognized for introducing fundamental change within ComEd's troubled nuclear programs.
Helwig solicited EPRI's Vice President of Power Delivery Dr. Karl E. Stahlkopf to participate closely in ComEd's investigation. Stahlkopf called ComEd's assessment “the fastest, fullest, most comprehensive T&D investigation ever launched in the history of the industry, taking a blunt look at equipment, design, personnel and operations.”
One of the most critical imperatives was to map out and identify the nature and extent of the most serious and time-sensitive challenges, and to do so quickly. During the first 10 days alone, ComEd employees inspected virtually every one of ComEd's 888 substations.
At the same time, Helwig assembled a team of experienced experts to assess the operation and management of ComEd's T&D system, drawing extensively on the technical expertise of EPRI and consulting with such industry leaders as General Electric, Kenny Construction and Asea Brown Boveri.
The Findings
In truth, the reliability hole that ComEd found itself in at the end of summer 1999 was dug over a long period of time. The findings were sobering. Substation and feeder capacity planning and capacity additions had not kept up with the heavy load growth spurred by Chicago's booming economy. The problem was not a lack of power; the problem was that the distribution system could not reliably deliver the power at peak times.
To make a bad situation worse, through the years the company had neglected routine maintenance in favor of other projects. The review team found that other major cities — operating T&D equipment not newer, not older, not fundamentally different from ComEd's — did not suffer the same breakdowns. The critical difference was rigorous care and maintenance. All in all, ComEd faced overloaded feeders and substations, inadequate capacity and redundancy, and poorly maintained equipment, exacerbated by weak planning. The system was cracking under the pressure.
The Recovery Plan
The findings were assembled in less than two months and made public in a 450-page recovery plan, the September 1999 Investigation Report. A blueprint for change, it set detailed criteria for a staggering array of tasks, projects and process improvements pledged to be completed before summer 2000. Physical improvement plans included more than 330 distribution feeder installations and upgrades, 27 large substation transformer upgrades or expansion projects, transmission line inspections and repairs, and the start of an extremely aggressive multi-year improvement program for key Chicago substations. It also called for new discipline and accountability, especially for T&D maintenance.
The Recovery, Summer 2000
Carl Croskey, then distribution group president, was charged with addressing deficiencies in system maintenance that had brought about the crisis. For example, to complete a comprehensive aerial inspection of the overhead transmission system, the company used the helicopter services of Haverfield Corp. The flights revealed more than 2700 critical deficiencies ranging from broken insulators to missing support pins to floating static wires.
Outage analysis in autumn 1999 showed that falling trees and branches were the cause of some 17% of outages. Thus, a key element of ComEd's new maintenance program included an exclusive partnership with Asplundh Tree Experts who took over all tree-trimming responsibilities. By May 2000, Asplundh had reduced the trimming cycle to four years, dramatically impacting the number of tree-related outages (Fig. 7).
ComEd also recognized that not everything would or could be completed by the summer of 2000. The company had to implement plans that went beyond immediate equipment upgrades and maintenance programs. To ensure that additional stress was not put on areas where improvements could not be accomplished for summer 2000, ComEd planning engineers joined efforts with the sales force to procure curtailable load. Surpassing its goal of more than 1200 MW, this targeted load-curtailment effort saved the immediate need for some upgrade projects. Ultimately, because of favorable weather and proactive load management, ComEd did not have to call for any system-wide curtailment programs during the summer of 2000.
To protect transformers that were not part of the 2000 improvement plan, additional monitoring was installed to identify potential degradation. The improved monitoring paid dividends. During 2000, newly installed transformer-monitoring systems sent alarms that triggered immediate and proactive inspections. The resulting maintenance was credited with saving imminent failure on no less than five occasions.
The Success
Manpower made a critical difference. ComEd employees worked an average of more than 60 hr a week, thus completing most of the distribution system and substation maintenance projects, along with smaller upgrades (Fig. 3). They were joined by more than 2100 contractors in a sustained partnership to complete the balance of the work, which included installing conduit, pulling cable, performing distribution feeder upgrades, completing substation projects, trimming trees, performing new business hookups, finishing overhead transmission maintenance repairs, and foundation and concrete work.
Other partnerships and alliances also were key to the turnaround. For example, GE/Harris provided turnkey projects for equipment, monitoring and relay upgrades while S&C Electric led a team that installed more than 100 pole top automated switches on the 34-kV system.
Of major concern was the growth of Chicago and the company's ability to support the energy needs of the city. The 2000 plan centered on six Chicago substations known as the “six-pack”: Northwest, Diversey, Lakeview, Kingsbury/Ohio, Grand and Jefferson. The most extensive modernization project accomplished was at the Northwest Substation, supplying power to more than 82,000 customers. An entire 12-kV substation was rebuilt over the top of the old one, two 75-MVA transformers were added and other upgrades were completed.
For Diversey, the challenge was even greater. An entirely new substation was erected on an urban site that had been commercial-use land as late as November 1999. Experts predicted construction would take two years to build the 138-/12-kV substation that would house four 50-MVA transformers, but ComEd didn't have two years. In the end, it was built and commissioned in about eight months, an almost unbelievable accomplishment of man, machine and management (Fig. 4).
The improvements on the remaining four of the Chicago six-pack included retiring the switchgear at the Lakeview substation and the conversion of feeders to 12 kV for better switching flexibility, making room for future substation work. A new GIS substation (Grand) was built to help load growth for the north end of the Loop business district. Finally, circuit breakers were refurbished and rebuilt at the Jefferson substation, the location that caused the severe 1999 outage in Chicago's growing South Loop.
ABB and Kenny Construction joined forces to complete the bulk of the Chicago six-pack projects by delivering a design, procurement and construction team able to fast track some complicated projects. Working through a minefield of equipment delivery lead times, permit approvals and ComEd workforce coordination, the team completed critical maintenance and other operations in some extremely confined urban areas.
While the fast-track nature of these Chicago six-pack projects is not the recommended course of action, the alliance team was able to deliver without significant outages or serious safety incidents. There were of course some trade-offs. The distribution system was at a greater risk because key elements were taken out of service for project cut-overs. Some customers experienced outages because of cable dig-ins, and cable failures caused more widely spread outages because many circuits were switched abnormally for construction purposes. Ultimately, risk mitigation and outage coordination were critical to the success of the projects.
Comprehensive contingency plans were created to address other areas where high loads were projected. ComEd secured temporary and portable generators. The centralized Distribution Dispatch Center (DDC) made load forecasts and called for load switching and generator deployment on a day-by-day basis. During the summer of 2000, the DDC did an impressive job of proactively managing the load forecasts with switching steps, thus avoiding any widespread generator use.
ComEd again found itself under the gun when a sidewalk network vault roof collapsed in July 2000, triggering a fire and shutting down power to three high-rise buildings in Chicago's downtown. But in less than an hour, generators and emergency personnel were effectively deployed. In sharp contrast to summer 1999, city leaders joined business owners in acknowledging ComEd's quick and professional response. Chicago Environment Commissioner William Abolt hailed ComEd's response to the fire as substantially better than previous years. He told a Chicago newspaper, “We were really pleased that the first real test of the new emergency plan worked.”
More than Equipment
For the T&D turnaround to succeed, John Rowe also demanded a complete overhaul of the company's communications efforts. In 1999, the mayor, the media and other critics described their acute frustration in getting information quickly and accurately. ComEd responded with a new plan to enhance communications with government leaders, the media and the Illinois Commerce Commission (ICC) to keep them informed about project plans, progress and outages. ComEd also established a timely communications process for informing the public when outages occur, what restoration efforts are underway and estimated times of when power will be restored.
The city of Chicago used Harza Engineering as a third-party overseer to act on the city's behalf and provide objective expertise about the reasonableness and timeliness of ComEd's turnaround efforts. This innovative partnership enabled ComEd to help regain credibility with stakeholders throughout its service area. Today, ComEd provides detailed monthly updates to the ICC and the city about work progress and system performance, a practice of continuous, bare-knuckled scrutiny that is said to be the most extensive public reporting system of any electric utility in the nation.
Fewer, Faster, Better
ComEd continues to refine its organization and enter long-term alliance partnerships for engineering and construction support. Helwig's T&D plans for 2001 are as aggressive as for 2000. Today, the company has begun to climb out of the reliability hole. The critical atmosphere has somewhat dissipated. There is ample evidence that the upgrades, maintenance and new construction are showing the kind of measurable results Rowe demanded. The frequency of outages has decreased more than 38% since December 1998 (Fig. 5). The duration of outages has decreased more than 46% for the same period (Fig. 6).
Each time the company makes another deadline, fulfills another commitment or answers a customer's question, another step is taken out of the hole. More hard work is left but ComEd's efforts to date clearly demonstrate that key partnerships, innovative risk taking, and unyielding corporate focus are bringing about the successful rebuilding of both the power delivery system and the confidence of its customers.
Carl Segneri is the vice president of substations and transmission for ComEd Energy Delivery. In his 20 years at ComEd, Segneri has managed construction, engineering, transmission design and operational analysis. He was also regional distribution leader for the Chicago region, overseeing the inspection and repair of facilities identified as crucial to the proper functioning of the system. In addition to work in T&D, he performed engineering testing work at Dresden Nuclear Station and Will County and Collins fossil generating stations. Segneri holds the BSEE degree from the University of Notre Dame. He is a member of the IEEE.
About ComEd
Commonwealth Edison (ComEd), a unit of Chicago, U.S.-based Exelon Corp., is one of the nation's largest electric utilities with nearly US$15 billion in revenues. ComEd provides service to more than 3.4 million customers across Northern Illinois. ComEd's customer base consists of more than 3 million residential customers and nearly 300,000 commercial and industrial customers located in Northern Illinois' six counties. With 8000 employees, ComEd designs, operates and maintains more than 75,000 miles (120,701 km) of electric transmission and distribution lines.
Executive Commitment Remains Steadfast
David Helwig's mission was expanded in late 2000 to executive vice president of energy delivery operations, responsible for running the entire T&D organization. Today, David Helwig and ComEd Vice Chair Pamela Strobel are continuing efforts to lead one of the most comprehensive utility turnarounds ever. And, as summer 2001 approaches, it appears the combined efforts of new leadership, outside experts and the massive investment of resources and planning are paying off with substantial reductions in the frequency and duration of outages. Still, ComEd cannot promise a summer free of outages. “What ComEd does pledge,” Stroebel told the Chicago city council, “is fewer interruptions, faster restoration and better communications.”
