After crews left for the day on a Friday evening, about 40,000 — or 5% — of TECO’s customers lost power due to extreme weather conditions. To call line crews back to work, the utility used an automated alert system. About 150 line workers restored power overnight, and by 5 a.m. the next morning, less than 500 customers remained in the dark.
Streamlining the Call-Out Process
Back in the 1980s, dispatchers called line workers individually to confirm they could return to work. In the 1990s and early 2000s, TECO used a more automated call-out system that it created, but it was inconsistent, said Cherie Jacobs, media spokesperson for TECO.
“Sometimes the system contacted more people than we needed; other times it didn’t contact anyone at all,” Jacobs said. “Starting in 2015, we began using software to manage our workforce, and the 2018 storm was our first ‘all hands-on deck’ call-out. Within minutes, we knew who was returning to work, which truck they were in and where they were located. It allowed us to begin restoration work faster.”
From a cell phone or tablet, TECO can broadcast a message or a call to work — it can be to everyone or targeted to specific departments or individuals. Within two or three minutes, TECO knows who is coming to work and where they are heading. Every night, the utility receives an electronic report detailing who was working, which trucks they were in and where they were located.
The high-tech callout process helps dispatchers track the status of crews during normal work hours to see who can take overtime in the event of an emergency. After business hours, when a dispatcher needs a crew for an emergency, the system automatically identifies only available workers by following a utility’s callout rules exactly; contacts workers via phone and/or text, and records how every person responds.
Using software to schedule line crews has also been extremely helpful during the pandemic, Jacobs said.
“We use it during ‘blue sky’ days—not just storms,” Jacobs said. “Every day our crews are different, and it has helped with contact tracing when someone tests positive for COVID-19. It has helped to manage our workforce effectively.”
Responding to the Instantaneous Storm
Unlike Hurricane Irma, which TECO battled a year earlier, there was no warning for the November 2018 storm. Consequently, no crews had been held over as is the case when expecting bad weather.
At 3:15 p.m., TECO received the first alert that the storm could be problematic. At 3:21 p.m., a wall of red and orange stretched across radars and red alerts began going out. By 5:15 p.m. the violent and sudden storm had passed and 38,972 TECO customers were out of power. At 5:47 p.m., with an automated callout, TECO issued an all-hands call for every line person – about 40 crews.
As TECO launched its callout, Ronnie Crossen, senior technology analyst for TECO, worked with Luke Diruzza, manager of distribution engineering and operations, and his team to tap into crew management system’s virtual dashboard to immediately begin modeling the best response based on where each crew was coming from and other projects planned for unaffected service territories within the TECO system. In the meantime, the callout system successfully reached 254 people with 101 arriving on the property in less than 30 minutes. Additional linemen began filing in soon afterward.
Crews reported to their normal work location, and Diruzza, Crossen and supervisors across the territory were able to collectively log in to the crew management tool to create different scenarios in seconds by moving crews around the virtual board to see how each movement of crews had a positive or negative chain reaction on restoration time, other service territory’s workloads and the availability of trucks and equipment. Each move of the crews was done to fine tune the best, safest, and fastest restoration response.
“A lot of people need this information, and at many levels,” Crossen said. “Automating crew management makes for a centralized location for everyone—executives to lead line supervisors—to extract the information they need. Hours would’ve been spent trying to do this modeling manually, no question.”
TECO was able to find resources down to an individual with a special certification (e.g., lead electrician, search-and-rescue, or a Doble-certified worker) and drag that person’s icon together with a crew who needed the specialty and immediately launch the request to pair them up.
“By automating our resources, we can store all those skill sets as well as the real-time location of the person or crew,” added Crossen. “We might not call on someone with a certain specialty for six months; with a manual system, you can misplace that record of what a person can do and forget where they’re located. By automating and storing our crew data, we can remember and put at our fingertips everything and everyone to work the storm.”
By 8:30 p.m., everyone, including contractors, were in place to bring back power. Throughout the restoration, storm management at TECO could track progress on each circuit with tracking icons on its virtual map of crews’ progress and total hours worked. Knowing how long each crew had been working at any moment helped TECO plan for any potential rest that might be required without sacrificing momentum.
Lessons Learned
The crew manager process enables TECO’s managers to forecast crew and equipment needs hours (or even days) in advance. In addition, the system allows managers to record and play back scenarios after the restoration event. That playback is akin to a sports team watching film from its last game to analyze how well it performed.
Crew management increases the accuracy of response and gives the utility more lead time in identifying which resources it might need because it can visualize any number of restoration scenarios on its digitized planning board, in seconds.
Each step TECO took — every callout and crew-building decision — was recorded for either playback after the fact as a best-practice sharing or a report on the outcome for the Public Service Commission (PSC). If the PSC wanted to know how many callouts had been made, how long the response took or how many crews were engaged, TECO not only had the information but also could serve up the data in a report within minutes.
Within seven hours, TECO had restored 20,000 customers by relying on its automated callout and crew management system. And by 5 a.m. the next day, all but a few hundred TECO customers had power restored. Without an automated system, TECO estimates the restoration would’ve taken at least 24 hours instead of less than 12. With a storm of this scale, there were also contractor resources, so cost-recovery efforts and accounting would’ve taken weeks of administrative time.
Capturing data about the number and make-up of crews, the nature of their work and how long their activities lasted is a linchpin for supporting a utility’s cost recovery, according to TECO. Whether a utility is a cooperative, IOU or part of a municipality, managers are generally capturing the same data.
An IOU, like TECO, will have a private carrier to which managers will submit data such as crew counts, amount and type of equipment damaged (and replaced) as well as the number of hours crews worked. Government agencies, regulators and utilities want a precise accounting of what it takes to restore service after a storm, especially a major event. Utility executives look at the impact of restoration on their balance sheet. TECO had a well-documented, minute-by-minute account of the crews, resources and equipment tapped for restoration and delivered electronically to whomever required the data. That kind of information also created a systematic way for storm managers to request, manage, track, and release external resources.
After the storm, TECO replayed the steps taken to understand what was done well and where improvements could be made. The replays go up and down the chain of command and, when necessary, get implemented as an improvement for the field. In the case of this storm, execution was nearly flawless. Managers noted some callouts were not accepted immediately and this related to employees failing to update their information within the system as well as extenuating situations in which the employee wasn’t able to respond to a call or text.
Reflecting on Accomplishments
Since 2015, TECO has achieved a 5,500-hr reduction in annual callout time, and completing a callout has dropped from 25 minutes to six minutes.
“We’ve had callouts for five workers after hours that we completed in 43 seconds,” Diruzza said.
Load growth has risen by 2% for TECO, and managers are working to keep headcount stable. TECO’s system is playing a part in making operations efficient enough to manage more work without increasing the number of crews.
As of 2020, the average callout is less than six minutes to assemble a four-person crew. Since 2015, there has not been a grievance about the callout system. TECO can also keep an eye on callout acceptance rates if they fall below 50%. Managers then coach workers back to (or to exceed) the 50% goal.
The system has also eliminated calls into the dispatchers by line workers looking for details about a callout; instead, the callout process automatically describes the nature of the callout and location for lineworkers. This, said TECO, has helped its dispatchers focus on storm restoration and develop plans for multiple circuits, instead of fielding those calls.
“Routine and predictable work can go to our contractors,” said Diruzza. “Our automated approach to crew management records what type of callout is happening, so we can discuss why choices were made and improve if necessary whether that’s daily operations or a major event like the November storm.”
Amy Fischbach ([email protected]) is the Field Editor for T&D World magazine.
