Imagine walking through a neighborhood wearing a mixed reality headset. Look up, and you can pinpoint the voltage of the overhead infrastructure. Shift your gaze downwards, and you can see the underground power lines running beneath your feet.
While this technology may seem like science fiction, it is commercially available today to those utilities with a robust geographic information system (GIS). For example, Toms River Municipal Utility Authority (TRMUA) in New Jersey worked with Meemim Inc., a Toronto-based firm, to combine its ArcGIS platform from Esri and the Microsoft HoloLens headset into a mixed reality application, which combines virtual reality with real-world imagery.
The technology doesn’t give the user X-ray vision, and users can’t see the utility lines with the HoloLens unless the data is loaded in a GIS. The better the GIS data, the more accurate the application, says Len Bundra, GIS/IT director for TRMUA.
In addition, he says it is not the same as virtual reality. Instead, it can give field workers both views simultaneously without obscuring their vision for safety reasons.
“Virtual reality doesn’t work in the field because it obscures your vision, and that is definitely a safety issue,” Bundra says. “It has to be mixed reality. With the HoloLens, you can see through the glass and see everything around you, and you won’t trip over a curb. You can see utility lines below your feet because it’s half real and half virtual.”
Creating a Mixed Reality
The history of this program can be traced back to 2013, when Google Glass came out with the first version of its technology. At that point, Bundra had an “aha” moment. He envisioned an application in which he could export GIS data and view it with the Google Glasses.
Eventually, the Google Glass technology fizzled out, but Bundra did not give up on his idea. He created a post on LinkedIn, and eventually, Alec Pestov, CEO of Meemim Inc., responded and suggested that he try Microsoft’s new HoloLens technology.
With expertise in a program called Unity, Meemim’s programmers discovered a way to use the software, which is commonly used to create video games, to work with coordinates in the real world. Bundra and Meemim brought GIS programmers and Unity programmers together to work on a solution.
“Game programmers are used to working on one room at one time with four walls and a monster,” says Bundra, who has worked at TRMUA since 1994 and has served as a GIS consultant for public and private utilities. “We asked them to use Unity to digest an entire city, with an area of 55 sq miles and 442 miles of underground pipelines. While the developers came from different worlds, they had enough acumen to get the GIS features into the gaming world and project it into the real world using survey coordinate systems. It was complicated and took a lot of work from both sides.”
Meemim and TRMUA spent eight months working together on the technology before it could be tested in the field for the pilot project. Because the TRMUA already had its GIS data online through the Esri ArcGIS platform, Meemim could upload it to its Cloud service, digest it, convert it and deliver it back down to the HoloLens in real time. “It was no simple task,” Bundra says. “It took a while before I could put on a HoloLens and see GIS features projected accurately.”
One of the key benefits of the technology is the ability to perform more accurate markouts, Bundra says. In the past, TRMUA worked with a third-party markout company, but today, its internal workforce handles this task. “They would spray paint where they thought the line was, and if they did it wrong, a utility line could be hit during construction,” Bundra says. “A simple dig would turn into a complex issue.”
Bundra says he has 100% confidence in the markups done by TRMUA’s field crews. In the future, however, Bundra hopes that the technology will help the markout companies to improve efficiency, and even more important, increase accuracy. “If they are wearing a headset and spray painting the lines, the markout would be a lot simpler,” Bundra says. “All they would have to do is paint the road where they see the underground lines in front of them.”
By extending this technology to the electric utility world, it could save lives, says Bill Meehan, director of utillity solutions for Esri. “Digging into a water main is a problem and could be dangerous,” says Meehan, who used to work for a power company in New England. “Digging into an electric line is deadly.”
Today utilities use a variety of methods for locating underground facilities, Meehan says. For example, they are sending a signal down electric lines and having a receiver in the field picking up the signal. They are also relying on methods such as sonar to locate metal underground infrastructure, but this approach becomes tougher for plastic pipes. More recently, companies have been burying RFID tags on their facilities. “It’s a tough problem since utilities have been burying their facilities for decades,” Meehan says. “In the past, utilities have not always clearly documented the location of their facilities.”
Compounding the problem, sometimes it’s nearly impossible to find assets because of asphalt paved over manholes or snow-covered switches or transformers. During floods and landslides, electric and gas equipment may be hidden from view. “The key is a good, accurate GIS so the user can see the equipment based on its location within the GIS,” Meehan says.
At this point, however, the technology faces one primary obstacle to widespread adoption — getting all the utilities within a city to share GIS data and get on the same page. “The HoloLens and the projection of the GIS data would be more powerful if all utilities within a city share their data,” Bundra says. “That way, we could see not only our utility lines, but also the underground electric, gas and telecom infrastructure.”
This approach would offer two primary benefits: a higher return on investment on markouts and more successful damage prevention. TRMUA and two private water companies are already working together, but they would like to get others on board as well. Bundra is hopeful that this project will serve as the impetus to bring disparate utilities to the same table. “If they look through the HoloLens app, they will immediately see how this technology and data sharing would benefit all parties involved,” he says.
Utilities can share GIS data in multiple ways. For example, they can request the data in digital form, coordinate it with their own data, and then display the data together. Next, they can perform a spatial analysis to determine if a facility is in the way of a proposed dig. This approach, however, could take days or weeks. Also, even if an electric utility does get a file transfer from another utility company, the data may be out of date.
To help solve this problem, Esri created a collaboration framework through its ArcGIS system. Using this platform, a water company could publish a web service of up-to-minute GIS data to be consumed by the electric or gas company. “This process would take seconds,” Meehan says. “The savings in labor and time would be significant, however, the prevention of damage and injury is of much greater value.”
Today, TRMUA’s GIS is stored in Esri’s ArcGIS Online Enterprise Service, which publishes a web service of underground utility infrastructure. If different utilities published their data the same way, then each utility would have its own GIS layer for display and analysis. For example, the app could display the message, “Watch out! The electrical conduit is only 6 inches away from the gas service.” And illustrate the area where a potential hazard would exist.
“Think of these things as layers on a map that can be turned on and turned off,” says Bill Meehan, head of Esri’s utility industry solutions practice. “The mixed reality app would then consume these services and present them to the worker.”
Helping Line Workers
While the TRMUA is conducting a pilot test of the technology for its underground main lines, Bundra says this application is even more exciting when it comes to applications for an electric utility. Since he says many electric utilities already have excellent GIS platforms, they are already halfway there when it comes to mixed reality applications.
From a GIS perspective, nothing is more complicated than electric utility infrastructure because of the sheer number of components on a typical pole, he says. Using mixed reality, however, helps give a line worker a way to make a quick situational assessment, especially in an emergency situation.
“If a car hits a pole, there may be downed wire, a car leaking fuel and a storm sewer inlet close by,” he says. “In this scenario, the fire, police, emergency responders and utilities will all be called on the scene. When they arrive, there will be seven different individuals sitting in seven different vehicles looking at seven different GIS systems on their laptops to try to figure out what is going on.”
With the mixed reality technology, responders can get out of their vehicles, strap on the HoloLens, look down and view the red electric lines, yellow gas lines, green sewer pipes and orange telecom lines. Looking ahead, Bundra envisions that possibly a County Office of Emergency Management could encourage different utilities to share their GIS data to make this simulated situation a real possibility one day.
When that happens, utilities can help each other, by discovering inaccuracies in each other’s GIS data. “If we were all seeing the same data, it becomes fairly obvious how many errors and inefficiencies we would solve,” Bundra says. “We would constantly be self-correcting each other, and improving all GIS data layers, within a given municipality.
Esri’s ArcGIS Online platform is making it simple for organizations to share their critical infrastructure information. The platform uses what Esri calls “Web GIS.” This innovative technology uses web services to share interactive maps to literally anyone authorized to view, edit and analyze the information. That means if a water company wants to share its water infrastructure with other organizations within in a city, it simply publishes a web service. If the gas and electric companies do the same, then each of them can view the other utilities, as if it was contained within their GIS. When the electric company makes a change in their GIS, the water company and gas company see the change immediately. “This concept is similar to today’s social medial platforms like Facebook,” Meehan says. “Someone shares a picture, then all friends see it.”
By applying this technology to the electric utility world, utilities can help to improve the productivity of the field workforce, whether they are working on underground or overhead infrastructure, Bundra says. “If line workers get out of their trucks in a subdivision off a main road, not only can they look down and see the color-coded underground electric lines running to the pad-mount transformer, but they can also look up and see the different phases and voltages of the overhead lines.”
At this point in time, many electric utilities are still using paper maps in the field. Mixed reality, which is part of the broader augmented reality category, can help utilities to improve communication between their field and office workforce. Case in point: through the ArcGIS platform and its web services, a lineman could take a photo of an asset in the field, and it can be immediately viewed on a dashboard in the office. “Extending this to mixed reality means that the field worker can be sharing the results of their field observation to anyone who is using the platform,” Meehan says.
By using mixed reality headsets, enabled by the ArcGIS platform, utility workers could visualize and analyze all of the other utilities beneath their feet, in relation to their own.
The technology exists, but there are barriers. The parties must agree to share their data using the platform. The data from each technology must be accurate and up to date and based on a consistent coordinate system. Yet the benefits to all would be substantial. The data sharing would not just be confined to utilities. Utility data can be shared with street departments, police and fire and emergency management agencies simply and easily. Seeing where utility lines are during a flood or on fire can alert first responders in the field to potential dangers.
Improving the Technology
By wearing the HoloLens headset, a line worker can not only see the power lines, but also other components such as fuses, secondary hot drops and insulated wires. An electric utility can configure the GIS so that certain elements are color coded a certain way or specific symbols stand for different components. “You could take an electric company’s GIS and make it happen,” Bundra says. “We did a lot of the hard work at Toms River, that doesn’t need to be redone, with each new implementation.
While the first technology that TRMUA tested in the field was hands-free, Bundra says it could also be used on a smart phone or tablet if a field worker prefers. Over time, Bundra says he expects the technology to continue to improve. “I compare the headset hardware available now, to a Sony Walkman in 1979. Back then, it was a huge brick, and over time, it got smaller, lighter, faster and better.”
To help make improvements to the technology, Michael Tesch a senior foreman for the TRMUA tested the HoloLens in the field. With more than 20 years of field utility experience, he made suggestions to refine the solution. For example, he stressed the need for improved locational accuracy, a different user interface for light versus day and a special device to wear the headset with a hard hat.
In addition, the application has improved the graphical user interface, which now can display more details for each type of utility feature. Because TRMUA’s infrastructure often runs beneath residential backyards, the headset can help the field workers more quickly identify utilities within easements. Utility lines can be located under sheds and fences, making it difficult, if not impossible, for a field worker to access for repair and maintenance.
In the past, field workers would access vehicle-mounted laptops to find the location of these assets, but times have changed. As the utility moves to provide further testing on this technology, Bundra says he hopes other utilities — whether they are gas, electric, water or fiber/telecom — will come together. “My mission is to get all utilities to share their GIS data, so we will all be looking through the same lens,” Bundra says. ♦