The substation visibility problem

When power, environment, security, and communications each report somewhere else

If a remote substation starts to fail overnight, which screen tells you first?

For most utilities, the honest answer is that it depends on what fails. A power problem shows up in one place. A temperature problem shows up in another. A forced door lands in a third. A dropped communications link may not show up at all, because the link that would have reported it is the one that failed.

We call this the substation visibility problem. It is not a shortage of data. Remote substations produce plenty of signals. The trouble is that those signals are split across systems bought at different times, for different reasons, by different teams. Some are Information Technology (IT) systems. Many are Operational Technology (OT) systems. Almost none were designed to describe the site as a whole.

The result is a site that is well instrumented and poorly understood. Each system knows its own piece. No system knows the story, and no one is assigned to read all of them together.

That matters most at night and at remote sites, where one dispatcher may watch many substations and the people who understand a site are not standing in it. We write this for the operations leaders who own that gap.

This article looks at where those signals go, what the split costs in time and money, and what it takes to see a remote site as one operational picture. The industry figures we cite come from the public sources named at the end. Where we tell a story, we label it as one.

Every system reports somewhere else

Walk through a typical remote substation and list what reports on its condition.

Power comes first. The batteries, chargers, and supplies that keep protection and communications equipment alive usually report to their own controller or a local panel. Environmental conditions come next: temperature and humidity in the control house, and whether the cooling is actually running. Those readings sometimes feed a building system and sometimes feed nothing at all.

Physical security is a third stream. Door contacts, gate sensors, and cameras typically report to a security platform that a different team owns. Fire and safety add a fourth, with detectors and alarm panels that often report to a panel of their own. Communications is the fifth. The routers, radios, and links that carry everything else back to the control center have their own management tools.

Each of these has a reasonable owner and a reasonable reason to exist. Taken one at a time, nothing is wrong with any of them.

The trouble is what happens between them. A real fault rarely respects the boundaries between systems. When a single fault touches more than one domain, no single system sees the whole event. Each one raises its own alert, if it raises one. Each alert lands in a different inbox, on a different screen, in front of a different person.

Those people are then reading symptoms. Someone has to notice that several unrelated messages describe one problem. That someone is usually the person with the most context and the least time.

An illustrative night at a remote substation

Illustrative scenario. This is not a documented incident. It is a constructed example, built to show how the gap works.

On a warm night, the cooling unit in a remote control house stops running. Nothing alarms. The unit reports to a local indicator, and nobody is looking at it. The fault is invisible to anyone who is not standing in the room.

Over the next hours the room warms. Equipment in the communications cabinet begins to struggle. The link back to the control center turns unreliable, then silent. The telemetry that would have flagged the problem is now the telemetry that is missing.

Each system records only its own piece. The temperature sensor logs a slow rise to a local record. The network tools register a link down, which looks like a routine communications fault. The power equipment keeps running. Nothing on any single screen says cooling failure.

Eventually a dispatcher notices that data from the site has gone stale. A call goes to the field crew. A truck rolls. The crew arrives, finds a hot room and a dead link, and works backward from the symptoms to the cause. The repair is quick once the cause is known. Finding the cause is what takes the time.

Nobody in this story made a mistake, and no tool failed to do its job. Every system did what it was built to do. The site was simply never viewed as one thing. The first signal anyone acted on was a person noticing, not an alarm.

Why the gap is measured in hours

That ending is not a freak event. Without active monitoring, a fault typically surfaces when someone reports it, 60 to 120 minutes after it begins. With proactive monitoring, it surfaces as an automated alarm in under 5 minutes.

Those figures describe remote sites in general. Your own substations will differ, and the difference is worth measuring. But the shape of the gap is the point. One path depends on a person noticing. The other depends on a system that was built to notice.

The time matters for a practical reason. A fault that is caught early is still the fault it started as. A fault that waits for a report has time to spread. In our scenario, a cooling problem had time to become a communications problem before anyone knew either existed. The longer the wait, the more the crew has to untangle, and the more the first question changes from what is happening to what happened.

It also matters because of where the signals sit. When power, environmental, security, and communications data each report somewhere different, correlation is manual. A person has to carry information between screens and decide that two events belong together. That works during the day with a seasoned operator. It works less well at night, at a remote site, with one dispatcher watching many.

This is why we say the problem is not a shortage of data. The data exists. It is simply not in one place, and it is not read together as one picture.

What the gap costs

The cost of waiting has two parts. One is the outage. The other is the response.

A typical remote site has 1 to 2 major outages a year, according to GSMA and Ericsson. The cost runs $2,000 to $5,000 per site-level outage, and $300,000 to more than $1 million per major hub or switching outage, according to FCC NORS and the Uptime Institute. Across ITU-T L.1310, ATIS NRSC, and FCC NORS, the finding is that 40 to 60 percent of site-level outages are preventable with active monitoring.

Dispatch is the visible part of the response. A truck roll costs about $400, according to Frost & Sullivan, and about $305 to $675 in dense urban conditions. Most of what this costs never appears as a single line item. It appears as overtime, as lost time, and as a crew sent to a site to find out what went wrong. The scenario above ended in a truck roll. An earlier alarm will not prevent every dispatch. It lets a dispatch be a decision instead of a reaction.

Here is a worked example. Take 50 sites, with 1.5 outages per site each year, 50 percent of them preventable, at $3,000 per outage. That is 50 x 1.5 x 50 percent x $3,000, or about $112,500 a year in avoidable cost.

That is arithmetic, not a forecast. Change the inputs and the answer changes. The industry sources behind these figures cover remote sites broadly, so we offer them as a way to frame the question for your own substations, not as a measurement of them. The useful exercise is to replace our inputs with yours.

What one operational view takes

Seeing a substation as one thing does not require replacing everything in it. It requires reading what is already there and bringing it together. A few practices make that workable. These are good practices, not requirements from any named standard.

Read the equipment you already have. Substations are full of devices that speak industrial protocols. A monitoring layer should connect to them without custom device drivers or APIs, so the project is not held up by integration work for each vendor.

Treat IT and OT as one picture. Network equipment and OT signals should land in the same operational view, so an operator can see a communications fault next to the power or temperature event that caused it.

Watch every domain, not just one. Power, environmental conditions, fire and safety, physical security, and network infrastructure each tell part of the story. The value comes from reading them together.

Secure the monitoring layer itself. A platform that watches critical infrastructure is part of that infrastructure. Role-based access control, encrypted communications, closed non-essential ports, and configuration compliance are practical starting points. SELinux enforcement is available as an option.

If you want to test your current setup, start with five questions.

1. When a fault begins at a remote substation, how does it first reach a person?
2. Which systems report on power, environment, physical security, fire and safety, and communications, and do they report to the same place?
3. If the communications link fails, what tells you, and how quickly?
4. Can an operator see one event across more than one domain without moving between screens?
5. Is the monitoring layer secured and reviewed with the same care as the equipment it watches?

IMCI Technologies is The Critical Infrastructure Protection Company. Our Open-i devices and IRiS software bring IT and OT signals from remote sites into one operational view, with up to 192 inputs in a single 1U chassis. Our technology has been in market since 1997, and our hardware is built for a 15-year MTBF.

References

1. ITU-T L.1310, ATIS NRSC, and FCC NORS: preventable site-level outage share.

2. GSMA and Ericsson: major outage frequency at a typical remote site.

3. FCC NORS and Uptime Institute: site-level and hub or switching outage costs.

4. Frost & Sullivan: cost per truck roll.