What parallels can be drawn from the golden age of arcade games to identify modern threats and risks in Operational Technology?
Just like Industry 4.0 has brought new life and technology into our industry; the golden age brought new life to entertainment.
The golden age of arcade video games was the period of rapid growth, creative and technological development and cultural influence throughout the late 1970s to the early 1980s. Made possible by new computing technology with greater power and lower costs, arcade video games revolutionized the entertainment industry.
Arcades were full of different arcade games, from different vendors, different ages, and different maintenance requirements. This is no different than your water treatment facilities. In both industries there is often missing, limited or obsolete documentation.
Arcades – just like water treatment plants – must stay operational. For arcades this is a financial requirement, while in treatment it’s a matter of safety.
At the 2022 BCWWA Annual Conference & Trade Show, ICI presented “The Golden Age of Arcade Games and Your Water Treatment Plant”. This presentation highlighted the similarities in operational requirements and roadblocks between these two industries with the following key points:
- Keep it simple! The Simplicity of Pong
- Avoid knowledge silos
- Implement defense in depth within your facilities
- Operational Technology Priorities. Availability comes first, then integrity, then confidentiality
The simplicity of Pong
Pong was originally released in 1972. The game was basically a digital version of ping pong, each player had a digital paddle and was required to bounce the ball past their opponent’s paddle – the first player to reach 15 points wins.
Pong was an amazing success and was one of the major pioneers of the video game industry. A big part of this was due to how simple the game was to use. Some games, such as The Adventures of Robby Roto! had less commercial success because they were too complex to learn quickly, and others like Star Fire because the game play was unfamiliar to the audience or not intuitive.
Simple interfaces allow for ease of play; no tutorials, no big book of instructions. Just a simple analogy and objective. This is an important parallel for us to draw with any interface we interact with.
HMIs and SCADA system should be treated the same way as pong. An operator with minimal instructions should be able to interact with and know what is going on within their SCADA system. When modernizing operator interfaces, we utilize workflows based on the ISA 101 Human Machine Interfaces standard, and the High Performance HMI Handbook.
As a baseline a typical SCADA system developed with high performance graphics should have
- Standard colour scheme
- Ability to navigate to any page in around 3 clicks and
- Require limited instructions to understand the graphics
The Darrel Dilemma
The year is 1982 and Darrel is the manager of the local arcade.
The arcade has 22 different games: from 5 manufacturers from 2 countries. These games all have different maintenance requirements as there was no industry standardization. The average life span of these arcade games was 4-6 months which keeps Darrel busy. His job is to keep everything operational. He is the only one that knows how to unjam the space invaders game. He is the only one that knows where the spare parts are Hidden. Darrel is busy. He hasn’t even heard about the new Donkey Kong game that’s been released that all other arcades seem to have.
Is there a Darrel at your plant? Is he overwhelmed?
Darrel could be a part of your operations team; he could also be your systems integrator. It can be detrimental to your operations to have only one person who holds the keys to your facilities. Having personnel cross-trained whether it be on your team or on your integrators’ team can improve business continuity. The other part of the Darrel dilemma is documentation. Ensure you have all your procedures documented in some way, so you can support one another and not be overly reliant on one person or integrator. This is for your benefit as much as it is for them.
Passwords, Policies and Pac-Man?
Pac-Man is one of the best-known games of the “Golden age of arcades.” You are in control of a little yellow circle and must eat smaller yellow circles and fruit while keeping away from ghosts in a big maze. As the you progress through game different parameters change which increases the difficulty.
If the maze were to be smaller the game will be harder, if the maze is bigger, the game would be easier as you can more easily avoid the ghosts.
This is fairly similar to protecting your control system. If you add more obstacles between your PLC and the outside world, it will make it hard for a threat actor to compromise your system. This is a concept known as “Defense in Depth”.
Priorities of Arcade Games and OT
- Operational Technology: 0 or 1 can affect the physical world
- Informational technology: Moving data from one place to another
- Availability: Ensuring your system is online, stable, and ready to function
- Integrity: Ensuring your data is accurate, and verified as accurate
- Confidentiality: Ensuring your data is protected from unwanted/ unauthorized access
Availability -> Integrity -> Confidentiality
In Operation Technology (OT) Control systems, we look at three priorities: availability, integrity, and confidentiality. With the priorities being in that order, while in IT systems the priorities are opposite.
Aside from “game over” one of the most tragic things to see in an arcade was an out of order sign. This can be just as upsetting to the customers as the arcade owner. When the machine is down it is not making money. Similarly, in your water plant, if your pumps are not operational, you are not moving water.
Having all this information documented and in mind can help you plan for the worst-case scenario that may arise.
There are many steps we can take to ensure our plant stays online. This can include things like redundancy, maintenance and upgrades inline with the corresponding equipment’s replacement lifecycle.
In many games the line is not clear between what is a glitch and what is a feature or “easter egg” as they have been called. In Pac-Man for example, after a certain level, half of the maze is covered with weird symbols. If you beat this level, you can keep advancing through the game infinitely.
Now this sounds like a glitch to me, but how can a user tell if the game is working properly or if it is glitching?
When we look at integrity of our system, we can look at protecting against malicious threat actors but there is also accidental damage or malfunctioning that we need to be concerned with.
Arcade games are known for their high score board. Who wouldn’t want to be on the top of that list. This is a form of documentation. In terms of confidentiality, we don’t care that everyone can see that we got the top score for Space Invaders.
Figure High Score Board
In OT control systems, confidentiality is the lowest priority of the past three topics. Although you might not want to public post what your reservoir level is, it probably won’t be that big of deal if that information was released.
More on documentation and high score boards
The high score table isn’t just the number, you also are able to put your initials up there so everyone can see that it was you that did it. This is the arcade game’s form of process documentation. Similarly, in your SCADA system we can track our process’s progress. Information like what has been changed, when was it changed, and by who can be super useful tools when troubleshooting an issue or investigating why decisions were made.
Process documentation for your system can include:
- Your historian and trends
- Back ups of your programs/ and computer
- Setpoint change log
Jason Marchese, P.Eng. PMP
Director of Engineering
- Winkler, I., Gomes, A. T. (2017) Advanced Persistent Security
- Kent, Steven L. (2001) The Ultimate History of Video Games
- Hansen, D. (2016) Game On! Video Games History from Pong and Pac-Man to Mario, Minecraft and More.
- Dragos ICS Cybersecurity Year In Review 2021 (https://www.dragos.com/year-in-review/)
- ISA 62443 IC32
- CISA Alert AA22-103A (https://www.cisa.gov/uscert/ncas/alerts/aa22-103a)
- Isaacson, Walter (2014) The Innovators
- Arcade Game Systems Timeline (https://www.sutori.com/en/story/arcade-game-systems-timeline–dJQ6jbZYtbrJDLzv7mKTcHvN)