Need for a robust safety culture in artificial intelligence

Dr. Rahul Remanan,
CEO, Moad Computer

"... Not one would mind, neither bird nor tree
If mankind perished utterly;

And Spring herself, when she woke at dawn,
Would scarcely know that we were gone."
  -- Sara Teasdale, There Will Come Soft Rains

This year marks the 35th anniversary of Chernobyl disaster. On 26th April, 1986, at 1:23:45 AM local time (21:23:45 UTC), the 1000 ton concrete lid located above the reactor 4 fuel elements of the Chernobyl nuclear power plant in the city of Pripyat, erstwhile Ukraine Soviet Socialist Republic (SSR), was blown open to the side by a massive steam explosion.

This explosion was caused by a prompt critical excursion of the uranium fuel based RBMK type, light water cooled, graphite moderated, channel type, nuclear reactor core. It was immediately followed by a massive secondary hydrogen explosion and a series of fires.

The secondary hydrogen explosion was due to the violent reaction between hydrogen that was accumulating in the core that went critical and oxygen in the atmospheric air that gushed into the reactor core, through the blown open reactor lid.

The build-up of hydrogen in the nuclear core occurred due to the high temperature decomposition of steam by oxidation of Zirconium. Zirconium is commonly used in the nuclear fuel cladding and it was also used in Chernobyl. The possibility of a hydrogen build-up in a water cooled reactor such as the RBMK has been a very well known problem among nuclear technology experts. Yet, the Chernobyl nuclear core design had no additional safety measures to prevent the hydrogen explosion in the event of a core integrity breach.

The secondary hydrogen explosion and the subsequent fires were due to:

1. Poor redundant safety measures such as the lack of a hermetically sealed containment structure for the nuclear core, omitted solely to cut costs
2. The use of highly combustible graphite moderators in the nuclear core
3. No automated safety measures to prevent the nuclear core from becoming highly unstable
4. Confusing regulations and operational procedures that forced the operators to take undue risks
5. The design flaws in the reactor housing itself that included the incorporation of cheaper but combustible materials such as bitumen in its construction.

This event released tons of radioactive materials such as Krypton-85, Strontium-89, Strontium-90, Zirconium-95, Molybdenum-99, Ruthenium-103, Ruthenium-106, Xenon 133, Tellurium-129m, Tellurium-132, Iodine-131, Iodine-133, Cesium 134, Cesium-136, Cesium-137, Barium-140, Cerium-141, Cerium-144, Plutonium-238, Neptunium-239, Plutonium-239, Plutonium-240, Plutonium-241, Plutonium-242 and Curium-242 to the surrounding areas. Some of these radionculides were detectable in regions as far away as Scotland, Sweden and Norway.

Ironically, the Swedish nuclear regulatory agency raised the very first alerts to the rest of the world, about the events unfolding at Pripyat. This was in no small part due to Sweden’s robust nuclear safety procedures to monitor its own nuclear industry. Unsurprisingly, during the initial days following the Chernobyl disaster, the Soviet Union and Ukraine SSR, resorted to blatant lies and propaganda, attempting to obfuscate the scale and severity of this tragic event. For a very long time, both before and after the Chernobyl disaster, the official Soviet propaganda about the RBMK nuclear reactors and their nuclear industry in-general, was that, a disaster like Chernobyl could never happen in the Soviet Union.

The prompt critical excursion event of the RBMK nuclear core of the reactor number 4 in Chernobyl was due to a combination of several factors. One of the major contributory factors for the Chernobyl disaster included a faulty physical design of the reactor.

Due to a design quirkiness called positive void coefficient, the nuclear reactor core at Chernobyl was difficult to operate at lower thermal power outputs of less than 700MW. There was also another design flaw that made it susceptible to significant thermal power surges while shutting down the reactor. These design flaws were well known to the designers of the reactor, but decided to keep these safety critical information as state secrets.

The secrecy shrouding the Soviet nuclear industry was instituted in a misguided attempt to prevent ‘outsiders’ from gaining access to the shortcomings of their industry. But, it only served the purpose of enveloping their own nuclear reactor operators in a blissful state of ignorance, that later proved to be fatal.

The smoke screen of secrecy in the name of protecting the state, served the sole purpose of causing a catastrophic event that infamously circulated all the short comings of the Soviet nuclear industry at an unprecedented global scale. The unnecessary cloak and dagger approach by the Soviet authoritarian regime, while handling the Chernobyl disaster, also acted as a significant catalyst in bringing an end to the authoritarian, communist rule in the Soviet bloc countries.

During the chain of the events that led to the Chernobyl disaster, the operators manning the reactor 4 ignored the warning signs of an unstable reactor core and attempted to run a risky safety test procedure. When things became worse, instead of attempting to stabilize the core, the operators initiated the emergency core shut-down. For the type of the reactor at Chernobyl, it was counter-intuitive to most people, including the operators of the reactor 4 at the time of the disaster, that the emergency core shutdown attempts would worsen things, if the reactor core was already in an unstable state.

Even more worrying was the fact that, when the nuclear reactor core was shutting down, there were no procedures in place to make sure that it was overseen by someone with a greater understanding of the fundamental nuclear physical principles behind the shutdown procedure. Even the chief engineer of the Chernobyl nuclear power plant had only very little understanding of the dangers of operating such an unstable design and stepped away from overseeing the safety test procedure.

The dangers associated with the powering down of the RBMK reactor core was never properly communicated to the operators at Chernobyl. Even the basic idea that the ‘AZ-5’ scram switch, touted as the emergency shutdown for the nuclear core, could only act as an emergency shutdown if the nuclear core was in a stable operational state, was withheld from the operators.

If the nuclear core was highly unstable, the graphite tipped Boron control rods installed in the RBMK nuclear core, would create even more nuclear fission, instead of reducing them. This extremely risky design choice and the lack of proper training to strengthen the operators’ understanding of how RBMK control rods worked, later proved to be catastrophic.

Also contributing to the Chernobyl disaster was the blatant disregard of the operators to adhere to the established safety protocols. Even though the procedure for the safety test, clearly directed the operators to keep the reactor thermal output above the 700MW threshold, the individual overseeing the test that day chose to ignore that highly important detail. Accompanied by a culture of retaliation and finger pointing that existed within the Chernobyl nuclear facility, the rest of the staff who were handling the safety test procedure that day, were forced to remain silent.

The disregard for safety critical procedural compliance was compounded by a series of human errors. These errors were due to the direct result of a culture of secrecy that prevailed particularly in the Soviet bloc countries’ nuclear industry.

A few years before the Chernobyl disaster, the Soviet intelligence apparatus produced a scathing report criticizing the management of the Chernobyl nuclear power plant. These legitimate concerns were categorized as alarmist and promptly swept under the carpet by the secretive, authoritarian Soviet government. It is therefore clear that the common denominator underscoring all these faults and failures that created the Chernobyl disaster, was the non-existent culture of safety in the Soviet nuclear industry.

To summarize, the lessons that could be learned from the Chernobyl disaster are as follows:

1. Safety is a feature in the system design itself and not an after thought
2. Always account for human errors and establish systems and procedures to monitor and rectify those errors
3. Complex ideas and concepts have to be communicated clearly and directly
4. Understanding the limits of the systems and documenting them well
5. Establishing a safety first culture with a clear mechanism for constant feedback and improvements

The reason why I spent nearly a thousand words to describe the events that caused the Chernobyl disaster and the lessons one could learn from them, is simply because, these lessons, are applicable to the design and maintenance of complex, industrial scale artificial intelligence systems. The parallels between designing safe and reliable artificial intelligence systems and the design of nuclear reactors are very uncanny. If 20th century was the age of the atom, the 21st century is turning out to be the age of data and artificial intelligence. The opportunities and challenges of the age of AI are similar to those of the nuclear age.

Autonomous decision systems that utilizes AI, such as self driving cars, trading algorithms, autonomous decision systems for health etc have to make similar compromises to a nuclear reactor’s design, such as: safety margins, development costs, operational costs, speed, efficiency, outlier operational modes etc.

From an AI development perspective, having a safety first AI systems design will foster greater levels of trust among the customers. An emphasis on customer education about the design limits of an AI system, instead of resorting to facile buzzwords and tangential analogies, can provide a greater level of long-term confidence in the AI systems. It also opens the door for the continuous development pipeline, a necessity for making current AI systems constantly adapt to the real-world.

As AI tools become more powerful and influence more aspects of our daily lives, newer regulations aimed at maximizing the effectiveness of these AI systems to our society and minimizing their ill-effects, will emerge. Having an open and transparent AI system design will help navigate these newer regulatory paradigms that could emerge in the near future. This approach will also help ward-off the gremlin of unnecessary regulations that could stymie the progress of the industry in-general.

The recent advancements in open-source tools around AI development such as the frameworks for developing deep neural networks have fostered a more open fundamental AI research. The academia supporting many of these groundbreaking works are constantly striving to be transparent and making their research efforts widely accessible.

But, the transparency and openness that exists in the AI research arena quickly vanishes when the AI systems enter production. This could be largely due to the perception of a competitive market landscape, where organizations have to find the latest tools to stay ahead of their perceived rivals. The common myth is that, such a closed-door approach to enterprise AI, could enable these organizations to keep their customers better updated with the latest features and services. In my mind, the current enterprise AI landscape is reminiscent of the cloak and dagger approach of the Soviet nuclear industry.

An alternate, more effective approach could be more transparency in how these AI systems are deployed. This would help customers have a better understanding of the risks and benefits of one tool over the other, thereby improving the level of trust in the businesses and organizations that chose to deploy AI. This approach could also help reduce the undue exposure of risks such as significant regulatory changes and unimaginable societal backlash from the perceived threats of a poorly understood automation technique.

The ongoing pandemic of COVID19 is a great illustration as to why data and AI transparency is highly critical. Despite the implementation of a robust epidemiological data collection and analytics strategy by the China Center for Disease Control, their system had an opaque reporting structure. This left the door open for manipulation of the disease outbreak data by the provincial bureaucrats.

Their collection of vast amounts of public surveillance data, including health information of individuals, came at significant economic and societal costs. The unspoken social contract that is often used to justify the existence of this massively expensive surveillance infrastructure in China is that, the individual freedom and privacy is sacrificed in exchange for a safe, sane and stable society.

Yet, these systems failed miserably at a critical juncture when they weren’t supposed to fail. The elaborate data collection and analytics infrastructure of China failed to contain the outbreak of a highly contagious disease. The implementation of flawed data driven decision making processes only helped provide a thin veil of false legitimacy to the efforts to obfuscate the real scale of the spread of the SARS-CoV2 in the Hubei province, by the Chinese bureaucrats.

These missteps acted as the proverbial trigger fuse for the ticking bomb of COVID19 pandemic. It also provided a false sense of confidence and hubris among the decision makers who encountered the SARS-CoV2 outbreak in its early stages. This short-sighted obfuscation of disease outbreak data, facilitated by a poorly designed data driven decision making process, significantly impeded the ability of not just the Chinese healthcare professionals, but the rest of the world, to effectively control the spread of the SARS-CoV2 virus early-on.

When the local journalists tried to fill the information void left by propaganda and misinformation, through their factual reporting of the seriousness of the SARS-CoV2 outbreak in Hubei province, the Chinese government immediately curtailed these journalistic efforts. Even the discussions by front-line healthcare workers about the seriousness of the disease outbreak in Hubei province was immediately silenced by the Chinese government. These actions are eerily similar to the Soviet Union’s attempts to quell any discussions about the Chernobyl disaster, both domestically and on the global stage.

In-short, the China’s COVID19 story is a cautionary tale of how to not use data and AI. By having a more transparent approach to leverage data and AI, it would empower organizations to innovate faster, solve problems efficiently and minimize the organizational risks of having a poorly understood automation and data analytics process. On the other hand an opaque data and AI infrastructure could end-up becoming an expensive, lie making machine, eventually causing the entire organization to unravel. The issues arising from the lack of data and AI transparency can often lead to far-reaching consequences that aren’t going to be confined within the walls of a single organization. It is therefore highly critical for organizations, to embrace openness and transparency in their data analytics and AI pipelines.

This transparent approach to data and AI has some unique challenges. It could amplify some existing organizational risks such as the emergence of newer enterprise security issues or unscrupulous actors attempting to manipulate these automated systems to their own advantage. It is very important to establish robust measures to map and mitigate these newer organizational risks associated with data and AI transparency. But, I am confident that having a more transparent approach to data and AI will help organizations become more resilient and earn them more public and customer trust and not the other way round.

TLDR: The access to safe, resilient, fault tolerant AI and data analytics systems for any organization is an important competitive advantage. How to implement them and how to bring more users to these new technology solutions should be a transparent conversation. While designing and implementing AI and data analytics systems, a lot can be learned from the glaring mistakes made by other promising technology solutions, such as the nuclear industry and from the flawed AI and data analytics implementations by the Chinese government to handle their disease outbreak data.