Chief Information Officers (CIOs) can implement measures now to mitigate risks in the current IT environment.
In the aftermath of a widespread system crash affecting approximately 8.5 million Windows machines worldwide due to a software update by cybersecurity firm CrowdStrike, technology experts have provided valuable insights to CIO Journal on preparing for future major IT outages.
Understanding Vendor Software Development Practices
Neil MacDonald, Vice President at Gartner, emphasized the critical need for IT leaders to hold vendors, especially those deeply integrated into IT systems like CrowdStrike, to exceptionally high standards of software development, quality assurance, and release practices. He stressed the importance of rigorous regression testing across all Windows versions before updates are deployed. MacDonald suggested that companies should inquire into vendors’ software development processes, testing methodologies, and options for managing update rollout speed.
Amy Farrow, Chief Information Officer of Infoblox, underscored the necessity for prioritizing deployment and testing procedures to ensure availability, reliability, and security within IT systems.
Re-evaluating Acceptance of Software Updates
Paul Davis, Chief Information Security Officer at JFrog, highlighted the need for organizations to pause and reconsider their approach to accepting software updates, despite it being a common security practice. He advocated for comprehensive testing of packages, upgrades, and new features to mitigate potential risks. Davis acknowledged the impracticality of testing every update from numerous vendors but stressed the prioritization of testing based on severity and scale, with potential support from automation and AI tools.
Jack Hidary, CEO of SandboxAQ, emphasized the role of AI in detecting intricate errors in extensive lines of code, proposing its use to analyze the interdependence of new software updates with existing software stacks.
Developing a Comprehensive Disaster Recovery Plan
Gartner’s MacDonald likened incidents rendering Windows systems unusable to natural disasters that knock systems offline. He recommended that businesses consider implementing disaster recovery plans akin to those used for natural disasters to ensure operational resilience. Chirag Mehta, a cybersecurity analyst at Constellation Research, suggested establishing isolated environments or “clean rooms” for restoring critical systems. Additionally, Mehta advised conducting tabletop exercises to simulate various risk scenarios, including IT outages and cyber threats, to enhance preparedness.
Victor Zyamzin, Chief Business Officer of Qrator Labs, noted that companies regularly backing up data were less affected by the CrowdStrike outage, emphasizing the importance of consistent and tested backup procedures.
Reviewing Vendor and Insurance Contracts
MacDonald advised companies to review contracts with vendors to ensure clauses guaranteeing reliable and stable software delivery, particularly for vendors impacting critical operations. He suggested negotiating for contractual coverage in case of update-induced outages and exploring compensation options during contract renewals. Peter Halprin, a partner at Haynes Boone specializing in cyber insurance, stressed the role of insurance in safeguarding companies against financial losses associated with IT outages, whether caused by the insured company or a service provider.
Considering Platform Choices
The CrowdStrike incident prompted discussions on the viability of relying on Windows-based systems. Chirag Mehta highlighted the differences in access levels between Windows, Apple’s macOS, and Linux systems, noting that CrowdStrike’s access to Windows kernel functions does not extend to other operating systems. He mentioned the increasing adoption of Chromebooks, which operate on Google’s Chrome OS and may not require the same level of access as traditional Windows devices. Mehta posed critical questions for enterprises to evaluate their reliance on Windows and consider alternative platforms based on security needs and operational requirements.
These insights and recommendations from industry experts provide a comprehensive guide for IT leaders to enhance preparedness and resilience against potential disruptions, ensuring the continuity and security of organizational operations in an increasingly interconnected digital landscape.
