By Suresh Ramasamy, Head of Group IT Security, Hong Leong Bank Berhad
Before I start, I’d like to congratulate Sungai Buloh Hospital on it’s excellent handling of the Coronavirus situation in Malaysia.
The Malay Mail reported that Sungai Buloh hospital (SBH) was recently hit with IT failures. Sg Buloh hospital is quite well known to the denizens of Klang Valley, being a governmental hospital of choice to many. I personally find the service is very good, doctors are friendly, professional and I don’t spend much time waiting as the processes are quite efficient.
The news report highlights difficulties in retrieving medical and investigation reports of patients. This problem was pinned to the hospital using Windows XP as the Operating System. Reports also mention that the hospital main servers were down for sometime. It was also quoted that the issue also stemmed due to limited storage space due to Windows XP OS limitation.
Some facts for consideration in this article, which to me sounded odd, but plausible. Lack of clarity on the matter, unfortunately fuels to the speculation (of which some are discussed in this article)
What about Windows XP?
Firstly the use of Windows XP. Microsoft positioned Windows XP as an end-user operating system, targeted to be installed on desktops and laptops. It was never meant to be used in server environments, though I have noticed small organisation turns a desktop into a file share for the organisation. The OS was never meant to be a server platform (though it can be at a minuscule deployment), and surely not for a hospital the size of SBH.
Microsoft declared Windows XP obsolete starting April 8, 2014. That means from the EOL (end-of-life) date, there will be no support. Meaning, if there is a bug or vulnerability in the OS, it will not be fixed.
This also implies that the software ecosystem, such as anti-virus, endpoint protection and other critical software will also not made available as the efforts to maintain the software will be focused towards supported operating system. So, not only you have an OS that doesn’t have any updates, you also lose the updates for other software that runs on that ecosystem.
Essentially a ticking time bomb.
Without any further information, one of the points of issues faced by SBH was the server failure. This can potentially be attributed to the use of Windows XP as the server OS (god forbid, but based on experience can/may happen), or a general failure at the server, be it at the OS, Application or Hardware level (may even be network, the server might be running on a 10Mbps network card, supporting the whole hospital). Again, without any further details, one can only speculate on this matter.
Another interesting point to note that the report also pointed out that Windows XP has a storage limitation. A quick check shows that the first limitation is at the memory level, and its due to the 32bit architecture.
There are also issues at the file system level, as XP primarily supports FAT (File Allocation Table) format, which has a hard limit of 32GB. Worth to note that Microsoft also released a version of Windows XP for 64 bit. Windows XP supports FAT32, however the format tool natively supplied with XP doesn’t allow creation of FAT32 partitions (Primer: FAT/FAT32 creates an index of where files are located, based on free space available, which gives the OS a location of the files in the disk).
Software Obsolescence
Software obsolescence isn’t new, but its worth revisiting to understand how it can contribute to this situation.
When a software companies declare that a product is at End-Of-Life, its a statement to inform customers that they will no longer support that product, and that the customer should opt for newer software. While at face value it looks pretty simple, the impact is far reaching.
Hardware Compatibility
Firstly, upgrading an OS requires the hardware the be compatible. This means, if my father had a PC that he uses at home, he needs to first check if the PC can be upgraded. At times, the change in the OS can be drastic. Meaning, that the PC may not be compatible due to several reasons. For example, moving to a new PC hardware because the processor and it’s architecture is also obsolete.
In this case, a new OS may only support 64bit architecture and may not have backwards compatibility to a 32 bit architecture. Windows XP supported both 32 bit and 64 bit architecture, paving way for the move from 32bit hardware to 64 bit hardware, which offers better scalability and flexibility. While this implies the link between OS and hardware, some subtle changes at software may have similar effect. Apple introduced MacOS Catalina which effectively prevented running on 32bit applications on it, making it a pure 64bit based Operating System. Many users reported the issue due to application availability, and some users ended up reverting back to MacOS Mojave.
Secondly, while the hardware becomes a factor to look in, another point to consider is the hardware compatibility towards the OS, from the point of drivers. Drivers are software that allows the OS to “talk” to the hardware.
Without the drivers, the hardware remains useless. I remember the time when we had a spectrum analyser in one of my previous roles, and the spectrum analyser on worked on Windows 3.11 for Workgroups due to limited driver support. We couldn’t move it to the latest OS as the manufacturer had stopped support, and their recommendation was to spend equal amount of money to get a new set of equipments, that had the latest software and drivers.
The organisation, of course, chose the path of least expenditure, had to isolate the PC so that it was only a single use, making it a standalone independent analyser. I still remember pulling the data off the analyser using floppy disk.
The same can be said to medical equipments. Imagine an MRI that was build using Windows XP as it’s operating system, cannot be migrated as the hardware support is no more there. No “sane” hospital would buy a new MRI machine just because the OS is outdated. As some IT experts will tell you – “If It ain’t broken, don’t fix it…”. The whole security industry was built around managing security risks, mind you.
Thirdly, organisations do not actively manage obsolescence. This is because obsolescence is an expensive affair. A cost-conscious organisation would do its best to “sweat its assets” and make their investments live past zero net book value.
It creates additional cost to business when it comes to obsolescence management, and we potentially see it happen previously at the MAHB issue. There is cost in replacing equipments, there is cost in migrating the data as well as time and resources required to carry out the project, not to mention training to all those involved so that they know how to use the new system (which may come with new UI/UX and workflow).
It’s a very daunting affair, and one that has far reaching effect throughout the organisation.
Software Dependencies
We depend on a number of software for our systems to work. Having a computer and the OS alone doesn’t do much, business runs based on its Line-Of-Business applications. Example, ERP (Enterprise Resource Planning), CRM (Customer Relationship Management) and many more. When the OS becomes deprecated, software developers also stop developing functionality for a now defunct OS, moving their codebase to a new platform. This means the code, depending on the extensivity of the change, may render the old code useless.
Some platforms provide some degree of cross version support, but that’s usually limited, more of in favor of newer platforms. Remember that the same support required by the user is also required by the software developers to build their code on. There are some cross platform frameworks available, but even these frameworks may stop supporting older, deprecated OS.
The OS also comes with SDK (Software Development Kit), crucial for software development teams to harness the power of the OS. Just like how the OS gets deprecated, so will the platform SDK’s.
Moving Forward
I discussed technology debt in greater details before, and it seems to be a recurring theme in large organisations in Malaysia.
Obsolescence is a huge debt, which most organisations overlook, eventually coming back to haunt them.
It’s not a discussion that any CEO/CFO would ever like to have, especially when the cost balloons and creates a huge dent in the balance sheets. The same can be seen in Governmental departments, where focus on maximising the taxpayers money can be seen as a prudent outcome of well-oiled administration.
Non-technology organisation fail to grapple the complexities of technology. It took us long enough to start trusting automation and computing, and now this becomes another headache to manage. Some organisations even opt not to capitalise on the computing/internet era, effectively creating a barrier to efficiency and economies of scale (when it comes to data/records management).
Having focus and attention to IT helps to alleviate such issues.
Most organisations establish an IT Steering Committee, comprising of senior leadership team to address such risks. Strategic discussion on project prioritisation, maximising annual budgets and looking at technology risks becomes a staple periodic discussion to look at IT and it’s associated risks/debts.
If at all, the business is faced up to the wall with no option but to manage its business with the existing infrastructure, there are some mitigation’s that can be applied. Ideally these systems should be run in isolation (similar to the spectrum analyzer case I spoke earlier).
If it has to be networked due to the nature of the application, thorough backup and restoration procedures need to be established, run and periodically tested. I’ve seen organisations that pride on doing backups, but had never tested their backups, nor even tried to restore the backup (there’s a reason why those systems are called backup software and not restore software).
Having a recovery plan helps, but the plan is as good as being tested periodically. Worst case scenario needs to be tested, break glass type procedures, and how business can run, in the event that complete failure is imminent.
All else fails, start saving for a new system, maybe look at cloud as an alternative?
This article was originally posted at https://www.drsuresh.net/2020/01/sg-buloh-hospital-jan2020-case-study/
References
- Malay Mail (22 Jan 2020)- https://www.malaymail.com/news/malaysia/2020/01/22/report-sungai-buloh-hospital-hit-by-it-breakdown/1830364
- Microsoft XP EOL statement – https://www.microsoft.com/en-us/microsoft-365/windows/end-of-windows-xp-support
- MAHB Case Study – https://www.drsuresh.net/2019/09/mahb-case-study-aug2019/
- Technology Debt – https://www.drsuresh.net/2019/08/cyber-tech-debt/