XZ Utils Hack

This week we talk about Linux, backdoors, and the Open Source community.

We also discuss CPU usage, state-backed hackers, and SSH.

Recommended Book: The Underworld by Susan Casey

Transcript

In the world of computers, a "backdoor" is a means of accessing a device or piece of software via an alternative entry point that allows one to bypass typical security measures and often, though not always, to do so in a subtle, undetected and maybe even undetectable manner.

While backdoors can be built into hardware and software systems by the companies that make those devices and apps and bits of internet architecture, and while some governments and agencies, including the Chinese government, and allegedly folks at the NSA, have at times installed backdoors in relevant hardware and software for surveillance purposes, backdoors are generally the domain of tech-oriented criminals of various stripes, most of whom make use of vulnerabilities that are baked into their targets in order to gain access, and then while inside the administration components of a system, they write some code or find some kind of management lever meant to give the company or other entity behind the target access for non-criminal, repair and security purposes, and that then allows them to continue to gain access in the future; like using a rock to prop open a door.

Concerns over a backdoor being installed in vital systems is fundamental to why the US and European governments have been so hesitant to allow Chinese-made 5G hardware into their wireless communication systems: there's a chance that, with the aid, or perhaps just at the prodding of the Chinese government, such hardware, or the software it utilizes, could contain a Trojan or other packet of code, hidden from view and hardcoded into the devices in some covert manner; these devices could also harbor even smaller devices, indistinguishable from hardware that's meat to be there, that would allow them to do the same via more tangible means.

Though there were almost certainly other economic and technology-dominance reasons for the clampdown on products made by Chinese tech company Huawei beginning in earnest in 2012, and escalating rapidly during the US Trump administration, that process was at least ostensibly tied to worries that a Chinese company, prone to spying and stealing foreign tech, already, might incorporate itself into fundamental global communication infrastructure.

It was underpricing everybody else, offering whizbang new high-end 5G technology at a discount, and supposedly, if the accusations are true, at least, doing so as part of a bigger plan to tap into all sorts of vital aspects of these systems, giving them unparalleled access to all communications, basically, but also giving them the ability, supposedly, to shut down those systems with the press of a button in the event that China wants or needs to do so at some point, if they ever decide to invade Taiwan, for instance, and want to distract the Western world until that invasion is complete, or just make rallying a defense a lot more difficult.

Other, confirmed and successfully deployed backdoors have been found in all sorts of products, ranging from counterfeit Cisco network products, like routers and modems, some of which were installed in military and government facilities back in 2008 before they were recognized for what they were, to Microsoft software, Wordpress plugins, and a brand of terminals that manage the data sent along fiber-optic cables, mostly for high-speed internet purposes.

Again, in some cases, the entities making these products sometimes do install what are literally or essentially backdoors in their hardware and software because it allows them to, for instance, help their customers retrieve lost passwords, fix issues, install security updates, and so on.

But backdoors of any shape or size are considered to be major security vulnerabilities, as stealing a password or getting access to a vital terminal could then grant someone with bad intentions access to absolutely everything, giving them god-like control over all aspects of a customer's information and operations, or maybe all of the company's customer's information and operations, and that creates a single point of failure that most companies want to avoid, because at a certain point there's no real way to prevent a truly determined and well-funded foe if they know the payout for investing in accessing that terminal or getting that password would be that substantial.

What I'd like to talk about today is a long-term effort to do exactly that, the target, in this case, being small, but the potential payoff of backdooring it being pretty much as big as you can imagine.

—

XZ Utils is the name of an Open Source data compression utility, which means that it squishes data in such a way that no information is lost, but so that big files and other packets of information become smaller, and that makes it faster and easier and cheaper to send that data from place to place.

XZ is popular in part because it's effective, in many cases outperforming other free alternatives, like gzip and bzip2, but it also supports an older compression model called LZMA, and it exists in the public domain, which means it's incredibly inexpensive to use, free, for most purposes.

It's especially popular in Linux and other Unix-like systems, and in practice that means it's used across these systems so that when data is moved from place to place, it's compressed and decompressed, putting less pressure on the systems themselves, almost like reducing the weight of everything you have to carry throughout the day, without any reduction in quality or the nature of those books and bags and laptops and other things you're hauling around all the time; even small reductions in that weight could make a big difference in the strain on your body, over time, and this utility accomplishes the same for the systems that incorporate it.

So this software utility is super useful, is free to anyone who wants to use it, and it's better than a lot of other options, and it's thus been baked into a bunch of fundamental computer infrastructure, like most Unix-like systems. And that's important for a lot of reasons, but the most immediately concerning issue is that the vast majority of servers that run the tech world—basically all the major tech companies, and all the companies they work with—manage their services with Linux.

XZ isn't just important for folks who have laptops running on Linux, then, it's also vital to the functionality of huge chunks of the internet; stats from the past few years show that about 96.3% of the top million web sites run on Linux servers, and a substantial amount of non-web-serving servers do, as well.

All of which sets the stage for the hubbub that arose on March 29, 2024, when a Microsoft employee named Andres Freund announced that, after looking into a decrease in performance in a version of Linux called Debian—a distinction between how fast it should have been going and how fast it was going of about 500 milliseconds, and that minor slowdown bugged him enough to look into what newer, experimental versions of XZ Utils were doing to the Debian operating system he was working with—after looking into that issue, he announced that he had discovered a backdoor in XZ that was causing errors in a memory debugging tool built into the software, and using more CPU power than Debian otherwise would have used.

So he announced this discovery, reported it to an open source security mailing list, to make it known amongst the right people, and that alerted the folks who were experimentally incorporating this new build of XZ into their software.

As it turns out, this backdoor, had it been implemented in all this software and spread across the servers that manage the web, would have granted whomever had access to it the ability to alter the behavior of the local instance of the Secure Shell Protocol, or SSH, which is what protects servers while they operate on open networks like the internet.

The degree to which this would have damaged the web, as it exists today, cannot be overstated. This problem was given a Common Vulnerability Scoring System ranking, which rates the alarmingness of software issues based on how much damage they could potentially cause, which helps computer security professionals figure out which problems to address first, a score of 10, which is the highest possible score.

In theory, this would have granted the person or other entity with backdoor access the ability to get into essentially any server touching the internet with full administrator privileges, making all that information transparent to them, providing them all information about users, passwords, banking information, everything everyone has ever posted to social media, private communications, research and technology secrets—it's really just boggling thinking about how much damage could have been caused by the right person or people, as such a backdoor would basically do away with most of the security measures they might encounter while attempting to infiltrate and even take over pretty much anyone.

Because it was discovered by Freund, though, and because he got word out to the right people as quickly as he did, the cybersecurity world was able to pivot pretty quickly, advising everyone who had implemented these test versions to roll back to earlier versions of the relevant software, and the folks behind XZ quickly released updated versions of the utility that removed the backdoor problem.

This also triggered a response in the wider software world as many developers have started to reduce the damage future, similar backdoors would be able to cause by reducing the connections and dependencies it took advantage of to function.

So this was a big enough deal that even something as arcane as compression utilities and SSH became front-page news around the world, but arguably one of the most interesting aspects of this story is what we know about the person or people who seem to have installed this backdoor.

Someone, or group of someones, going by the name Jia Tan, alongside an array of sock puppet accounts—fake accounts with different names that they also managed—started to contribute to the maintenance and development of this project, which is common in the open source world; that's part of what makes open source software and systems so powerful and desirable, despite often not having much in the way of funding or official support from big-name companies; they're often passion projects maintained by maybe just one or a few or a handful of dedicated developers.

In 2021, this entity that became known as Jia Tan started contributing to open source projects, and then contributed a patch to XZ via its mailing list.

Around that same time, several people who hadn't been seen in this project's community, previously, started to complain that it wasn't being updated fast enough, and arguing that another maintainer should be brought on board, to help it move along faster.

This Jia Tan character then started making a lot more contributions to the project, all of them seemingly innocuous and helpful, though in retrospect at least one of them changed a function that would have detected the more malicious changes they ultimately submitted, later.

In February of 2024, Tan submitted changes for the new version of XZ Utils that incorporated a backdoor, and groups of people in this larger open source community, possibly sock puppet accounts, started telling the developers who run Debian, Ubuntu, and Red Hat, all popular versions of Linux, they should incorporate this new version with those backdoor-incorporating changes into their operating systems.

There are strong suspicions, but little evidence, at this point at least, that Jia Tan and those other sock puppet accounts were run by a well-funded and skilled, probably government-backed hacking group, like one of the entities that often work as proxies for Russia's SVR—their intelligence agency that tends to support local hacking groups to do this sort of dirty work; though again, we can't say that with any certainty, as a lot of government-backed hacking groups could pull off something like this, with enough patience, years worth of patience, and it's still possible that this was a single hacker seeing a soft-target and the potential for a huge payoff if it all worked out.

That said, because of the approach this threat actor, whomever they actually are, took to target this utility, and because of how close they got to doing what they intended to do, which would have been devastating, probably even world-changing in some ways, the relationship that big tech and governance has with the open source world is being reassessed, because often the folks running these projects are just individual people doing all this important work in their free time. But because of how the tech world has evolved, huge swathes of the internet and other vital infrastructure are reliant on these single-person, passion-projects that are potential targets for cooption or, as seems to have been the case here, using what's called social engineering to manipulate the folks behind these projects, which can then gives more access to all the stuff they manage, and thus, the things that rely on the stuff they manage, to entities that want to cause harm.

Again, and this cannot be emphasized enough, we just barely dodged a bullet here, and the only thing that prevented a huge amount of potential destruction was the effort of another single person who was, almost on a whim, hacking away on a little problem they wanted to look into, and who thus stumbled upon this issue right before it reached a scale that would have been truly problematic.

And all of these issues were arguably the result of someone who found themself in the position of maintaining, more or less solo, a utility that became vital to global cybersecurity, and which thus made them the target of a sophisticated social engineering campaign.

Show Notes

https://en.wikipedia.org/wiki/Backdoor_(computing)

https://en.wikipedia.org/wiki/Hardware_backdoor

https://en.wikipedia.org/wiki/Social_engineering_(security)

https://www.zdnet.com/article/linux-has-over-3-of-the-desktop-market-its-more-complicated-than-that/

https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/

https://research.swtch.com/xz-timeline

https://research.swtch.com/xz-script

https://news.ycombinator.com/item?id=39895344

https://www.runtime.news/sabotage-in-the-software-supply-chain/

https://news.ycombinator.com/item?id=39903685

https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt

https://www.wired.com/story/jia-tan-xz-backdoor/

https://www.404media.co/xz-backdoor-bullying-in-open-source-software-is-a-massive-security-vulnerability/

https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html

https://www.economist.com/science-and-technology/2024/04/02/a-stealth-attack-came-close-to-compromising-the-worlds-computers

This is a public episode. If you’d like to discuss this with other subscribers or get access to bonus episodes, visit letsknowthings.substack.com/subscribe