The New Dependency Dilemma

Granted, the dilemma isn’t really new. But AI lift it to new levels.

AI helps attackers find and exploit security holes much faster than ever before. To stay on the safe side, you’d have to update affected dependencies as quick as possible; otherwise, your app could be vulnerable to attacks. Yet, you shouldn’t update any dependency in a haste, because nowadays, the rate of supply chain attacks increases, too. The dependency you need to update could contain malicious code that installs a backdoor or extracts data from your app. The reasonable response to this would be to let some time pass by before updating. The community or some security researchers might discover a supply chain attack during this time.

However, this cooldown phase contradicts the need to close security holes as soon as possible. It’s a Catch 22: Decide for a quick update and you might install malicous code; decide for a cooldown phase and your app might get hijacked from the net. Either way seems wrong.

So what to do?

The perils of quick dependency updates

When one of your dependencies announces a new vulnerability, assume that a zero-day exploit is already created and actively scanning servers for vulnerabilities. It’s only a matter of time until they find and attack your app. The sooner the issue is fixed in your app, the smaller the attack window.

But what if the fix is contaminated with malware? That’s not a purely theoretical thought. Attacks like this happened already, and they often work like this: An attacker establishes themselves as a contributor to a popular open source project. When someone detects a vulnerability in the project, quick action is key, and the attacker is quick at providing a PR that fixes the vulnerability but also sneaks in malicious code. The urgency of the fix lets the rest of the team run the tests that prove the vulerability to be fixed but keeps them from investigating the PR deeper, possibly overlooking the malicious code.

Releasing the fix along with a prominent security advice causes dozens, hundreds, or thousands of users to upgrade their projects to include the fix. In no time, the malicious code has infected a large number of servers.

So let’s wait a bit? For a day or a week, maybe?

This seems a valid mitigation strategy: Just wait a bit and see if someone discovers anything unusual. If no warning signals appear, go ahead and update the dependency. On a closer look, this strategy has two flaws:

  1. It depends on others to eventually find something. How can you tell if anyone is actively scanning the code for a supply-chain attack? Does the lack of security alerts mean that the code is safe? Or did the malicous code simply evade detection?
  2. It requires to decide how long to wait. The shorter the time span, the less likely is it that malicious code gets detected. The longer the time span, the longer your app is vulnerable to outside attacks.

So here we are: Quick is wrong, and slow is wrong.

Is there something we can do that’s a bit less wrong?

Stay calm and weigh the risk carefully

Remember this is about critical fixes only. For any non-critical update to a dependency, a cooldown phase is the way to go. Go’s dependency management system follows the minimum version selection (MVS) approach; so no dependency updates happen without you taking action (by bumping a dependency version in go.mod and calling go mod tidy, or by calling go get). You have the time to scan a new release for anything suspicious (or chase security scanners or LLMs across the diffs).

How the Go ecosystem can help

Go has two quite effective measures to secure a project’s supply chain.

The first measure consists of the Go module proxy and the checksum database.

In a nutshell:

  • Whenever a new module release (technically, a tagged commit) is pulled for the first time, the Go proxy calculates a cryptographic hash of the module and stores it in the Sum database
  • Subsequent pulls are pinned to this exact hash, provided your toolchain uses a Go proxy for go getting dependencies (which is the default behavior of a newly installed Go toolchain but can be configured)

Manipulating the repository therefore has no effect on proxied downloads; even unpublishing the repo cannot break client code.

This mechanism ensures that tampering with an existing release is detected; however, neither the proxy nor the sum DB can protect the very first download of a release, when the proxy still has to calculate the hash. A malicious author or an attacker could provide a new release with malicious code in it, which is why it’s so important to not download new releases right away.

While the Go proxy and the sum DB cannot prevent malicious releases, they protect you from post-publish tampering of whatever library you pull in. With Go’s minimal version selection approach, you can stay on a known good version until the new release has been verified through close analysis.

The mechanisms behind Go’s module managment that protect against supply chain attacks are much more detailed than I can describe in the context of this spotlight; for an overview, check out this Go Blog article.

How to treat a critical security fix

For critical releases, a quick reaction narrows the window where the app is vulnerable to attacks. Rather than setting a cooldown phase and hoping that others may scan the release and probably detect malicious code if there is any, it’s safer to apply the usual mitigations (as one would do, BTW, in any ecosystem): diff reviews, provenance checks, and employing security scanners.

You could even automate the latter part by setting up your own custom Go proxy that runs security scans and vulnerability checks for every initial download that has no cryptographic hash yet. Have your favorite LLM screen even the innocent-looking parts of new code. Malware might mimic test code processing some compressed test data that actually contains the main part of the exploit (like in the famous xz incidenet).

Supply chain attacks become more sophisticated than ever while security vulnerabilities are exploited faster than ever. I’m not saying protection is easy, but every bit counts.

Remember seat belts that many considered pointless or even dangerous when introduced. Today, seat belts have saved countless of lives.

Fasten your security seatbelts.