crash leaks case study

Leak case study 5: Leaks when the VPN process crashes

Last updated:

While a good VPN application should rarely crash, if ever, it should be prepared for such a scenario and protect the user from leaks in the event of a crash.

This case study from our Digital Security Lab describes leaks that can result from crashes, and how to test for them using the ExpressVPN Leak Testing Tools.

Overview of crashes

We have currently only focused on crashes within the VPN processes themselves, for example crashes within the OpenVPN process, as opposed to the UI applications or support binaries that normally come with VPNs.

Providing a generic test suite to test crashes of UI applications isn’t really practical, as those aspects are very idiosyncratic to each provider. We hope to add some generic tests at a later date which will allow third parties to stress test their own applications.

Internally, we have tested crashing each component of our own VPN apps and are confident that no leaks would occur if any one component fails.

As most well-designed VPN applications crash infrequently, we consider this test case among a class of “stress tests,” but certainly still a category of leak tests which VPNs should pass.

Leaks during crashes

There are two broad categories of leaks, of differing severity, that can occur in the event of a crash:

Temporary leaks

These are leaks which occur during a relatively short, limited window of time. Generally they take the following form:

  • VPN application detects the crash
  • VPN application tries to correct the issue
  • Integrity of VPN connection is restored

These are the least severe leaks. They are only temporary and given the (hopefully) low probability of a VPN crash, should not expose the user’s privacy too much.

Persistent visible leaks

These occur when the VPN application detects the crash but fails to correct it. Sometimes this is because the application can’t correct the issue; other times it’s because the application hasn’t been designed to fix the issue.

The usual behaviour is that the VPN application disconnects and leaves the user in an unprotected state.

The severity of these leaks can vary and primarily depends on whether the user is present when the crash occurs. If the user is using the device and witnesses the issue, then they can potentially correct the problem promptly. However, if the user is away from the device, then the issue could go uncorrected for a long period of time.

Note that some VPN applications offer “kill switch” or “network lock” functionality which is designed to shut down the user’s internet connection when issues like this occur. Even with such protection, often these VPNs leak for a small period of time before the “kill switch” takes effect. These then fall into the previous category of ”temporary leaks”.

Persistent invisible leaks

These are leaks caused by the VPN application not detecting the crash. The effect is to leave the user’s privacy exposed for potentially long periods of times, e.g. hours or even days.

Generally the user will detect these issues, as it’s rare for a VPN process to crash and leave the network in a fully usable state. More commonly, a small amount of traffic leaks, such as DNS requests, while the rest of traffic remains blocked.

Manual testing

Manually testing for crashes is quite difficult. Simulating a crash of the VPN process is not too complex, however testing for leaks is tricker as it generally requires packet capture. We thus consider these steps appropriate only for power users to attempt.

We recommend using a tool like tcpdump to capture all traffic leaving a device’s network interfaces. The simplest way of doing this would be to run the device in a virtual machine (VM) and capture traffic leaving the VM on host device.

We first give some example steps on how to simulate a crash then give full repro steps for the test.

Simulating a crash on a Posix device

  • pgrep openvpn | xargs -n1 kill -9

Simulating a crash on a Windows device

  • taskkill /F /IM openvpn.exe

Repro steps

  • Connect your device to the VPN
  • Initiate packet capture
  • Crash the VPN process
  • Wait for either
    • the VPN application to notice the crash and stabilize;
    • a period of 2 minutes.
  • Stop packet capture
  • Disconnect the VPN
  • Check the traffic capture for packets which went unencrypted out of the device

Note: Checking traffic capture can be tricky as there can be noise generated from local IP traffic, ARP and other acceptable traffic.

Testing using the ExpressVPN Leak Testing Tools

The ExpressVPN Leak Testing Tools are an extensible suite of open-source Python tools designed for both manual and automated leak testing of VPN applications. Please see our introduction to the tools for instructions on downloading and setting up the tools.

Once you’ve set up the tools, ensure you are in the tools root directory and execute:

./run_tests.sh -c configs/case_studies/vpn_process_crashes.py

These tests perform exactly the steps described above. They currently don’t use packet capture to determine if there was a leak. Instead, they look specifically for DNS leaks and IP leaks using various tools. For automated tests, we generally avoid catch all packet capture as the results can be too noisy and lead to false positives.