Opportunistic (low-effort) microbenchmarks with OpenSSL

Whenever I get my hands on a new device or machine, one of the first things I do is run openssl speed -evp <cipher> and log the throughput for 16k blocks, specifically from AES-256-GCM and ChaCha20-Poly1305 ciphers. These ciphers are widely used for VPNs and TLS, and both ciphers benefit from dedicated instructions and accelerated units present on modern CPUs: AES have dedicated instructions for performing rounds (AES-NI), while ChaCha20 benefits from SIMD with vector operations (AVX2, Neon). High throughput from modern CPUs regardless of clock speed immediately reflect this.

All these efforts started from me trying to find a cheap VM where I could run my self-hosted stuff. There are a lot of cloud providers with Linux VMs using different CPUs at multiple price points. I wanted to do quick tests and compare them to something I could run on my own machines, putting into consideration how much value the cloud VMs give me compared to just hosting them at home. These quick tests turned into a habit: any device I ha, or I can borrow that has OpenSSL would have me running openssl speed on the terminal.

Things get a bit trickier for non-Linux devices, but the tests can still be done with low effort. For Windows, OpenSSL comes from either an installation of Git or through WSL. In Macs, the OpenSSL which have these ciphers come from Homebrew, and from my experience people who usually install Homebrew used it for installing something that would depend on OpenSSL as well. The weird one is with Android, which can be done by installing Termux APK. Rarely do people have this on their phones, so you have to make sure they’re okay with installing an APK.

I have to emphasize that the results here are imprecise and can only be taken at face value. Unaccounted variables include thermal and power constraints, server tenancy (for cloud VMs), background processes, OpenSSL version and build flags, etc. Putting effort into accounting these would make the testing scheme less opportunistic. My rule of thumb is that I have to be able to do all these on a device I borrowed for under 3 minutes without installing anything new.

So here’s my record of me running OpenSSL speed on as many devices that I can have my hands on:

Note: I will add more entries and observations as I get more devices which I can run the tests with. I might have to restructure the columns as well. The active document can be found here.

Some observations:

• The CPU generation matters a lot: the increase in throughput for both ciphers shows up as SIMD lanes get wider and instruction sets get more advanced. For cloud providers, you have to make sure that the price point they offer the VMs with are in line with what generation of CPU is used.
• Expect the throughput of server CPUs to be slower than their desktop and laptop counterparts, despite belonging to the same generation. Servers don’t have aggressive clock boosting that desktops or even phones have.
• The Apple M1 had the highest discrepancy between measured throughputs. While undoubtably M1 is the most advanced processor we had in 2020 as shown by having the best AES performance, it had moderate SIMD performance gains shown by ChaCha.
• On the other hand, very old CPUs tend to have better Chacha performance than AES. This is the whole point of having Chacha20-Poly1305 as the cipher for Wireguard: old devices and phones can still get decent VPN throughput despite the lack of dedicated AES units.

With all the information above, what did I end up using to host my stuff online? So far I have GCP’s e2-micro at asia-east-2 winning in terms of value. Before comparing performance results, I had to consider the latency and price: I think 15 USD is too much to pay for anything monthly, and the ping has to be sub-30 ms from home (latency-wise, it’s HK>TW>SG regardless of ISP here). While e2-micro at asia-east-1 also fits the bill, you are more likely to get E2 instances that are Skylake or later in HK. Who knows, this might change as CPU offerings get more modern and pricing become more competitive.