Analysis of poor Asterisk SRTP Performance on Debian Linux

Intro

Although I have experience running Asterisk at significant scale, the number of SRTP channels has always been relatively few. However, SRTP is an increasingly growing requirement for Enterprises and Voice Carriers alike… meaning SRTP usage, relative to unencrypted RTP, is growing.

But what overhead does it create, on a modern CPU with AES-NI and SHA-NI? For a cypher such as AES_CM_128_HMAC_SHA1_80 experience from 15-odd years in “the industry” tells us that we’d usually assume around +15% CPU overhead. So if we were previously happily processing 2000 concurrent unencrypted channels we should be able to do about 1740 encrypted calls. Alas… when you assume, you make an ass out of you and me.

Load Testing

When performing a 1000 concurrent call load test the results were quite alarming. Test setup was as follows:

• Asterisk v20
• Debian 12 (Bookworm)
• AWS c6i.4xlarge instance
• No media codec transcoding
• No unnecessary extra processes (log processors, security tooling, etc.)
• pjsua as the load generator, running across a fleet of 3 separate AWS instances

Tests were run with plain unencrypted RTP and a single SRTP channel per call.

CPU baseline for 2000 unencrypted channels was 26%. Load was spread evenly across all cores

When repeating the test with 1000 unencrypted channels and 1000 encrypted channels, the test failed miserably. Asterisk locked up and stopped responding to SIP requests, with re-transmissions exacerbating the issue. I managed to ramp it up to 500 concurrent calls (500 unencrypted and 500 encrypted channels). CPU usage was 28%. That represents a 112% overhead for encryption.

WTF??

As one might imagine, 112% CPU overhead with a hard cap well below CPU exhaustion was not expected. I’ve done similar tests in years gone by… so what changed? Well… Debian changed. In Debian 10 and below, libsrtp was built in a way which meant that it used its own internal crypto implementation for SRTP encryption and decryption. Starting in Debian 11, and persisting to this day (up to at least Debian 13, at time of writing), libsrtp is built to use libnss for encryption/decryption.

Debian 10 (no external links):

$ ldd /usr/lib/x86_64-linux-gnu/libsrtp2.so.1
        linux-vdso.so.1 (0x00007ffd6d766000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f19d3416000)
        /lib64/ld-linux-x86-64.so.2

Debian 12 (linked against libnss):

$ ldd /usr/lib/x86_64-linux-gnu/libsrtp2.so.1
        linux-vdso.so.1 (0x00007ffc4577e000)
        libnss3.so => /lib/x86_64-linux-gnu/libnss3.so (0x00007ff91085c000)
        libnspr4.so => /lib/x86_64-linux-gnu/libnspr4.so (0x00007ff91081a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff910638000)
        libnssutil3.so => /lib/x86_64-linux-gnu/libnssutil3.so (0x00007ff910606000)
        libplc4.so => /lib/x86_64-linux-gnu/libplc4.so (0x00007ff9105ff000)
        libplds4.so => /lib/x86_64-linux-gnu/libplds4.so (0x00007ff9105f8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff9109d5000)

NSS uses the CPU’s AES-NI instruction-set too. So the 2x gap and the ~500-call ceiling are not about missing CPU features; they’re due to NSS’s per-operation PKCS#11 context creation and use of global locks.

What’s the fix?

Well, libsrtp supports OpenSSL. Due to some licensing willy-waving that has been going on for years between Debian and OpenSSL, Debian chooses to use libnss rather than libssl.

So, with some moderate effort, we can rebuild the stock Debian libsrtp2 package against openssl. It’s a single compile-time flag swap: –enable-openssl

   # 1. Build tooling
   apt-get install -y build-essential devscripts dpkg-dev fakeroot

   # 2. Enable deb-src (Debian 12 default sources.list has no deb-src)
   echo "deb-src https://cdn-aws.deb.debian.org/debian bookworm main" \
     > /etc/apt/sources.list.d/libsrtp2-src.list
   apt-get update

   # 3. Fetch the source package
   mkdir -p ~/srtpbuild && cd ~/srtpbuild
   apt-get source libsrtp2=2.5.0-3
   cd libsrtp2-2.5.0

   # 4. Flip the crypto backend
   sed -i 's/--enable-nss/--enable-openssl/' debian/rules
   sed -i 's/ libnss3-dev,/ libssl-dev,/' debian/control

   # 5. Install build deps from the MODIFIED control (note the ./)
   apt-get build-dep -y ./

   # 6. Version suffix so it's distinguishable and upgradeable
   dch --local +openssl "Build against OpenSSL instead of NSS for SRTP performance"

   # 7. Build binary packages only
   dpkg-buildpackage -us -uc -b

   # 8. Install the new package
   dpkg -i libsrtp2-1_2.5.0-3+openssl1_amd64.deb

We can see it’s now linked against libcrypto (OpenSSL):

$ ldd /usr/lib/x86_64-linux-gnu/libsrtp2.so.1
        linux-vdso.so.1 (0x00007fff6f9ba000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f417d200000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f417d01e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f417d7e0000)

Re-testing under load

Performing exactly the same 1000 concurrent call (1000 encrypted channels, 1000 unencrypted channels) load test as before yields significantly better results. Not least, Asterisk didn’t stop working at ~500 concurrent.

CPU usage was 30.5%. That represents ~17% overhead for encryption. Load was spread evenly across all cores.

These results are much more what we would expect, from past experience.