Securing Celery and Redis without SSL

featured 18

Recently we wanted to deploy a bunch of Celery workers and connect them using a Redis backend for tasks scheduling and result storage.

Since the workers and Redis would be internet-facing, we wanted to make sure it was not leaking any data, so we made a few tests.

Securing the communication with Redis


Redis does not support SSL, and has no plan to support SSL. Period.

Despite a clear and informative list of ways to make your Redis instance more secure, their only suggestion is to use spiped to setup secure channels between my instances and Redis.

Since we are also building a high-availabilty system, and the number of workers can be high, this was quickly going to turn into the three utilities problem.

So, no chance for SSL, but we still noted some good tips on securing a Redis instance, so not all was lost.


For some time, we thought about deploying VPNs, and we considered OpenVPN and wireguard.

OpenVPN High-availability setup is rather complicated, and we didn’t have seamless failover (so if the route X from A to B goes down, your client needs to know that Y is also a route from A to B, but then B is called “D”).

Not something our small team would feel comfortable doing and maintaining for ages.

Wireguard is basically spiped on steroids, and is not yet “production ready” to their own confession. So not something we would recommend to our customers.

Giving up on networking

Since we couldn’t agree on a cost-effective way to encrypt the network traffic, we decided to move on encrypting data before it went on the wire.

Basically if anyone could read our traffic, we would just need to encrypt what was sent so it would be unreadable without the proper key, which is second best to having SSL on Redis.

Kombu Fernet Serializers

During our search for encrypting the content of a Celery request, we landed on this blog post from Heroku blog where they explain how to use a library with a funny name to do exactly what we wanted to do.

What a chance ! So we gave it a spin, and after some adaptations to our code, we started a dummy task … and this is the traffic we saw on the network:

23:19:03.799507 IP localhost.58354 > localhost.6379: Flags [P.], seq
194:1234, ack 66, win 12757, options [nop,nop,TS val 1301471707 ecr
1301471706], length 1040: RESP "LPUSH" "test" "{"body": "Z0FBQUFBQmJUb
0V2p5S2dfWGp6SDB1UXlHc0RRUExwVkY4VE5ReDVkYVRLQ0ZR", "content-
encoding": "utf-8", "content-type": "application/x-fernet-json",
"headers": {"lang": "py", "task": "bluesnake.tasks.simple.add", "id":
"37c8d7ce-0e62-4306-b89b-866f787c498a", "eta": null, "expires": null,
"group": null, "retries": 0, "timelimit": [null, null], "root_id":
"37c8d7ce-0e62-4306-b89b-866f787c498a", "parent_id": null, "argsrepr":
"(2, 3)", "kwargsrepr": "{}", "origin": "[email protected]"},
"properties": {"correlation_id": "37c8d7ce-0e62-4306-b89b-
866f787c498a", "reply_to": "2254248b-f276-3c6e-825d-b7c83d0e2e89",
"delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key":
"test"}, "priority": 0, "body_encoding": "base64", "delivery_tag":

If you compare to what goes normally over the network, you might wonder what is exaclty being encrypted …

23:17:06.175108 IP localhost.58345 > localhost.6379: Flags [P.], seq
194:1060, ack 66, win 12757, options [nop,nop,TS val 1301354432 ecr
1301354432], length 866: RESP "LPUSH" "test" "{"body": "W1syLCAzXSwge3
bCwgImNob3JkIjogbnVsbH1d", "content-encoding": "utf-8", "content-
type": "application/json", "headers": {"lang": "py", "task":
"bluesnake.tasks.simple.add", "id": "0be78bb8-d7dc-4dde-
a2b7-7579187ab952", "eta": null, "expires": null, "group": null,
"retries": 0, "timelimit": [null, null], "root_id": "0be78bb8-d7dc-
4dde-a2b7-7579187ab952", "parent_id": null, "argsrepr": "(2, 3)",
"kwargsrepr": "{}", "origin": "[email protected]"},
"properties": {"correlation_id": "0be78bb8-d7dc-4dde-
a2b7-7579187ab952", "reply_to": "95eaf326-0e9b-3324-a11e-
1855ba095e05", "delivery_mode": 2, "delivery_info": {"exchange": "",
"routing_key": "test"}, "priority": 0, "body_encoding": "base64",
"delivery_tag": "8f80d7e6-3a4e-417e-9243-940307ca7e4d"}}"

Turns out it’s only encrypting the body field of the message, which is usually my_function_name(argument1, argument2, ...). But since arguments are also repeated (in clear) in the argsrepr field, basically you’re just encrypting the name of your function.

And then you are leaking

  • the origin field that contains the name of the node that emitted the request
  • the task name that gives out the code path that could be sensitive
  • the argsrepr and kwargsrepr that could contain sensitive data (if you run calculations by passing sensitive data to your Celery tasks


Kombu Fernet (Redneck) Engineering

Thanks to that blog post, we learned about Fernet, and using it to encrypt data seemed indeed a good idea. Deploying keys to our workers and applications servers would not be an issue at all, it definitively makes life harder for people to lisen to our our network, and we don’t need to secure redis. If only if was not leaking so much data…

But wait. What not just leave the default encoder, and encrypt the whole thing before it gets emitted to Redis ?

Then you just decrypt the traffic as it comes back from Redis and before Celery can read it, and you’re good, no ?


The quest for the mighty serializer

You only have two ways to figure out how a library is communicating with a server: you look at where it (supposedly) connects to the server and instrument each function call until you find the one that actually sends the data, or you read the network capture, isolate the instructions (here LPUSH) that’s being used to actually push data, and grep/find the instruction on the library codebase.

Obviously the second option has more chances for success. Unfortunately I did not think this through and started with the first way.

Turns out that Celery is a quite complex piece of code, and that it implements a backend for each supported backend (AMQP, Redis, MongoDB, …). Yes, you read backend, so to store the results, while I was busy looking to encrypt the message. Needless to say that my quest was a failure.

Then a coworker pointed me at this weird LPUSH command, casually mentionned it was not used often in the codebase, and two grep later I was editing the Kombu library, which is from the team behind Celery, and handles Celery messaging.

I very quickly then came over the kombu.utils.json module that contains two lovely functionsdumps and loads that are called by Celery/Kombu to send the whole payload to Redis from the application server, and retrieve it from the worker.

Literally 5 minutes later I managed to use Fernet to encode and decode traffic going to Redis and now it looks like this:

23:15:47.759061 IP localhost.58336 > localhost.6379: Flags [P.], seq
194:1648, ack 66, win 12757, options [nop,nop,TS val 1301276260 ecr
1301276260], length 1454: RESP "LPUSH" "test"


This big blob is the previous payload, Fernet-encrypted, and readable only by one of our workers or application servers.

Time spent on the issue: 4h, which is actually pretty fast compared to the time of setting up and operating a HA VPN network.

I still haven’t checked that this is sufficient to encrypt the result of the tasks, but if it does, I’ll probably release a piece of code to patch Kombu so you can store data in Redis in a secure manner, and like me enjoy using Celery over the internet safely.