Human Resources

To know where to stop

featured 2

This is the kind of horror story that happens now and then.
It’s a story where pride and arrogance will lead you into a pit of despair.

This is the story of the time I tried to modify an LDAP schema inside a Docker container.

It’s not really about the technical issue itself, it’s about the sequence
of bad choices that made me waste an inconsiderate amount of time to solve a
trivial problem.

The problem is as follows:

You need to make two different changes on the schema of an OpenLDAP server running within a Docker container.
The container is not mounting the host filesystem.
You can only run one blocking command from a Docker container.

Here is the short timeline of what happened:

  • I tried to patch the files in the /etc/ldap/slapd.d/ directory before slapd was started. It took me some time to come up with a proper way to do this (and you shouldn’t do this). For some unknown reason (spent some time reading the logs) one of my change was not taken into account, even though the file was ok on the disk.
  • I thought I could use another container and connect using TCP to send ldapadd and ldapmodify commands, but it was not very user-friendly to ask my customers to go through this manual step.
  • As the application connected to LDAP can send ADD and MODIFY commands, I hard-coded the changes into my application (wrote the tests, made them pass) to do it, and added a flag to the application to take care of installing the changes when needed. But then I realized you cannot alter a schema via TCP at all. You need to connect using Unix IPC (with the ldapi:// scheme).
  • No problem, the library I’m using to connect to the LDAP server (the otherwise excellent library ldap3) says it’s possible to use IPC to connect. But then I need to mount my OpenLDAP socket from the LDAP container into the application container.
  • It turns out, you cannot mount a socket directly between two containers while using docker-compose. You need to use a third-party data volume, and make both your LDAP and app containers use it. Now I needed to configure my app to use IPC.
  • I changed my application configuration to accept a new type of LDAP server URL, and tested with the new settings. It turns out that ldap3 has a bug that prevents use of ldapi:// scheme in the host definition.

I’ll probably submit a pull request soon to the library at some point, but not today.

Today I realize that I have spent about 20h of my life trying to fix a trivial problem. And I failed.
I failed because I didn’t fail solving much more complicated problems that I thought were between me and my target.
Each problem led to another one, and I solved each of these. But in the end, I learned a lot, and didn’t make any valuable progress in what I really needed to achieve.

So I’m giving up. I have an application to ship, and I need to go back to the drawing board and find a dead-simple and reproductible way to run a few commands about 10 seconds after my LDAP container starts.

I think the solution is something like this:

(sleep 10; sh /tmp/
/usr/sbin/slapd -h "ldap:/// ldapi:///" -u openldap -g openldap -d 0

The lesson is this: we should focus on solving the initial problem, not to fix the fixes. In my quest I passed through 5 layers of issues, and I should have stopped after each of them.
But I did hang on because I thought that “once this works, it’s going to be perfect”.
I finally reached the level where the cost of winning this challenge almost dwarfed the cost of the whole application I’m writing in the first place.

After each victory, ask yourself if it actually serves the big-picture goal, or if it was a fancy distraction.

Note: on the same topic, I recommend this article by Janet Choi.