Kabuto, grid computing

docker, ammonite, servers, scripting
etl, training, development, open-source
  • allows non-privileged users to run their scripts in a sandboxed environment (Docker containers)
  • scripts can be organized as parents and siblings (in a directed acyclic graph)
  • “child” scripts read their input from their “parents” ouptut
  • architectured as one “master” server (Kabuto project) and several “computing nodes” (Ammonite project)
  • computing nodes communicate over HTTP and message queues
  • fallback system designed to recover from dead computing nodes and restart the job it was doing
  • master server can be horizontally load-balanced to handle a large number of computing nodes
  • input is always read-only, this way a malicious user cannot damage existing files
rabbitmq logo strap docker python