Container Escape in Bitbucket Pipelines Kata Containers

We discussed this vulnerability as part of our weekly podcast on 20 April 2021

tl;dr Uses a known docker breakout to escape into the wrapping VM, then by replacing a logfile with a symlink you could post to locations on the host machine.

  • Kata Containers - A secure container runtime with lightweight VMs that free like containers but provider stronger isolation.
  • BitBucket Pipelines - Runs build jobs from Bitbucket repos. Uses Kata containers to separate build jobs of different users. Container host runs a Kata VM that contains several containers, the builder container, containers for various services and a Docker-in-Docker container which was privileged for executing docker commands.
  • Container Escape - The builder container could be escaped using a known privileged container escape that abused the release_agent.

Vuln 1 - Tampering With Other Builds tl;dr: /usr/bin/docker was externally provided and was r/w mounted in the VM so it could be backdoored after escaping the container.

Inside the container there are some files mounted from the host, notable /usr/bin/docker. Unfortunately this is a read-only mount. Inside the VM however, there is just a single read-write mount that is the parent directory. So you could overwrite the docker binary used by all the build containers.

Vuln 2 - Escaping the VM tl;dr: The stdout logs were mounted inside the VM, but were written to by a process on the host. Symlinking the log file, along wth triggering a log rotation to force the file to be reopened resulted in redirecting logs to any location on the host machine.

Another hostpath mount was under the /var/log/pods/… contained stdout logs from each container. These were written by a process on the host, and used by the Web UI. Using a symlink one could get the output written to any location on the host, but couldn’t overwrite files. This had to do with containerd not reopening the file after the symlink was made and ignoring errors. The new file would be created because a separate watch-dog process would see that 0.log didn’t exist (or where it pointed) and would send a message to containerd to open it. When symlinked to an existing file, this didn’t happen. This could be triggered in a race by triggering a log rotation, which could copy the log to 0.log. and tell containerd to reopen 0.log. You could race the rotation and the reopen command to get an arb host-filesystem write.