Docker in Workspaces

Docker in Workspaces

There are a few ways to run Docker within container-based Coder workspaces.

Sysbox container runtime

The Sysbox container runtime allows unprivileged users to run system-level applications, such as Docker, securely from the workspace containers. Sysbox requires a compatible Linux distribution to implement these security features. Sysbox can also be used to run systemd inside Coder workspaces. See Systemd in Docker.

Use Sysbox in Docker-based templates

After installing Sysbox on the Coder host, modify your template to use the sysbox-runc runtime:

resource "docker_container" "workspace" {
  # ...
  name    = "coder-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"
  image   = "codercom/enterprise-base:ubuntu"
  env     = ["CODER_AGENT_TOKEN=${coder_agent.main.token}"]
  command = ["sh", "-c", coder_agent.main.init_script]
  # Use the Sysbox container runtime (required)
  runtime = "sysbox-runc"
}

resource "coder_agent" "main" {
  arch           = data.coder_provisioner.me.arch
  os             = "linux"
  startup_script = < \
  --docker-server=us.gcr.io \
  --docker-username=_json_key \
  --docker-password="$(cat ./json-key-file.yaml)" \
  --docker-email=

env {
  name = "CODER_IMAGE_PULL_SECRET"
  value_from {
    secret_key_ref {
      name = ""
      key = ".dockerconfigjson"
    }
  }
}

Rootless podman

Podman is Docker alternative that is compatible with OCI containers specification. which can run rootless inside Kubernetes pods. No custom RuntimeClass is required.

Before using Podman, please review the following documentation:

• Basic setup and use of Podman in a rootless environment

• Shortcomings of Rootless Podman

1. Enable smart-device-manager to securely expose a FUSE devices to pods.

   cat < ⚠️ **Warning**: If you are using a managed Kubernetes distribution (e.g.
   > AKS, EKS, GKE), be sure to set node labels via your cloud provider.
   > Otherwise, your nodes may drop the labels and break podman functionality.
   
   

2. For systems running SELinux (typically Fedora-, CentOS-, and Red Hat-based systems), you might need to disable SELinux or set it to permissive mode.

3. Use this kubernetes-with-podman example template, or make your own.

   echo "kubernetes-with-podman" | coder templates init
   cd ./kubernetes-with-podman
   coder templates create
   

   > For more information around the requirements of rootless podman pods, see: > How to run Podman inside of Kubernetes

Rootless Podman on Bottlerocket nodes

Rootless containers rely on Linux user-namespaces. Bottlerocket disables them by default (user.max_user_namespaces = 0), so Podman commands will return an error until you raise the limit:

cannot clone: Invalid argument
user namespaces are not enabled in /proc/sys/user/max_user_namespaces

1. Add a user.max_user_namespaces value to your Bottlerocket user data to use rootless Podman on the node:

[settings.kernel.sysctl]
"user.max_user_namespaces" = "65536"

1. Reboot the node.
2. Verify that the value is more than 0:

sysctl -n user.max_user_namespaces

For Karpenter-managed Bottlerocket nodes, add the user.max_user_namespaces setting in your EC2NodeClass:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: bottlerocket-rootless
spec:
  amiFamily: Bottlerocket # required for BR-style userData
  # …
  userData: |
    [settings.kernel]
    sysctl = { "user.max_user_namespaces" = "65536" }

Privileged sidecar container

A privileged container can be added to your templates to add docker support. This may come in handy if your nodes cannot run Sysbox.

> [!WARNING] > This is insecure. Workspaces will be able to gain root access to the host machine.

Use a privileged sidecar container in Docker-based templates

resource "coder_agent" "main" {
  os             = "linux"
  arch           = "amd64"
}

resource "docker_network" "private_network" {
  name = "network-${data.coder_workspace.me.id}"
}

resource "docker_container" "dind" {
  image      = "docker:dind"
  privileged = true
  name       = "dind-${data.coder_workspace.me.id}"
  entrypoint = ["dockerd", "-H", "tcp://0.0.0.0:2375"]
  networks_advanced {
    name = docker_network.private_network.name
  }
}

resource "docker_container" "workspace" {
  count   = data.coder_workspace.me.start_count
  image   = "codercom/enterprise-base:ubuntu"
  name    = "dev-${data.coder_workspace.me.id}"
  command = ["sh", "-c", coder_agent.main.init_script]
  env = [
    "CODER_AGENT_TOKEN=${coder_agent.main.token}",
    "DOCKER_HOST=${docker_container.dind.name}:2375"
  ]
  networks_advanced {
    name = docker_network.private_network.name
  }
}

Use a privileged sidecar container in Kubernetes-based templates

terraform {
  required_providers {
    coder = {
      source  = "coder/coder"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.16.0"
    }
  }
}

variable "workspaces_namespace" {
  default = "coder-namespace"
}

data "coder_workspace" "me" {}

resource "coder_agent" "main" {
  os             = "linux"
  arch           = "amd64"
}

resource "kubernetes_pod" "main" {
  count = data.coder_workspace.me.start_count
  metadata {
    name      = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
    namespace = var.namespace
  }
  spec {
    # Run a privileged dind (Docker in Docker) container
    container {
      name  = "docker-sidecar"
      image = "docker:dind"
      security_context {
        privileged = true
        run_as_user = 0
      }
      command = ["dockerd", "-H", "tcp://127.0.0.1:2375"]
    }
    container {
      name    = "dev"
      image   = "codercom/enterprise-base:ubuntu"
      command = ["sh", "-c", coder_agent.main.init_script]
      security_context {
        run_as_user = "1000"
      }
      env {
        name  = "CODER_AGENT_TOKEN"
        value = coder_agent.main.token
      }
      # Use the Docker daemon in the "docker-sidecar" container
      env {
        name  = "DOCKER_HOST"
        value = "localhost:2375"
      }
    }
  }
}

Systemd in Docker

Additionally, Sysbox can be used to give workspaces full systemd capabilities.

After installing Sysbox on Kubernetes, modify your template to use the sysbox-runc RuntimeClass. This requires the Kubernetes Terraform provider version 2.16.0 or greater.

terraform {
  required_providers {
    coder = {
      source  = "coder/coder"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.16.0"
    }
  }
}

variable "workspaces_namespace" {
  default = "coder-namespace"
}

data "coder_workspace" "me" {}

resource "coder_agent" "main" {
  os   = "linux"
  arch = "amd64"
  dir  = "/home/coder"
}

resource "kubernetes_pod" "dev" {
  count = data.coder_workspace.me.start_count
  metadata {
    name      = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
    namespace = var.workspaces_namespace
    annotations = {
      "io.kubernetes.cri-o.userns-mode" = "auto:size=65536"
    }
  }

  spec {

    # Use Sysbox container runtime (required)
    runtime_class_name = "sysbox-runc"

    # Run as root in order to start systemd (required)
    security_context {
      run_as_user = 0
      fs_group    = 0
    }

    container {
      name = "dev"
      env {
        name  = "CODER_AGENT_TOKEN"
        value = coder_agent.main.token
      }
      image = "codercom/enterprise-base:ubuntu"
      command = ["sh", "-c", <