Docker daemon fails to start after container forced termination due to stale PID file

[taruishi-ma]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

N/A (not using Kubernetes controller, running dind image directly on Docker)

Deployment Method

Other

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Start an actions-runner-dind container with the command above
2. Verify inner Docker is working: docker exec github-runner-1 docker ps (should succeed)
3. Forcefully terminate the container: docker kill github-runner-1
4. Restart the container: docker start github-runner-1
5. Check inner Docker: docker exec github-runner-1 docker ps
   -> Fails with: "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

Describe the bug

When an actions-runner-dind container is forcefully terminated (e.g., via docker kill, VM preemption, or system crash), the inner Docker daemon's PID file (/var/run/docker.pid) remains inside the container. On subsequent container restart, the inner Docker daemon fails to start because it detects the stale PID file.

This is a common scenario when running self-hosted runners on Spot/Preemptible VMs, where the VM can be terminated at any time without graceful shutdown.

Root Cause:
When the container is killed with SIGKILL, the inner Docker daemon doesn't have a chance to clean up /var/run/docker.pid. When the container restarts, the stale PID file prevents the new dockerd process from starting.

Describe the expected behavior

The inner Docker daemon should start successfully after container restart, even if the container was previously forcefully terminated.

Additional Context

Environment:

• Image: ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner-dind:ubuntu-22.04
• Runner Version: 2.320.0
• Platform: GCP Spot VMs (both amd64 and arm64)
• Docker version on host: 24.x

Workaround:
After container restart, manually clean up the PID file and restart dockerd:

# For a single runner
docker exec github-runner-1 bash -c 'sudo rm -f /var/run/docker.pid && sudo dockerd &'
sleep 3

# For all runners
docker ps -a --format json | jq .Names | grep github-runner | cut -d'"' -f2 | while read runner; do
  echo $runner
  docker exec $runner docker ps 2>&1 || \
    docker exec $runner bash -c 'sudo rm -f /var/run/docker.pid && sudo dockerd & sleep 3'
done

Suggested Fix:
The container's entrypoint script should clean up stale PID files before starting the Docker daemon:

# In the entrypoint script, before starting dockerd
rm -f /var/run/docker.pid /var/run/containerd/containerd.pid

Related Issues:

• Intermittently getting "Cannot connect to the Docker daemon at unix:///var/run/docker.sock" #3794 - Intermittently getting "Cannot connect to the Docker daemon" (different root cause: iptables)
• Runners created with actions-runner-controller in we have a lot of pods with errors: "Cannot connect to the Docker daemon at unix:///run/docker.sock. Is the docker daemon running?" #3257 - Cannot connect to the Docker daemon (general issue)
• How to make sure docker daemon sucessfully started even through PID file existed moby/moby#46988 - How to make sure docker daemon successfully started even through PID file existed

Controller Logs

N/A (not using Kubernetes controller)

Runner Pod Logs

# After forced termination and restart, checking docker inside container:
$ docker exec github-runner-1 docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

# The stale PID file exists:
$ docker exec github-runner-1 ls -la /var/run/docker.pid
-rw-r--r-- 1 root root 5 Jan 25 07:00 /var/run/docker.pid

$ docker exec github-runner-1 cat /var/run/docker.pid
123  # Old PID from before termination

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[taruishi-ma]

Additional context: Why the fix should be in the dind image entrypoint

I'd like to add some context regarding the Docker PID file behavior.

As noted by @thaJeztah in moby/moby#46988, the PID file check in Docker is intentional - it prevents multiple daemon instances from starting and potentially corrupting the data store. On a clean shutdown, the PID file is properly removed.

However, the Docker-in-Docker scenario is fundamentally different:

1. SIGKILL cannot be caught: When a container is forcefully terminated via docker kill, VM preemption, or system crash, the inner processes receive SIGKILL which cannot be caught or handled. There's no opportunity to clean up the PID file.

2. Container restart guarantees process termination: After a container restart, the old dockerd process is guaranteed to not be running - all processes inside the container were terminated when the container stopped. The PID file is definitively stale.

3. Safe to clean up: Unlike the host Docker daemon scenario, cleaning up /var/run/docker.pid at container startup is completely safe because we know with certainty that no old dockerd process exists.

Suggested fix location: The actions-runner-dind image's entrypoint script should remove stale PID files before starting the Docker daemon:

# At the beginning of the entrypoint, before starting dockerd
rm -f /var/run/docker.pid /var/run/containerd/containerd.pid

This is not a change to Docker core behavior (which is correct), but rather a necessary adaptation for the Docker-in-Docker use case where container lifecycle differs from host daemon lifecycle.

cc @thaJeztah - Would appreciate your thoughts on this approach for the dind image.

[thaJeztah]

Yeah, I guess this issue gets amplified because /run (/var/run) is currently not a tmpfs inside the container, so the content is preserved between starts.

That actually made me go down the rabbit hole a bit, because I thought we used a tmpfs for /run, but looks like the containerd OCI defaults and Docker / Moby defaults diverged, so I opened a PR to consider making it a tmpfs;

• OCI: make /run a tmpfs moby/moby#51950

Removing the pid-file(s) seems reasonable to me; I recall something similar is done in the official Apache2 (https://hub.docker.com/_/httpd) image;
https://github.com/docker-library/httpd/blob/b8bf24dec3fb94efd3d81ac495bea8247d5115d9/2.4/httpd-foreground#L4-L5

That said; both dockerd and containerd should check if the PID that's stored inside the PID-file is a running process, so it would only fail to start if a pid-file is left behind AND there's process running already with that PID 🤔 but I guess it's .. possible.

docker run -dq --privileged docker:29.2-dind
89ade5115756823952e9dd88cde6541a4b470c378051db30f90d78da428e274c

docker container diff 89ade5115756823952e9dd88cde6541a4b470c378051db30f90d78da428e274c | grep '.pid'
A /run/docker/containerd/containerd.pid
A /run/docker.pid

docker container kill -s KILL 89ade5115756823952e9dd88cde6541a4b470c378051db30f90d78da428e274c
docker container start 89ade5115756823952e9dd88cde6541a4b470c378051db30f90d78da428e274c

docker container logs 89ade5115756823952e9dd88cde6541a4b470c378051db30f90d78da428e274c 2>&1 | grep 'Starting up'
time="2026-01-27T14:02:56.505943760Z" level=info msg="Starting up"
time="2026-01-27T14:04:28.295384011Z" level=info msg="Starting up"

In general, sig-KILL-ing seems like a large hammer if the container is not expected to be ephemeral (i.e., if it's expected to be ran after); if containers are running, it's not impossible that other pid-files are left behind (for containers itself) or that state gets corrupted. So until a tmpfs is used by default, it's still possible to have issues in other ways if the container is forcefully terminated.

docker diff 42c3add442b1 | grep '\.pid'
A /run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/887cd34aea9baea7c16f0d42f97a5bafff3f3358f1b27eada20fce3128f5a9b7/init.pid
A /run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/d23736561eb66a8deeae6ff0750a9960bd1ea59a6e0f709243a966c077184bb4/init.pid
A /run/docker/containerd/containerd.pid
A /run/docker.pid

[thaJeztah]

Removing the pid-file(s) seems reasonable to me; I recall something similar is done in the official Apache2 (https://hub.docker.com/_/httpd) image;
https://github.com/docker-library/httpd/blob/b8bf24dec3fb94efd3d81ac495bea8247d5115d9/2.4/httpd-foreground#L4-L5

@taruishi-ma perhaps you can open a ticket in the issue tracker for the DIND image for discussion? https://github.com/docker-library/docker/issues

[taruishi-ma]

I did some reproduction testing to understand why @thaJeztah 's test worked but we're seeing failures in production.
​

What I found

​
I compared the official docker:dind (v29.2.0) with actions-runner-dind:ubuntu-22.04 (v28.0.4, hardcoded here).
​
After docker kill + docker start:

• Official docker:dind: Inner Docker works
• actions-runner-dind: Inner Docker fails with "Cannot connect to the Docker daemon"
  ​

Root cause

​
The official docker:dind has this in its entrypoint (source):
​

# explicitly remove Docker's default PID file to ensure that it can start properly if it was stopped uncleanly
find /run /var/run -iname 'docker*.pid' -delete || :

​
The actions-runner-dind entrypoint (source) doesn't have this. It just starts dockerd directly (line 48) without cleaning up stale PID files.
​

Suggested fix

​
Add the same cleanup before starting dockerd in entrypoint-dind.sh:
​

+ # Remove stale PID files to ensure dockerd can start after unclean shutdown
+ find /run /var/run -iname 'docker*.pid' -delete 2>/dev/null || :
+ find /run /var/run -iname 'containerd*.pid' -delete 2>/dev/null || :
​
log.debug 'Starting Docker daemon'
sudo /usr/bin/dockerd &

​
This is exactly what the official Docker image does, and it's safe because at container startup there's no way the old dockerd process could still be running.