GitLab CI Runner on Rootless Podman

GitLab CI runner can be contained in a completely rootless environment. It can start as a non-root user, and work with a rootless Podman instance as a Docker runner. And here is how I achieved it.

My CI host configuration:

  • Ubuntu 20.04
  • Podman 3.4.2
  • GitLab Runner 14.5.0

A similar procedure can be applied to other distros as well.

Install Podman

Use the following script for installing the latest version of Podman onto a Ubuntu.

. /etc/os-release
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
curl -L "https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key" | sudo apt-key add -
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y install podman

For other distros, please refer to the official documentation.

Install GitLab Runner

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner strace

chown -R root:gitlab-runner /etc/gitlab-runner
chmod 750 /etc/gitlab-runner
chmod 640 /etc/gitlab-runner/config.toml

For other distros, please refer to the official documentation.

Note that we installed strace here, which we will be using later.

Config User Namespace Mapping

You need to allocate a range of UID/GIDs per user. You might adjust them for your needs.

sudo usermod --add-subuids 200000:265536 --add-subgids 200000:265536 gitlab-runner

cat > /etc/sysctl.d/90-podman.conf <<EOF
kernel.unprivileged_userns_clone=1
EOF
sysctl -f /etc/sysctl.d/90-podman.conf

Enable Podman Service

loginctl enable-linger gitlab-runner

mkdir -p ~gitlab-runner/.config/systemd/user/sockets.target.wants ~gitlab-runner/.config/systemd/user/multi-user.target.wants
chown gitlab-runner:gitlab-runner -R ~gitlab-runner/.config/systemd/user/\*
XDG_RUNTIME_DIR="/run/user/$(id -u gitlab-runner)" sudo -Eu gitlab-runner -- systemctl --user daemon-reload
XDG_RUNTIME_DIR="/run/user/$(id -u gitlab-runner)" sudo -Eu gitlab-runner -- systemctl --user enable --now podman.socket podman.service
sudo -u gitlab-runner -- podman network create podman

Then we need to verify if Podman is enabled correctly:

sudo su - gitlab-runner bash -c 'podman system info'
sudo su - gitlab-runner bash -c 'podman run --rm hello-world'

Enable GitLab Runner Service

systemctl stop gitlab-runner
mkdir -p /etc/systemd/system/gitlab-runner.service.d
cat > /etc/systemd/system/gitlab-runner.service.d/override.conf <<EOF
[Service]
User=gitlab-runner
Group=gitlab-runner
ExecStart=
ExecStart=strace -e getuid,getgid -e inject=getuid:retval=0 -e inject=getgid:retval=0 -- /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner

EOF
systemctl daemon-reload
systemctl enable --now gitlab-runner

Run gitlab-runner register as usual. After registration, set runners[].docker.host = "unix:////run/user/996/podman.sock", then restart the runner service. (996 is the UID for gitlab-runner user. It might differ in your environment. )

Note: the weird strace command tricks GitLab runner into thinking it is started by the root user, so it launches into system mode. By default, the runner will be started in user mode which is not helpful in our use case.

Known Issues

CentOS 6 containers does not work

If you need to run CentOS 6 or other similar old Docker containers, you need to append vsyscall=emulate to the kernel command line and reboot.

Not able to pull container using short names

cat > /etc/containers/registries.conf <<EOF
unqualified-search-registries = ["docker.io"]
EOF

Using Self-signed CA

For GitLab runner to connect to an HTTPS endpoint, you only need to install the CA to the system-level certificate storage, then change runners[].url to a HTTPS URL.

For Git to successfully pull repos using HTTPS, you have to first modify the official helper images, then set runners[].docker.helper_image = "my.registry.local/gitlab/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}". If you really don’t care about security, set runners[].docker.environment = ["GIT_SSL_NO_VERIFY=true"] (very insecure, very dangerous).

For the actual CI workload to connect to HTTPS sites with self-signed CA, set runners[].docker.volumes = ["/cache", “/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro”, “/etc/ssl/certs:/etc/ssl/certs:ro”] . This might not work if your container or host OS doesn’t follow the conventions.

Changed subuid/subgid mapping, need to reset podman

If you changed subuid/subgid mapping or did not set it up correctly initially, Podman might act weird. Run the following commands to resolve (and you need to manually fix any others errors displayed when running these commands). Note the commands will completely wipe every bit of data (container, image, volumes) stored in your Podman database.

USERNAME="gitlab-runner"
USER_UID=$(id -u "$USERNAME")
rm -rf ~"$USERNAME"/.{config,local/share}/containers /run/user/"$USER_UID"/{libpod,runc,vfs-\*}
su - "$USERNAME" systemctl --user restart dbus
su - "$USERNAME" podman system reset
su - "$USERNAME" podman system migrate

Unable to upgrade to tcp, received 409

You might receive the following error:

ERROR: Job failed (system failure): prepare environment: unable to upgrade to tcp, received 409 (exec.go:59:0s). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Check your file permissions. Try disable the cache configuration on the runner and try again. The outermost cache directory might need a permission of 0755.

CNI network “podman” not found

Run sudo -u gitlab-runner -- podman network create podman.

Generic Debugging Advices

To enable verbose mode on the GitLab runner, you can either add log_level = "debug" on the top level of the config file, or use gitlab-runner --debug run -c /etc/gitlab-runner/config.toml to launch a temporary runner.

To debug Docker API errors, you can use BPF to capture traffic on the UNIX socket, or convert the socket to a TCP listener with socat TCP-LISTEN:12345,reuseaddr,fork UNIX-CLIENT:/run/user/996/podman/podman.sock, use host = "tcp://127.0.0.1:12345" in the runner config to connect, then use tcpdump -i lo -w docker.pcap tcp port 12345 to see what happened.

Bonus: Use tmpfs to run the containers

This will clear all your previous podman data, so make sure you back up important images and containers.

mkdir -p /mnt/gitlab-runner-volatile
echo "tmpfs /mnt/gitlab-runner-volatile tmpfs defaults,noatime,size=64g,uid=996,gid=996,mode=0755,huge=always" >> /etc/fstab
mount /mnt/gitlab-runner-volatile

su - gitlab-runner
podman system reset
cat > ~gitlab-runner/.config/containers/storage.conf <<EOF
[storage]
driver = "overlay"
runroot = "/mnt/gitlab-runner-volatile/podman_containers/runroot"
graphroot = "/mnt/gitlab-runner-volatile/podman_containers/graphroot"
rootless_storage_path = "/mnt/gitlab-runner-volatile/podman_containers/storage"

[storage.options.overlay]
mountopt = "nodev,metacopy=on"
mount_program = "/usr/bin/fuse-overlayfs"
EOF

XDG_RUNTIME_DIR=/run/user/$(id -u gitlab-runner) systemctl --user restart podman.service

Additional configuration required for SELinux-enabled hosts.

References

This entry was tagged Linux, GitLab on .