cAdvisor container metrics fix — offline install

For hosts where docker info shows Storage Driver: overlayfs. Replaces only the cAdvisor container — Docker and the application containers keep running.

1 · Get the image onto the host

On a machine with internet, then copy to the host. Skip if the image is already loaded.

curl -fLO https://hub.friendly-tech.com/docs/pdf/cadvisor-v0.55.1.tar.gz
sha256sum cadvisor-v0.55.1.tar.gz
# expect: 3439641369dea7121fc3ed1fb2722e979421b0bab3c574d8115fd12d977aa204
scp cadvisor-v0.55.1.tar.gz root@<host>:/tmp/

2 · On the host, as root — copy, paste, done

One block: loads the image, replaces docker-compose.yml (base64, so pasting cannot break the YAML indentation), finds the containerd socket, updates .env, restarts cAdvisor and prints the result.

cd /opt/grafana/exporters/base
[ -f /tmp/cadvisor-v0.55.1.tar.gz ] && docker load -i /tmp/cadvisor-v0.55.1.tar.gz
\cp -a docker-compose.yml docker-compose.yml.bak
echo "c2VydmljZXM6CiAgbm9kZS1leHBvcnRlcjoKICAgIGltYWdlOiBwcm9tL25vZGUtZXhwb3J0ZXI6JHtOT0RFX0VYUE9SVEVSX1ZFUlNJT046LXYxLjEwLjJ9CiAgICBjb250YWluZXJfbmFtZTogbm9kZS1leHBvcnRlcgogICAgcmVzdGFydDogdW5sZXNzLXN0b3BwZWQKICAgIG5ldHdvcmtfbW9kZTogaG9zdAogICAgdm9sdW1lczoKICAgICAgLSAvcHJvYzovaG9zdC9wcm9jOnJvCiAgICAgIC0gL3N5czovaG9zdC9zeXM6cm8KICAgICAgLSAvOi9yb290ZnM6cm8KICAgICAgLSAvcnVuL2RidXMvc3lzdGVtX2J1c19zb2NrZXQ6L3Zhci9ydW4vZGJ1cy9zeXN0ZW1fYnVzX3NvY2tldAogICAgY29tbWFuZDoKICAgICAgLSAnLS13ZWIubGlzdGVuLWFkZHJlc3M9OiR7Tk9ERV9FWFBPUlRFUl9QT1JUOi05MTAwfScKICAgICAgLSAnLS1wYXRoLnByb2Nmcz0vaG9zdC9wcm9jJwogICAgICAtICctLXBhdGgucm9vdGZzPS9yb290ZnMnCiAgICAgIC0gJy0tcGF0aC5zeXNmcz0vaG9zdC9zeXMnCiAgICAgIC0gJy0tY29sbGVjdG9yLmZpbGVzeXN0ZW0ubW91bnQtcG9pbnRzLWV4Y2x1ZGU9Xi8oc3lzfHByb2N8ZGV2fGhvc3R8ZXRjKSgkJHwvKScKICAgICAgLSAnLS1jb2xsZWN0b3IubmV0c3RhdCcKICAgICAgLSAnLS1jb2xsZWN0b3IucHJvY2Vzc2VzJwogICAgICAtICctLWNvbGxlY3Rvci5zdGF0JwogICAgICAtICctLWNvbGxlY3Rvci5pbnRlcnJ1cHRzJwogICAgICAtICctLWNvbGxlY3Rvci5zeXN0ZW1kJwogICAgICAtICctLWNvbGxlY3Rvci50Y3BzdGF0JwoKICBwcm9jZXNzLWV4cG9ydGVyOgogICAgaW1hZ2U6IG5jYWJhdG9mZi9wcm9jZXNzLWV4cG9ydGVyOiR7UFJPQ0VTU19FWFBPUlRFUl9WRVJTSU9OOi0wLjguN30KICAgIGNvbnRhaW5lcl9uYW1lOiBwcm9jZXNzLWV4cG9ydGVyCiAgICByZXN0YXJ0OiB1bmxlc3Mtc3RvcHBlZAogICAgbmV0d29ya19tb2RlOiBob3N0CiAgICBwcml2aWxlZ2VkOiB0cnVlCiAgICB2b2x1bWVzOgogICAgICAtIC9wcm9jOi9ob3N0L3Byb2M6cm8KICAgICAgLSAuL3Byb2Nlc3MtZXhwb3J0ZXIueW1sOi9jb25maWcueW1sOnJvCiAgICBjb21tYW5kOgogICAgICAtICctLXdlYi5saXN0ZW4tYWRkcmVzcz06JHtQUk9DRVNTX0VYUE9SVEVSX1BPUlQ6LTkyNTZ9JwogICAgICAtICctLXByb2Nmcz0vaG9zdC9wcm9jJwogICAgICAtICctLWNvbmZpZy5wYXRoPS9jb25maWcueW1sJwogICAgZW52aXJvbm1lbnQ6CiAgICAgIC0gIlBST0NFU1NfRVhQT1JURVJfTkFNRVNQQUNFPSR7UFJPQ0VTU19FWFBPUlRFUl9OQU1FU1BBQ0U6LXByb2Nlc3N9IgogICAgICAtICJQUk9DRVNTX0VYUE9SVEVSX1NMRUVQX1NFQ09ORFM9JHtQUk9DRVNTX0VYUE9SVEVSX1NMRUVQX1NFQ09ORFM6LTEwfSIKCiAgY2Fkdmlzb3I6CiAgICBpbWFnZTogZ2NyLmlvL2NhZHZpc29yL2NhZHZpc29yOiR7Q0FEVklTT1JfVkVSU0lPTjotdjAuNTUuMX0KICAgIGNvbnRhaW5lcl9uYW1lOiBjYWR2aXNvcgogICAgcmVzdGFydDogdW5sZXNzLXN0b3BwZWQKICAgIG5ldHdvcmtfbW9kZTogaG9zdAogICAgcHJpdmlsZWdlZDogdHJ1ZQogICAgdm9sdW1lczoKICAgICAgLSAvOi9yb290ZnM6cm8KICAgICAgLSAvdmFyL3J1bjovdmFyL3J1bjpybwogICAgICAtIC9zeXM6L3N5czpybwogICAgICAtIC92YXIvbGliL2RvY2tlci86L3Zhci9saWIvZG9ja2VyOnJvCiAgICAgIC0gL2Rldi9kaXNrLzovZGV2L2Rpc2s6cm8KICAgIGNvbW1hbmQ6CiAgICAgIC0gJy0tcG9ydD0ke0NBRFZJU09SX1BPUlQ6LTkxODN9JwogICAgICAtICctLWRvY2tlcl9vbmx5PXRydWUnCiAgICAgICMgY0Fkdmlzb3IgPj0gdjAuNTQuMCByZWFkcyBjb250YWluZXIgcm9vdGZzIGZyb20gdGhlIGNvbnRhaW5lcmQgc3BlYzoKICAgICAgIyB0aGlzIG11c3QgcG9pbnQgYXQgdGhlIHNvY2tldCBkb2NrZXJkJ3MgY29udGFpbmVyZCBsaXN0ZW5zIG9uLgogICAgICAtICctLWNvbnRhaW5lcmQ9JHtDT05UQUlORVJEX1NPQ0s6LS9ydW4vY29udGFpbmVyZC9jb250YWluZXJkLnNvY2t9JwogICAgICAtICctLWhvdXNla2VlcGluZ19pbnRlcnZhbD0zMHMnCiAgICAgIC0gJy0tZGlzYWJsZV9tZXRyaWNzPXBlcmNwdSxzY2hlZCx0Y3AsdWRwLGRpc2ssZGlza0lPLGh1Z2V0bGIscmVmZXJlbmNlZF9tZW1vcnksY3B1X3RvcG9sb2d5LHJlc2N0cmwnCg==" | base64 -d > docker-compose.yml
SOCK=$(find /run -maxdepth 4 -type s -name 'containerd*.sock' ! -name '*-debug.sock' | head -1)
grep -q '^CADVISOR_VERSION=' .env && sed -i 's|^CADVISOR_VERSION=.*|CADVISOR_VERSION=v0.55.1|' .env || echo 'CADVISOR_VERSION=v0.55.1' >> .env
grep -q '^CONTAINERD_SOCK=' .env && sed -i "s|^CONTAINERD_SOCK=.*|CONTAINERD_SOCK=$SOCK|" .env || echo "CONTAINERD_SOCK=$SOCK" >> .env
echo "containerd socket: $SOCK"
docker compose up -d cadvisor
sleep 20
docker logs --since 2m cadvisor 2>&1 | grep 'docker container factory'
echo "layer errors: $(docker logs --since 2m cadvisor 2>&1 | grep -c 'read-write layer ID')"
echo "named containers: $(curl -s 127.0.0.1:9183/metrics | grep -o 'name="[^"]\+"' | sort -u | wc -l)"

3 · Expected output

containerd socket: /run/docker/containerd/containerd.sock
Registration of the docker container factory successfully
layer errors: 0
named containers: 20

"named containers" must match your container count, not 0. If the docker factory line is missing, the socket path is wrong — everything looks healthy but metrics arrive without container names; send us find /run -maxdepth 4 -type s -name '*.sock'. Grafana panels fill in within a minute; nothing to change in Prometheus. Rollback: \cp -a docker-compose.yml.bak docker-compose.yml && docker compose up -d cadvisor.

What the new docker-compose.yml contains (for reading, not for pasting)

services:
  node-exporter:
    image: prom/node-exporter:${NODE_EXPORTER_VERSION:-v1.10.2}
    container_name: node-exporter
    restart: unless-stopped
    network_mode: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
      - /run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
    command:
      - '--web.listen-address=:${NODE_EXPORTER_PORT:-9100}'
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.netstat'
      - '--collector.processes'
      - '--collector.stat'
      - '--collector.interrupts'
      - '--collector.systemd'
      - '--collector.tcpstat'

  process-exporter:
    image: ncabatoff/process-exporter:${PROCESS_EXPORTER_VERSION:-0.8.7}
    container_name: process-exporter
    restart: unless-stopped
    network_mode: host
    privileged: true
    volumes:
      - /proc:/host/proc:ro
      - ./process-exporter.yml:/config.yml:ro
    command:
      - '--web.listen-address=:${PROCESS_EXPORTER_PORT:-9256}'
      - '--procfs=/host/proc'
      - '--config.path=/config.yml'
    environment:
      - "PROCESS_EXPORTER_NAMESPACE=${PROCESS_EXPORTER_NAMESPACE:-process}"
      - "PROCESS_EXPORTER_SLEEP_SECONDS=${PROCESS_EXPORTER_SLEEP_SECONDS:-10}"

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:${CADVISOR_VERSION:-v0.55.1}
    container_name: cadvisor
    restart: unless-stopped
    network_mode: host
    privileged: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    command:
      - '--port=${CADVISOR_PORT:-9183}'
      - '--docker_only=true'
      # cAdvisor >= v0.54.0 reads container rootfs from the containerd spec:
      # this must point at the socket dockerd's containerd listens on.
      - '--containerd=${CONTAINERD_SOCK:-/run/containerd/containerd.sock}'
      - '--housekeeping_interval=30s'
      - '--disable_metrics=percpu,sched,tcp,udp,disk,diskIO,hugetlb,referenced_memory,cpu_topology,resctrl'

Only two changes versus the shipped 1.6.8 file: cAdvisor default v0.51.0 → v0.55.1, and the new --containerd=${CONTAINERD_SOCK:-/run/containerd/containerd.sock} flag. sha256 of the file: 7462deb24343f98dab9a5339d82459f4e785a26db20a0a1af900c3f80744e6f1