Nanxuan Blog

Category: K8S

  • Fixing a NotReady kind Node: A Short Debugging Walkthrough

    My kind node was NotReady, although kubelet was running. The cause was a stale API server address in kubelet’s configuration.

    Names are anonymized. IPs are documentation examples. Replace them with your verified values and select the correct Kubernetes and Docker contexts.

    1. Check the node conditions

    kubectl describe node example-control-plane

    The key output was:

    Ready   Unknown   NodeStatusUnknown
Kubelet stopped posting node status.

    Unknown meant the control plane was missing node updates. It did not prove the node was out of memory or disk space.

    2. Read kubelet’s errors

    docker exec example-control-plane \
  systemctl status kubelet --no-pager -l

    Kubelet was running, but its logs showed:

    dial tcp 192.0.2.10:6443: connect: connection refused

    Both node registration and Lease updates were failing. The process was alive, but it could not reach its API endpoint.

    3. Compare the actual IP with the configured address

    docker inspect example-control-plane \
  --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{println}}{{end}}'

docker exec example-control-plane \
  grep 'server:' /etc/kubernetes/kubelet.conf

    The results revealed a mismatch:

    Container IP:   192.0.2.20
Configured API: https://192.0.2.10:6443

    Kubelet appeared to be using an old address. Before editing the configuration, I tested the current address.

    4. Verify the new endpoint

    docker exec example-control-plane \
  kubectl --kubeconfig=/etc/kubernetes/admin.conf \
  --server=https://192.0.2.20:6443 \
  --request-timeout=10s get --raw='/readyz?verbose'

    All readiness checks passed with TLS verification enabled.

    I also used kubelet.conf with the same server override to read the node, confirming kubelet’s credentials worked. These tests did not modify the running kubelet.

    5. Back up, fix, and verify

    I backed up the configuration, replaced the old endpoint, and restarted kubelet:

    docker exec example-control-plane sh -ec '
  config=/etc/kubernetes/kubelet.conf
  grep -Fq "https://192.0.2.10:6443" "$config"

  backup=$(mktemp /etc/kubernetes/kubelet.conf.backup.XXXXXX)
  cp -p "$config" "$backup"
  printf "Backup: %s\n" "$backup"

  sed -i "s#https://192\.0\.2\.10:6443#https://192.0.2.20:6443#g" "$config"
  systemctl restart kubelet
'

    Then I checked the original symptom:

    kubectl wait --for=condition=Ready \
  node/example-control-plane --timeout=90s

kubectl get node example-control-plane -o wide

    The node became Ready, and its reported IP updated to the current container address.

    A Troubleshooting Approach

    This investigation followed a sequence that can help with future incidents:

    1. Start with the reported condition. NotReady is a symptom. Reading Conditions revealed Unknown, which pointed toward missing node updates.
    2. Check the responsible component. Kubelet was running, so the next question was whether it could perform its work.
    3. Follow the specific error. Connection refused directed the investigation toward the configured API endpoint.
    4. Compare configuration with reality. Checking the container’s actual IP against kubelet’s configuration exposed the mismatch.
    5. Test the hypothesis before changing anything. Temporarily overriding the endpoint verified connectivity, TLS, and credentials.
    6. Make a small, reversible change. Back up the configuration, correct the address, and restart the affected service.
    7. Verify the original symptom. Confirm that the node returns to Ready, rather than treating a successful restart as proof of recovery.

  • Understanding Container Isolation Inside a Kubernetes Pod

    I had a few questions about containers running inside the same Kubernetes Pod:

    • Can they see each other?
    • Do they share the same network?
    • Do they share the same filesystem?
    • What do the process IDs look like inside each container?

    To verify these behaviors, I created a Pod with three containers and checked them one by one.

    Experiment Setup

    apiVersion: v1
kind: Pod
metadata:
  name: three-containers
spec:
  containers:
    - name: container-1
      image: busybox:1.36
      command: ["sh", "-c", "sleep infinity"]

    - name: container-2
      image: busybox:1.36
      command: ["sh", "-c", "sleep infinity"]

    - name: container-3
      image: busybox:1.36
      command: ["sh", "-c", "sleep infinity"]

    Processes

    First, let’s check the processes inside each container.

    Container 1

    k exec -it three-containers -c container-1 -- sh

/ # ps
PID   USER     TIME  COMMAND
   1 root      0:00 sh -c sleep infinity
  28 root      0:00 sh
  34 root      0:00 ps

    Container 2

    k exec -it three-containers -c container-2 -- sh

/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 sh -c sleep infinity
   11 root      0:00 sh
   17 root      0:00 ps

    Container 3

    k exec -it three-containers -c container-3 -- sh

/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 sh -c sleep infinity
   11 root      0:00 sh
   17 root      0:00 ps

    Each container can only see its own processes.

    They do not share processes automatically, and the main process inside each container is PID 1.

    Filesystem

    Next, let’s check the root filesystem mount information.

    Container 2

    kubectl exec three-containers -c container-2 -- \
  sh -c 'grep " / / " /proc/self/mountinfo'
    2783 2360 0:867 / / rw,relatime - overlay overlay rw,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/41/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/5717/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/5717/work,uuid=on

    Container 3

    kubectl exec three-containers -c container-3 -- \
  sh -c 'grep " / / " /proc/self/mountinfo'
    2838 2404 0:870 / / rw,relatime - overlay overlay rw,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/41/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/5718/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/5718/work,uuid=on

    The important part here is the upperdir.

    Container 2 uses:

    snapshots/5717/fs

    Container 3 uses:

    snapshots/5718/fs

    They are using different writable layers with different snapshot IDs: 5717 and 5718.

    This shows that their writable container filesystems are separate, so changes made in one container’s writable layer are not directly visible in the other container.

    Network

    Now let’s check the network namespace.

    Container 2

    kubectl exec three-containers -c container-2 -- \
  readlink /proc/1/ns/net
    net:[4026538057]

    Container 3

    kubectl exec three-containers -c container-3 -- \
  readlink /proc/1/ns/net
    net:[4026538057]

    Both containers have the same network namespace ID:

    net:[4026538057]

    This proves that the containers are sharing the same network namespace.

    Conclusion

    From these experiments, we can see that containers in the same Pod behave differently depending on the resource:

    • Processes are isolated by default.
    • Each container has its own writable filesystem layer.
    • Containers in the same Pod share the same network namespace.

    So, containers in the same Pod are isolated from each other in terms of processes and writable filesystems, while sharing the same network.