Fixing a NotReady kind Node: A Short Debugging Walkthrough

My kind node was NotReady, although kubelet was running. The cause was a stale API server address in kubelet’s configuration.

Names are anonymized. IPs are documentation examples. Replace them with your verified values and select the correct Kubernetes and Docker contexts.

1. Check the node conditions

kubectl describe node example-control-plane

The key output was:

Ready   Unknown   NodeStatusUnknown
Kubelet stopped posting node status.

Unknown meant the control plane was missing node updates. It did not prove the node was out of memory or disk space.

2. Read kubelet’s errors

docker exec example-control-plane \
  systemctl status kubelet --no-pager -l

Kubelet was running, but its logs showed:

dial tcp 192.0.2.10:6443: connect: connection refused

Both node registration and Lease updates were failing. The process was alive, but it could not reach its API endpoint.

3. Compare the actual IP with the configured address

docker inspect example-control-plane \
  --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{println}}{{end}}'

docker exec example-control-plane \
  grep 'server:' /etc/kubernetes/kubelet.conf

The results revealed a mismatch:

Container IP:   192.0.2.20
Configured API: https://192.0.2.10:6443

Kubelet appeared to be using an old address. Before editing the configuration, I tested the current address.

4. Verify the new endpoint

docker exec example-control-plane \
  kubectl --kubeconfig=/etc/kubernetes/admin.conf \
  --server=https://192.0.2.20:6443 \
  --request-timeout=10s get --raw='/readyz?verbose'

All readiness checks passed with TLS verification enabled.

I also used kubelet.conf with the same server override to read the node, confirming kubelet’s credentials worked. These tests did not modify the running kubelet.

5. Back up, fix, and verify

I backed up the configuration, replaced the old endpoint, and restarted kubelet:

docker exec example-control-plane sh -ec '
  config=/etc/kubernetes/kubelet.conf
  grep -Fq "https://192.0.2.10:6443" "$config"

  backup=$(mktemp /etc/kubernetes/kubelet.conf.backup.XXXXXX)
  cp -p "$config" "$backup"
  printf "Backup: %s\n" "$backup"

  sed -i "s#https://192\.0\.2\.10:6443#https://192.0.2.20:6443#g" "$config"
  systemctl restart kubelet
'

Then I checked the original symptom:

kubectl wait --for=condition=Ready \
  node/example-control-plane --timeout=90s

kubectl get node example-control-plane -o wide

The node became Ready, and its reported IP updated to the current container address.

A Troubleshooting Approach

This investigation followed a sequence that can help with future incidents:

1. Start with the reported condition. NotReady is a symptom. Reading Conditions revealed Unknown, which pointed toward missing node updates.
2. Check the responsible component. Kubelet was running, so the next question was whether it could perform its work.
3. Follow the specific error. Connection refused directed the investigation toward the configured API endpoint.
4. Compare configuration with reality. Checking the container’s actual IP against kubelet’s configuration exposed the mismatch.
5. Test the hypothesis before changing anything. Temporarily overriding the endpoint verified connectivity, TLS, and credentials.
6. Make a small, reversible change. Back up the configuration, correct the address, and restart the affected service.
7. Verify the original symptom. Confirm that the node returns to Ready, rather than treating a successful restart as proof of recovery.