My kind node was NotReady, although kubelet was running. The cause was a stale API server address in kubelet’s configuration.
Names are anonymized. IPs are documentation examples. Replace them with your verified values and select the correct Kubernetes and Docker contexts.
1. Check the node conditions
kubectl describe node example-control-planeThe key output was:
Ready Unknown NodeStatusUnknown
Kubelet stopped posting node status.Unknown meant the control plane was missing node updates. It did not prove the node was out of memory or disk space.
2. Read kubelet’s errors
docker exec example-control-plane \
systemctl status kubelet --no-pager -lKubelet was running, but its logs showed:
dial tcp 192.0.2.10:6443: connect: connection refusedBoth node registration and Lease updates were failing. The process was alive, but it could not reach its API endpoint.
3. Compare the actual IP with the configured address
docker inspect example-control-plane \
--format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{println}}{{end}}'
docker exec example-control-plane \
grep 'server:' /etc/kubernetes/kubelet.confThe results revealed a mismatch:
Container IP: 192.0.2.20
Configured API: https://192.0.2.10:6443Kubelet appeared to be using an old address. Before editing the configuration, I tested the current address.
4. Verify the new endpoint
docker exec example-control-plane \
kubectl --kubeconfig=/etc/kubernetes/admin.conf \
--server=https://192.0.2.20:6443 \
--request-timeout=10s get --raw='/readyz?verbose'All readiness checks passed with TLS verification enabled.
I also used kubelet.conf with the same server override to read the node, confirming kubelet’s credentials worked. These tests did not modify the running kubelet.
5. Back up, fix, and verify
I backed up the configuration, replaced the old endpoint, and restarted kubelet:
docker exec example-control-plane sh -ec '
config=/etc/kubernetes/kubelet.conf
grep -Fq "https://192.0.2.10:6443" "$config"
backup=$(mktemp /etc/kubernetes/kubelet.conf.backup.XXXXXX)
cp -p "$config" "$backup"
printf "Backup: %s\n" "$backup"
sed -i "s#https://192\.0\.2\.10:6443#https://192.0.2.20:6443#g" "$config"
systemctl restart kubelet
'Then I checked the original symptom:
kubectl wait --for=condition=Ready \
node/example-control-plane --timeout=90s
kubectl get node example-control-plane -o wideThe node became Ready, and its reported IP updated to the current container address.
A Troubleshooting Approach
This investigation followed a sequence that can help with future incidents:
- Start with the reported condition.
NotReadyis a symptom. ReadingConditionsrevealedUnknown, which pointed toward missing node updates. - Check the responsible component. Kubelet was running, so the next question was whether it could perform its work.
- Follow the specific error.
Connection refuseddirected the investigation toward the configured API endpoint. - Compare configuration with reality. Checking the container’s actual IP against kubelet’s configuration exposed the mismatch.
- Test the hypothesis before changing anything. Temporarily overriding the endpoint verified connectivity, TLS, and credentials.
- Make a small, reversible change. Back up the configuration, correct the address, and restart the affected service.
- Verify the original symptom. Confirm that the node returns to
Ready, rather than treating a successful restart as proof of recovery.
