Your Kubernetes cluster has stopped working. kubectl commands return certificate errors. The API server is rejecting connections because its certificates have expired. This typically happens after one year if certificates weren't renewed.
Certificate expiration is a critical issue that renders your cluster inaccessible. Let's diagnose and fix this.
Introduction
This article covers troubleshooting steps and solutions for Fix Kubernetes API Server Certificate Expired. The error typically occurs in production environments and can cause service disruptions if not addressed promptly.
Symptoms
Common error messages include:
# On control plane node
kubeadm certs check-expirationCERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Apr 04, 2026 10:00 UTC <1d ca no
apiserver Apr 04, 2026 10:00 UTC <1d ca no
apiserver-etcd-client Apr 04, 2026 10:00 UTC <1d etcd-ca no
apiserver-kubelet-client Apr 04, 2026 10:00 UTC <1d ca no
controller-manager.conf Apr 04, 2026 10:00 UTC <1d ca no
etcd-healthcheck-client Apr 04, 2026 10:00 UTC <1d etcd-ca no
etcd-peer Apr 04, 2026 10:00 UTC <1d etcd-ca no
etcd-server Apr 04, 2026 10:00 UTC <1d etcd-ca no
front-proxy-client Apr 04, 2026 10:00 UTC <1d front-proxy-ca no
scheduler.conf Apr 04, 2026 10:00 UTC <1d ca no```bash # Check API server pod status kubectl get pods -n kube-system -l component=kube-apiserver
# If this fails, check directly on control plane crictl pods | grep kube-apiserver ```
Common Causes
- Configuration misconfiguration
- Missing or incorrect credentials
- Network connectivity issues
- Version compatibility problems
- Resource exhaustion or limits
- Permission or access denied
Step-by-Step Fix
- 1.Check logs for specific error messages
- 2.Verify configuration settings
- 3.Test network connectivity
- 4.Review recent changes
- 5.Apply corrective action
- 6.Verify the fix
Understanding Kubernetes Certificates
Kubernetes uses multiple certificates: - API server certificate: For securing API server communication - Kubelet certificates: For node-to-control-plane communication - etcd certificates: For etcd cluster communication - Front-proxy certificate: For API aggregation layer
These certificates are typically valid for one year and must be renewed before expiration.
Step 1: Identify Certificate Expiration
Check certificate expiration status:
# On control plane node
kubeadm certs check-expirationOutput:
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Apr 04, 2026 10:00 UTC <1d ca no
apiserver Apr 04, 2026 10:00 UTC <1d ca no
apiserver-etcd-client Apr 04, 2026 10:00 UTC <1d etcd-ca no
apiserver-kubelet-client Apr 04, 2026 10:00 UTC <1d ca no
controller-manager.conf Apr 04, 2026 10:00 UTC <1d ca no
etcd-healthcheck-client Apr 04, 2026 10:00 UTC <1d etcd-ca no
etcd-peer Apr 04, 2026 10:00 UTC <1d etcd-ca no
etcd-server Apr 04, 2026 10:00 UTC <1d etcd-ca no
front-proxy-client Apr 04, 2026 10:00 UTC <1d front-proxy-ca no
scheduler.conf Apr 04, 2026 10:00 UTC <1d ca noThe "RESIDUAL TIME" column shows how much time remains. If it shows negative or expired, the certificates need immediate renewal.
Step 2: Check API Server Status
When certificates expire, the API server may still be running but rejecting connections:
```bash # Check API server pod status kubectl get pods -n kube-system -l component=kube-apiserver
# If this fails, check directly on control plane crictl pods | grep kube-apiserver ```
Check kube-apiserver logs:
```bash kubectl logs -n kube-system kube-apiserver-control-plane
# Or directly: journalctl -u kube-apiserver -n 50 ```
Look for certificate errors:
x509: certificate has expired or is not yet validStep 3: Renew Control Plane Certificates
On the control plane node, renew all certificates:
# Renew all certificates
kubeadm certs renew allOutput:
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] certificate renewed: admin.conf
[renew] certificate renewed: apiserver
[renew] certificate renewed: apiserver-etcd-client
[renew] certificate renewed: apiserver-kubelet-client
[renew] certificate renewed: controller-manager.conf
[renew] certificate renewed: etcd-healthcheck-client
[renew] certificate renewed: etcd-peer
[renew] certificate renewed: etcd-server
[renew] certificate renewed: front-proxy-client
[renew] certificate renewed: scheduler.confStep 4: Restart Control Plane Components
After renewing certificates, restart the affected pods:
```bash # Restart API server kubectl rollout restart deployment kube-apiserver -n kube-system
# Or kill the pod to force restart kubectl delete pod kube-apiserver-control-plane -n kube-system ```
For static pods managed by kubelet:
# On control plane node
# Move manifest files to trigger restart
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sleep 5
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/Restart other control plane components:
```bash # Restart controller manager kubectl delete pod kube-controller-manager-control-plane -n kube-system
# Restart scheduler kubectl delete pod kube-scheduler-control-plane -n kube-system
# Restart etcd kubectl delete pod etcd-control-plane -n kube-system ```
Step 5: Update kubeconfig Files
After certificate renewal, update your kubeconfig:
```bash # Copy new admin.conf to your kubeconfig location cp /etc/kubernetes/admin.conf ~/.kube/config
# Or for other users sudo cp /etc/kubernetes/admin.conf /home/user/.kube/config sudo chown user:user /home/user/.kube/config ```
Verify the new configuration works:
kubectl get nodesStep 6: Fix Kubelet Certificates on Worker Nodes
Worker node kubelet certificates also expire. On each worker node:
# Check kubelet certificate
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -text -noout | grep -A 2 ValidityIf expired, the kubelet needs new certificates:
```bash # On control plane, generate new bootstrap token kubeadm token create --print-join-command
# On worker node, delete old certs and restart kubelet rm /var/lib/kubelet/pki/kubelet-client-current.pem systemctl restart kubelet ```
The kubelet will request a new certificate. Approve the CSR on control plane:
```bash # List pending CSRs kubectl get csr
# Approve the kubelet CSR kubectl certificate approve <csr-name> ```
Alternatively, manually renew kubelet certs:
```bash # On control plane kubeadm kubeconfig user --org system:nodes --client-name system:node:worker-1 > kubelet.conf
# Copy to worker node and restart scp kubelet.conf worker-1:/var/lib/kubelet/config.yaml ssh worker-1 "systemctl restart kubelet" ```
Step 7: Fix etcd Certificate Issues
If etcd certificates expired, etcd might not start:
```bash # Check etcd pod logs kubectl logs -n kube-system etcd-control-plane
# Look for cert errors # "x509: certificate has expired" ```
Etcd certificates were renewed in step 3. Restart etcd:
```bash # Kill etcd pod to restart kubectl delete pod etcd-control-plane -n kube-system
# Wait for new pod kubectl get pods -n kube-system -w | grep etcd ```
Verify etcd is healthy:
# Check etcd health
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
--key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
endpoint healthStep 8: Verify Cluster Health
After all certificates are renewed, verify the cluster:
```bash # Check nodes are Ready kubectl get nodes
# Check all system pods running kubectl get pods -n kube-system
# Check component statuses kubectl get componentstatuses ```
Output should show:
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}Step 9: Update Client Configurations
Update kubeconfig for all clients that access the cluster:
```bash # For kubectl on other machines scp /etc/kubernetes/admin.conf user@client-machine:~/.kube/config
# For CI/CD systems, update the kubeconfig secret kubectl create secret generic kubeconfig --from-file=config=/etc/kubernetes/admin.conf -n ci-namespace --dry-run=client -o yaml | kubectl apply -f - ```
Common Issues and Solutions
Issue: kubeadm certs renew fails
error: cannot load kubeconfig /etc/kubernetes/admin.confCause: The admin.conf itself has expired, preventing kubeadm from accessing the cluster.
Solution: Use an external CA or regenerate manually:
# If using external CA, regenerate certs externally
# Otherwise, force renew without cluster access
kubeadm certs renew all --config /etc/kubernetes/kubeadm-config.yamlIssue: Worker node still NotReady after cert renewal
kubectl get nodes
# worker-1 NotReadyCause: Kubelet certificate CSR not approved.
Solution:
```bash # Check pending CSRs kubectl get csr
# Approve node CSR kubectl certificate approve node-csr-xxxx
# Wait for node to become Ready kubectl get nodes -w ```
Issue: etcd not starting after cert renewal
etcd: x509: certificate signed by unknown authorityCause: etcd certs don't match CA.
Solution: Verify etcd CA certificate:
```bash # Check etcd CA openssl x509 -in /etc/kubernetes/pki/etcd/ca.crt -text -noout | grep -A 2 Validity
# If CA expired, regenerate all etcd certs kubeadm certs renew etcd-ca kubeadm certs renew etcd-server kubeadm certs renew etcd-peer kubeadm certs renew etcd-healthcheck-client kubeadm certs renew apiserver-etcd-client ```
Issue: kubectl connection refused after cert renewal
The connection to the server x.x.x.x:6443 was refusedCause: API server pod not restarted with new certs.
Solution:
```bash # Force restart API server kubectl delete pod -n kube-system -l component=kube-apiserver
# Or on control plane directly crictl stopp $(crictl pods -q | grep apiserver) ```
Issue: Certificate rotation not automatic
Kubernetes supports automatic certificate rotation for kubelet, but not for control plane.
Solution for kubelet: Enable automatic rotation:
# kubelet configuration
rotateCertificates: trueSolution for control plane: Set up cron job:
# Check monthly
0 0 1 * * kubeadm certs check-expiration >> /var/log/kubeadm-certs.logPrevention: Set Up Certificate Monitoring
Monitor certificate expiration proactively:
```bash # Script to check and alert #!/bin/bash EXPIRY=$(kubeadm certs check-expiration | grep -E "EXPIRES" | head -1 | awk '{print $3, $4}') DAYS=$(( ( $(date -d "$EXPIRY" +%s) - $(date +%s) ) / 86400 ))
if [ $DAYS -lt 30 ]; then echo "WARNING: Certificates expire in $DAYS days" # Send alert fi ```
Or use a Prometheus exporter for certificate monitoring.
Verification
After completing all steps:
```bash # Verify certificate expiration kubeadm certs check-expiration
# All should show > 300 days residual time
# Verify cluster access kubectl get nodes kubectl get pods -A
# Verify no cert errors in logs kubectl logs -n kube-system -l component=kube-apiserver | grep -i "x509" | tail -5 ```
Key Takeaways
- 1.Kubernetes certificates expire after one year by default
- 2.Use
kubeadm certs check-expirationto monitor certificate status - 3.Renew with
kubeadm certs renew allbefore expiration - 4.Control plane pods must be restarted after certificate renewal
- 5.Worker node kubelet certs need separate renewal via CSR
- 6.Update kubeconfig files after renewing admin.conf
- 7.Set up monitoring to alert 30+ days before expiration
Certificate expiration is preventable with proper monitoring. Check your certificates monthly and renew at least 30 days before they expire.
Related Articles
- [Fix Envoy Rate Limit Configuration with envoyproxy/ratelimit](envoyproxy-ratelimit-configuration-guide)
- [Fix Fix Argocd App Not Syncing Issue in Kubernetes](fix-argocd-app-not-syncing)
- [Fix Fix Argocd Sync Conflict Issue in Kubernetes](fix-argocd-sync-conflict)
- [Fix ArgoCD Sync Timeout](fix-argocd-sync-timeout)
- [How to Fix Cilium Identity Exhaustion and Endpoint Allocation Failed](fix-cilium-identity-exhaustion)
<script type="application/ld+json"> { "@context": "https://schema.org", "@type": "TechArticle", "headline": "Fix Kubernetes API Server Certificate Expired", "description": "Learn how to renew expired Kubernetes certificates with kubeadm commands, fix kubelet certs, and restore cluster access after certificate expiration.", "url": "https://www.fixwikihub.com/fix-kubernetes-certificate-expired", "publisher": { "@type": "Organization", "name": "FixWikiHub", "url": "https://www.fixwikihub.com" }, "author": { "@type": "Person", "name": "FixWikiHub Editorial Team" }, "datePublished": "2025-11-23T16:53:39.225Z", "dateModified": "2025-11-23T16:53:39.225Z" } </script>