Introduction
Istio sidecar injection is not working, causing pods to start without the Envoy proxy sidecar container. This means your services aren't participating in the service mesh - no mTLS, no traffic management, no observability, no policy enforcement. The pods run but bypass all Istio features.
This can happen due to missing namespace labels, disabled injection, webhook configuration issues, pod template problems, resource constraints, or conflicts with other controllers. The impact ranges from incomplete mesh coverage to security vulnerabilities and broken traffic routing.
Symptoms
```bash # Pod running without sidecar $ kubectl get pod myapp-12345-abcde -o jsonpath='{.spec.containers[*].name}' myapp # Expected: myapp istio-proxy
# Pod describes show no istio-proxy $ kubectl describe pod myapp-12345-abcde | grep -A5 Containers: Containers: myapp: Container ID: containerd://...
# Namespace not labeled for injection $ kubectl get namespace myns -o yaml | grep istio-injection # (empty or disabled)
# Webhook errors in kube-apiserver logs E0408 18:15:00.123456 1 dispatcher.go:138] Failed calling webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.0.0.1:443: connect: connection refused
# Pod creation events show injection skipped $ kubectl get events --field-selector reason=SidecarInjection Warning 2m default pod/myapp-12345 Injection disabled by annotation
# Istiod logs show errors $ kubectl logs -n istio-system istiod-xxx | grep -i inject error: failed to inject pod: no matching template
# MutatingWebhookConfiguration issues $ kubectl get mutatingwebhookconfiguration istiod-istio-system -o yaml # May show wrong CA bundle or service reference ```
Common Causes
- 1.Namespace not labeled: The namespace doesn't have
istio-injection=enabledlabel, which is required for automatic injection. - 2.Injection disabled at pod level: Pod has annotation
sidecar.istio.io/inject: "false"explicitly disabling injection. - 3.MutatingWebhookConfiguration issues: The webhook isn't configured correctly, wrong service reference, missing CA bundle, or wrong namespace selector.
- 4.Istiod not running or accessible: The Istio control plane (istiod) isn't running, isn't reachable, or the webhook endpoint is failing.
- 5.Resource constraints: Pod doesn't have enough resources scheduled for the sidecar container, or namespace has ResourceQuota limiting containers.
- 6.HostNetwork enabled: Pods with
hostNetwork: trueare not injected by default for security reasons. - 7.Conflicting webhook: Another mutating webhook runs before Istio and modifies the pod in ways that prevent injection.
- 8.Pod template issues: Init containers, special configurations, or non-standard pod specs that the injector can't process.
- 9.Version mismatch: Istioctl/client version doesn't match Istio control plane version.
- 10.IstioOperator misconfiguration: IstioOperator config has disabled injection or has incorrect settings.
Step-by-Step Fix
- 1.Check logs for specific error messages
- 2.Verify configuration settings
- 3.Test network connectivity
- 4.Review recent changes
- 5.Apply corrective action
- 6.Verify the fix
Step 1: Verify Namespace and Pod Labels
Check injection configuration at namespace and pod level:
```bash # Check if namespace is labeled for injection kubectl get namespace -L istio-injection
# Check specific namespace kubectl get namespace myns -o yaml | grep -A2 labels: # Should show istio-injection: enabled
# Label namespace for injection kubectl label namespace myns istio-injection=enabled
# Check pod annotations kubectl get pod myapp-12345-abcde -o yaml | grep -A5 annotations:
# Common injection annotations # sidecar.istio.io/inject: "true" # Force injection # sidecar.istio.io/inject: "false" # Disable injection # traffic.sidecar.istio.io/includeOutboundIPRanges: "*" # Capture all outbound
# Check if there's a pod annotation disabling injection kubectl get pod myapp-12345-abcde -o jsonpath='{.metadata.annotations.sidecar\.istio\.io/inject}' # Empty means default, "false" means disabled
# Check for multiple labels kubectl get namespace myns -o jsonpath='{.metadata.labels}'
# Verify label is correct (case-sensitive!) kubectl label namespace myns istio-injection=enabled --overwrite ```
Set proper namespace labels:
```bash # Enable injection for namespace kubectl label namespace myns istio-injection=enabled
# Disable injection for namespace kubectl label namespace myns istio-injection=disabled
# Remove label entirely kubectl label namespace myns istio-injection-
# Set annotation on specific pod/deployment kubectl patch deployment myapp -n myns -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}'
# Or in deployment YAML ```
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
namespace: myns
spec:
template:
metadata:
annotations:
sidecar.istio.io/inject: "true"
traffic.sidecar.istio.io/includeOutboundIPRanges: "*"
proxy.istio.io/config: |
proxyStatsMatcher:
inclusionRegexps:
- ".*"
spec:
containers:
- name: myapp
image: myapp:latestStep 2: Verify Istio Control Plane
Ensure Istio control plane is running correctly:
```bash # Check istiod pods kubectl get pods -n istio-system -l app=istiod
# Should show Running status # NAME READY STATUS RESTARTS AGE # istiod-xxxxxxxxxx-xxxxx 1/1 Running 0 1h
# Check istiod logs for errors kubectl logs -n istio-system -l app=istiod --tail=100
# Check istiod is ready kubectl exec -n istio-system deploy/istiod -- pilot-agent ready
# Check Istio version istioctl version kubectl exec -n istio-system deploy/istiod -- pilot-agent version
# Check Istio components istioctl verify-install
# Check istiod services kubectl get svc -n istio-system
# Check istiod endpoints kubectl get endpoints -n istio-system istiod
# Check webhook service kubectl get svc -n istio-system istiod -o yaml
# Test webhook connectivity kubectl run test-webhook --rm -it --image=curlimages/curl -- curl -k https://istiod.istio-system.svc:15017/inject -X POST -d '{}' ```
Check IstioOperator configuration:
```bash # Get IstioOperator config kubectl get istiooperator -n istio-system -o yaml
# Check if injection is enabled kubectl get istiooperator -n istio-system -o jsonpath='{.spec.values.sidecarInjectorWebhook.enabled}'
# Check proxy configuration kubectl get istiooperator -n istio-system -o jsonpath='{.spec.values.global.proxy}'
# If using Helm, check values helm get values istiod -n istio-system ```
Step 3: Verify MutatingWebhookConfiguration
Check the webhook is configured correctly:
```bash # Get webhook configuration kubectl get mutatingwebhookconfigurations
# Describe istio webhook kubectl describe mutatingwebhookconfiguration istiod-istio-system
# Get detailed config kubectl get mutatingwebhookconfiguration istiod-istio-system -o yaml
# Check CA bundle kubectl get mutatingwebhookconfiguration istiod-istio-system -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d
# Check service reference kubectl get mutatingwebhookconfiguration istiod-istio-system -o jsonpath='{.webhooks[0].clientConfig.service}'
# Check namespace selector kubectl get mutatingwebhookconfiguration istiod-istio-system -o jsonpath='{.webhooks[0].namespaceSelector}'
# Check object selector kubectl get mutatingwebhookconfiguration istiod-istio-system -o jsonpath='{.webhooks[0].objectSelector}'
# Check failure policy kubectl get mutatingwebhookconfiguration istiod-istio-system -o jsonpath='{.webhooks[0].failurePolicy}' # Should be "Fail" or "Ignore"
# Check timeout kubectl get mutatingwebhookconfiguration istiod-istio-system -o jsonpath='{.webhooks[0].timeoutSeconds}' ```
Fix webhook configuration:
```bash # If webhook is missing, reinstall or reconfigure istioctl install --set values.global.proxy.privileged=true
# Or restore webhook kubectl apply -f - <<EOF apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: istiod-istio-system labels: app: istiod istio: sidecar-injector webhooks: - name: sidecar-injector.istio.io clientConfig: service: name: istiod namespace: istio-system path: /inject port: 443 rules: - apiGroups: [""] apiVersions: ["v1"] operations: ["CREATE"] resources: ["pods"] failurePolicy: Fail timeoutSeconds: 10 namespaceSelector: matchLabels: istio-injection: enabled EOF
# Restart istiod to regenerate webhook kubectl rollout restart deployment istiod -n istio-system ```
Step 4: Debug Pod Creation
See what happens during pod creation:
```bash # Create a test pod and watch events kubectl run test-inject --image=nginx -n myns --dry-run=server -v=6 2>&1 | grep -i inject
# Check kube-apiserver logs for webhook calls kubectl logs -n kube-system kube-apiserver-xxx | grep -i webhook
# Check kube-apiserver audit logs kubectl logs -n kube-system kube-apiserver-xxx | grep -i "mutatingwebhook"
# Get injection template kubectl get configmap istio-sidecar-injector -n istio-system -o jsonpath='{.data.config}'
# Check injection config kubectl get configmap istio -n istio-system -o yaml
# Use istioctl to debug istioctl experimental precheck
# Analyze pod for injection istioctl analyze -n myns
# Check kubelet logs on node # SSH to node journalctl -u kubelet | grep -i webhook
# Dry run pod creation to see if injection happens cat <<EOF | kubectl apply -f - --dry-run=server -o yaml apiVersion: v1 kind: Pod metadata: name: test-pod namespace: myns spec: containers: - name: nginx image: nginx EOF
# The output should show istio-proxy container if injection works ```
Step 5: Check Resource Constraints
Ensure pods have resources for sidecar:
```bash # Check namespace resource quotas kubectl get resourcequota -n myns
# Describe quota kubectl describe resourcequota -n myns
# Check limit ranges kubectl get limitrange -n myns
# Check pod resources kubectl get pod myapp-12345-abcde -o yaml | grep -A10 resources:
# Check node resources kubectl describe node | grep -A5 "Allocated resources"
# If ResourceQuota limits containers, update it kubectl patch resourcequota compute-resources -n myns --type='json' -p='[{"op": "replace", "path": "/spec/hard/count/pods", "value": "20"}]' ```
Configure sidecar resources:
# In IstioOperator or values
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
proxy:
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 2000m
memory: 1024MiOr set resources per-pod:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
metadata:
annotations:
sidecar.istio.io/proxyCPU: "100m"
sidecar.istio.io/proxyMemory: "128Mi"
sidecar.istio.io/proxyCPULimit: "2000m"
sidecar.istio.io/proxyMemoryLimit: "1024Mi"
spec:
containers:
- name: myapp
image: myapp:latestStep 6: Handle Special Pod Configurations
Fix injection for special pod types:
```yaml # hostNetwork pods - injection disabled by default # Enable explicitly apiVersion: apps/v1 kind: DaemonSet metadata: name: my-daemon spec: template: metadata: annotations: sidecar.istio.io/inject: "true" spec: hostNetwork: true # Requires explicit injection containers: - name: app image: myapp:latest
# Init containers may interfere # Ensure init containers complete before sidecar starts apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: template: metadata: annotations: sidecar.istio.io/inject: "true" spec: initContainers: - name: init image: busybox command: ['sh', '-c', 'echo init'] containers: - name: myapp image: myapp:latest
# Pods with privileged containers apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: template: metadata: annotations: sidecar.istio.io/inject: "true" spec: containers: - name: myapp image: myapp:latest securityContext: privileged: true ```
Step 7: Verify Injection Is Working
Confirm sidecar is injected and working:
```bash # Check pod containers kubectl get pod myapp-12345-abcde -o jsonpath='{.spec.containers[*].name}' # Should include: myapp istio-proxy
# Check init containers kubectl get pod myapp-12345-abcde -o jsonpath='{.spec.initContainers[*].name}' # Should include: istio-init
# Check proxy status istioctl proxy-status
# Check proxy config istioctl proxy-config cluster myapp-12345-abcde.myns
# Check proxy listeners istioctl proxy-config listener myapp-12345-abcde.myns
# Check proxy routes istioctl proxy-config route myapp-12345-abcde.myns
# Check mTLS status istioctl authn tls-check myapp-12345-abcde.myns
# Check envoy admin interface kubectl exec myapp-12345-abcde -n myns -c istio-proxy -- curl localhost:15000/server_info
# Check proxy logs kubectl logs myapp-12345-abcde -n myns -c istio-proxy
# Check for mTLS traffic kubectl logs myapp-12345-abcde -n myns -c istio-proxy | grep -i tls ```
Step 8: Common Injection Scenarios
Handle specific injection scenarios:
```yaml # Exclude certain IP ranges from redirection apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: template: metadata: annotations: traffic.sidecar.istio.io/excludeOutboundIPRanges: "10.0.0.0/8" traffic.sidecar.istio.io/includeOutboundIPRanges: "*" spec: containers: - name: myapp image: myapp:latest
# Exclude ports apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: template: metadata: annotations: traffic.sidecar.istio.io/excludeInboundPorts: "3306,5432" traffic.sidecar.istio.io/excludeOutboundPorts: "3306" spec: containers: - name: myapp image: myapp:latest
# Custom proxy configuration apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: template: metadata: annotations: proxy.istio.io/config: | proxyStatsMatcher: inclusionRegexps: - ".*" terminationDrainDuration: 30s concurrency: 2 spec: containers: - name: myapp image: myapp:latest ```
Step 9: Reinstall or Reset Istio
If all else fails, reinstall:
```bash # Backup current config kubectl get istiooperator -n istio-system -o yaml > istio-backup.yaml kubectl get mutatingwebhookconfiguration istiod-istio-system -o yaml > webhook-backup.yaml
# Check what's installed istioctl manifest generate > current-manifest.yaml
# Uninstall Istio istioctl uninstall --purge -y
# Clean up CRDs kubectl delete crd istiooperators.install.istio.io --ignore-not-found kubectl delete crd istiooperators.install.istio.io --ignore-not-found
# Clean up namespace kubectl delete namespace istio-system --ignore-not-found
# Reinstall istioctl install --set values.global.mtls.enabled=true \ --set values.global.proxy.privileged=true \ --set profile=default
# Wait for istiod kubectl rollout status deployment istiod -n istio-system
# Verify installation istioctl verify-install
# Re-label namespaces kubectl label namespace myns istio-injection=enabled --overwrite
# Restart pods to get injection kubectl rollout restart deployment -n myns ```
Step 10: Create Diagnostic Runbook
Build comprehensive troubleshooting:
```bash #!/bin/bash # diagnose_istio_injection.sh
NAMESPACE=${1:-"default"} POD=${2:-""}
echo "=== Istio Sidecar Injection Diagnostic ===" echo "Namespace: $NAMESPACE" if [ -n "$POD" ]; then echo "Pod: $POD" fi echo ""
echo "1. Istio Control Plane Status" echo "-----------------------------" kubectl get pods -n istio-system -l app=istiod echo ""
echo "2. Namespace Labels" echo "-------------------" kubectl get namespace $NAMESPACE -o yaml | grep -A3 labels: echo ""
echo "3. MutatingWebhookConfiguration" echo "--------------------------------" kubectl get mutatingwebhookconfiguration | grep istio echo ""
echo "4. Webhook Details" echo "------------------" kubectl get mutatingwebhookconfiguration istiod-istio-system -o jsonpath='{.webhooks[0].clientConfig.service}' 2>/dev/null || echo "Not found" echo "" echo ""
echo "5. Test Pod Creation" echo "--------------------" cat <<EOF | kubectl apply -f - --dry-run=server -o yaml 2>&1 | grep -E "containers:|- name:" | head -10 apiVersion: v1 kind: Pod metadata: name: test-inject namespace: $NAMESPACE spec: containers: - name: nginx image: nginx EOF echo ""
if [ -n "$POD" ]; then echo "6. Specified Pod Analysis" echo "-------------------------" echo "Containers:" kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.spec.containers[*].name}' echo "" echo "Init Containers:" kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.spec.initContainers[*].name}' echo "" echo "Annotations:" kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.metadata.annotations}' | jq . echo "" fi
echo "7. Recent Injection Events" echo "--------------------------" kubectl get events -n $NAMESPACE --field-selector reason=SidecarInjection 2>/dev/null || echo "No events" echo ""
echo "8. istioctl Precheck" echo "--------------------" istioctl experimental precheck 2>&1 | head -20 echo ""
echo "=== Diagnostic Complete ===" ```
Prevention
| Step | Action | Verified |
|---|---|---|
| 1 | Verified namespace and pod labels | ☐ |
| 2 | Verified Istio control plane | ☐ |
| 3 | Verified MutatingWebhookConfiguration | ☐ |
| 4 | Debugged pod creation | ☐ |
| 5 | Checked resource constraints | ☐ |
| 6 | Handled special pod configurations | ☐ |
| 7 | Verified injection is working | ☐ |
| 8 | Handled common injection scenarios | ☐ |
| 9 | Reinstalled or reset Istio | ☐ |
| 10 | Created diagnostic runbook | ☐ |
Verification
- 1.Check namespace label:
- 2.```bash
- 3.kubectl get namespace myns -L istio-injection
- 4.
` - 5.Check pod has sidecar:
- 6.```bash
- 7.kubectl get pod myapp-xxx -o jsonpath='{.spec.containers[*].name}'
- 8.# Should include istio-proxy
- 9.
` - 10.Check proxy status:
- 11.```bash
- 12.istioctl proxy-status | grep myapp
- 13.
` - 14.Test mTLS:
- 15.```bash
- 16.istioctl authn tls-check myapp-xxx.myns
- 17.
`
Related Issues
- [Fix Istio Gateway 503 Service Unavailable](/articles/fix-istio-gateway-503-service-unavailable)
- [Fix Istio Virtual Service Not Routing](/articles/fix-istio-virtual-service-not-routing)
- [Fix Istio mTLS Not Working](/articles/fix-istio-mtls-not-working)
- [Fix Kubernetes Service Not Accessible](/articles/fix-kubernetes-service-not-accessible)
- [Fix Istio Traffic Not Reaching Pods](/articles/fix-istio-traffic-not-reaching-pods)
Related Articles
- [Fix Distributed Tracing Spans Missing Requests Issue in Service Mesh](distributed-tracing-spans-missing-requests)
- [Fix Envoy Proxy Connection Refused Upstream Issue in Service Mesh](envoy-proxy-connection-refused-upstream)
- [Fix Cilium Network Policy Not Enforcing](fix-cilium-network-policy-not-enforcing)
- [Fix Consul Agent Not Starting](fix-consul-agent-not-starting)
- [Fix Fix Envoy Filter Not Applying Issue in Service Mesh](fix-envoy-filter-not-applying)
<script type="application/ld+json"> { "@context": "https://schema.org", "@type": "TechArticle", "headline": "Fix Istio Sidecar Injection Not Working", "description": "Learn how to fix Istio sidecar injection not working. Includes namespace labeling, webhook configuration, pod template issues, and injection debugging.", "url": "https://www.fixwikihub.com/fix-istio-sidecar-injection-not-working", "publisher": { "@type": "Organization", "name": "FixWikiHub", "url": "https://www.fixwikihub.com" }, "author": { "@type": "Person", "name": "FixWikiHub Editorial Team" }, "datePublished": "2025-12-13T09:39:12.282Z", "dateModified": "2025-12-13T09:39:12.282Z" } </script>