Namespace Cleanup Procedures
This guide provides step-by-step instructions for cleaning up Kubernetes namespaces that may be stuck or containing resources that prevent proper deletion.
Overview
Kubernetes namespaces occasionally become difficult to delete due to resources with finalizers, stuck controllers, or other issues. This guide provides systematic approaches to identify and resolve these issues.
Standard Namespace Deletion
Under normal circumstances, deleting a namespace should be straightforward:
kubectl delete namespace <namespace-name>
However, if the namespace becomes stuck in the “Terminating” state, you’ll need the procedures outlined below.
Identifying Stuck Resources
Check Namespace Status
First, verify the namespace is actually stuck:
kubectl get namespace <namespace-name>
If it shows Terminating status for more than a few minutes, it’s likely stuck.
List All Resources in the Namespace
To identify what resources might be preventing deletion:
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace-name>
Identify Resources with Finalizers
Finalizers often prevent namespace deletion. Find resources with finalizers:
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 -I{} bash -c "kubectl get {} -n <namespace-name> -o json | jq '.items[] | select(.metadata.finalizers != null and .metadata.finalizers | length > 0) | \"\(.kind) \(.metadata.name) has finalizers: \(.metadata.finalizers)\"'"
Cleanup Procedures
Method 1: Remove Finalizers from Resources
For each resource with finalizers:
kubectl patch <resource-type> <resource-name> -n <namespace-name> --type='json' -p='[{"op":"remove","path":"/metadata/finalizers"}]'
Example:
kubectl patch deployment stuck-deployment -n stuck-namespace --type='json' -p='[{"op":"remove","path":"/metadata/finalizers"}]'
Method 2: Force Remove Finalizers from Namespace
If the namespace itself has finalizers:
kubectl get namespace <namespace-name> -o json | jq '.spec.finalizers = []' > ns.json
kubectl replace --raw "/api/v1/namespaces/<namespace-name>/finalize" -f ns.json
Method 3: Remove Specific Known Problematic Resources
Clear ArgoCD Applications
kubectl -n <namespace-name> get applications -o name | xargs -I{} kubectl patch {} -n <namespace-name> --type='json' -p='[{"op":"remove","path":"/metadata/finalizers"}]'
Clear Custom Resources
If you know specific CRDs causing problems:
kubectl get <crd-type> -n <namespace-name> -o name | xargs -I{} kubectl patch {} -n <namespace-name> --type='json' -p='[{"op":"remove","path":"/metadata/finalizers"}]'
Remove Problematic Pods
kubectl get pod -n <namespace-name> -o name | xargs -I{} kubectl delete {} -n <namespace-name> --force --grace-period=0
Handling Special Cases
Persistent Volumes and Claims
PVCs often prevent namespace deletion:
# List PVCs in the namespace
kubectl get pvc -n <namespace-name>
# Remove finalizers from PVCs
kubectl get pvc -n <namespace-name> -o name | xargs -I{} kubectl patch {} -n <namespace-name> --type='json' -p='[{"op":"remove","path":"/metadata/finalizers"}]'
Webhook Configurations
If validating or mutating webhooks are causing issues:
# List webhook configurations
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
# Delete specific webhook if needed
kubectl delete validatingwebhookconfigurations <webhook-name>
Service Account Tokens
kubectl get secrets -n <namespace-name> | grep service-account-token | awk '{print $1}' | xargs -I{} kubectl delete secret {} -n <namespace-name>
Namespace Recovery Scripts
For batch cleanup operations, here’s a more comprehensive script:
#!/bin/bash
NAMESPACE=$1
# Print namespace status
echo "Namespace status:"
kubectl get namespace $NAMESPACE -o yaml
# Get all resources with finalizers
echo "Resources with finalizers:"
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 -I{} bash -c "kubectl get {} -n $NAMESPACE -o json 2>/dev/null | jq -r '.items[] | select(.metadata.finalizers != null and .metadata.finalizers | length > 0) | \"\(.kind) \(.metadata.name) has finalizers: \(.metadata.finalizers)\"'"
# Remove finalizers from all resources
echo "Removing finalizers from resources..."
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 -I{} bash -c "kubectl get {} -n $NAMESPACE -o json 2>/dev/null | jq -r '.items[] | select(.metadata.finalizers != null and .metadata.finalizers | length > 0) | \"\(.kind)/\(.metadata.name)\"'" | xargs -I{} kubectl patch {} -n $NAMESPACE --type='json' -p='[{"op":"remove","path":"/metadata/finalizers"}]'
# Force delete pods
echo "Force deleting pods..."
kubectl get pods -n $NAMESPACE -o name | xargs -I{} kubectl delete {} -n $NAMESPACE --force --grace-period=0
# Remove namespace finalizers
echo "Removing namespace finalizers..."
kubectl get namespace $NAMESPACE -o json | jq '.spec.finalizers = []' > ns.json
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ns.json
echo "Namespace status after cleanup:"
kubectl get namespace $NAMESPACE -o yaml
Save this as cleanup-namespace.sh and run with ./cleanup-namespace.sh <namespace-name>.
Prevention Best Practices
To avoid stuck namespaces in the future:
- Delete Resources Before Namespace: Delete key resources before deleting the namespace
- Use Resource Quotas: Set resource quotas to limit the number of resources
- Implement Namespace Lifecycle Policies: Create policies for namespace creation and deletion
- Regular Maintenance: Periodically check for orphaned resources
- Use Labels: Label resources consistently for easier cleanup
See Also
