The Namespace is stuck the terminating state

Aug 19, 2021 · 2 min read · kubernetes  ·
Share on:

使用 kubectl 刪除 Kubernetes namespace 時，發生狀態卡在 Terminating 無法移除。原因是相關資源未被釋放，所以卡住。

下面列出 namespace cert-manager 的狀態。

1bash$ kubectl describe namespace cert-manager
2Name:         cert-manager
3Labels:       <none>
4Annotations:  <none>
5Status:       Terminating
6
7No resource quota.
8
9No LimitRange resource.

下面將匯出 namespace cert-manager 當前的定義 json。

1bash$ kubectl get namespace cert-manager -o json > tmp.json

原因

從 json 檔案中可看到目前 namespace 中的狀況，有 conditions 執行未完成，所以造成 namespace 卡住。

 1"status": {
 2    "conditions": [
 3        {
 4            "lastTransitionTime": "2021-08-19T09:14:05Z",
 5            "message": "Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request",
 6            "reason": "DiscoveryFailed",
 7            "status": "True",
 8            "type": "NamespaceDeletionDiscoveryFailure"
 9        },
10        {
11            "lastTransitionTime": "2021-08-19T09:14:06Z",
12            "message": "All legacy kube types successfully parsed",
13            "reason": "ParsedGroupVersions",
14            "status": "False",
15            "type": "NamespaceDeletionGroupVersionParsingFailure"
16        },
17        {
18            "lastTransitionTime": "2021-08-19T09:14:06Z",
19            "message": "All content successfully deleted, may be waiting on finalization",
20            "reason": "ContentDeleted",
21            "status": "False",
22            "type": "NamespaceDeletionContentFailure"
23        },
24        {
25            "lastTransitionTime": "2021-08-19T09:14:06Z",
26            "message": "All content successfully removed",
27            "reason": "ContentRemoved",
28            "status": "False",
29            "type": "NamespaceContentRemaining"
30        },
31        {
32            "lastTransitionTime": "2021-08-19T09:14:06Z",
33            "message": "All content-preserving finalizers finished",
34            "reason": "ContentHasNoFinalizers",
35            "status": "False",
36            "type": "NamespaceFinalizersRemaining"
37        }
38    ],
39    "phase": "Terminating"
40}

排除

可以看到在 json 檔中有 spec.finalizers 讀欄位，在 kubernetes 官網中有解釋 Finalizers 會等到 conditions 內的資源都移除後才會標記為 deletion，確保 namespace 擁有的資源已被釋放。

1"spec": {
2    "finalizers": [
3        "kubernetes"
4    ]
5},

我們可以手動變更定義，不要等待其他資源。搜尋 json 檔案中的 spec.finalizers，把 kubernetes 字串給刪掉，會變成像下面的結構。

1"spec": {
2    "finalizers": [
3    ]
4},

開啟 kubectl proxy，直接對 API 發變更過後的請求。完成後就換看到 namespace 被移除。

1bash$ kubectl proxy &
2bash$ PID=$!
3bash$ curl -X PUT http://localhost:8001/api/v1/namespaces/cert-manager/finalize -H "Content-Type: application/json" --data-binary @tmp.json
4bash$ kill $PID
comments powered by Disqus