The Namespace is stuck the terminating state

使用 kubectl 刪除 kubernetes namespace 時,發生狀態卡在 Terminating 無法移除。原因是相關資源未被釋放,所以卡住。

下面列出 namespace cert-manager 的狀態。

1➜ kubectl describe namespace cert-manager
2Name:         cert-manager
3Labels:       <none>
4Annotations:  <none>
5Status:       Terminating
6
7No resource quota.
8
9No LimitRange resource.

下面將匯出 namespace cert-manager 當前的定義 json。

1➜ kubectl get namespace cert-manager -o json > tmp.json

原因

從 json 檔案中可看到目前 namespace 中的狀況,有 conditions 執行未完成,所以造成 namespace 卡住。

 1"status": {
 2    "conditions": [
 3        {
 4            "lastTransitionTime": "2021-08-19T09:14:05Z",
 5            "message": "Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request",
 6            "reason": "DiscoveryFailed",
 7            "status": "True",
 8            "type": "NamespaceDeletionDiscoveryFailure"
 9        },
10        {
11            "lastTransitionTime": "2021-08-19T09:14:06Z",
12            "message": "All legacy kube types successfully parsed",
13            "reason": "ParsedGroupVersions",
14            "status": "False",
15            "type": "NamespaceDeletionGroupVersionParsingFailure"
16        },
17        {
18            "lastTransitionTime": "2021-08-19T09:14:06Z",
19            "message": "All content successfully deleted, may be waiting on finalization",
20            "reason": "ContentDeleted",
21            "status": "False",
22            "type": "NamespaceDeletionContentFailure"
23        },
24        {
25            "lastTransitionTime": "2021-08-19T09:14:06Z",
26            "message": "All content successfully removed",
27            "reason": "ContentRemoved",
28            "status": "False",
29            "type": "NamespaceContentRemaining"
30        },
31        {
32            "lastTransitionTime": "2021-08-19T09:14:06Z",
33            "message": "All content-preserving finalizers finished",
34            "reason": "ContentHasNoFinalizers",
35            "status": "False",
36            "type": "NamespaceFinalizersRemaining"
37        }
38    ],
39    "phase": "Terminating"
40}

排除

可以看到在 json 檔中有 spec.finalizers 讀欄位,在 kubernetes 官網中有解釋 Finalizers 會等到 conditions 內的資源都移除後才會標記為 deletion,確保 namespace 擁有的資源已被釋放。

1"spec": {
2    "finalizers": [
3        "kubernetes"
4    ]
5},

我們可以手動變更定義,不要等待其他資源。搜尋 json 檔案中的 spec.finalizers,把 kubernetes 字串給刪掉,會變成像下面的結構。

1"spec": {
2    "finalizers": [
3    ]
4},

開啟 kubectl proxy,直接對 API 發變更過後的請求。完成後就換看到 namespace 被移除。

1➜ kubectl proxy &
2PID=$!
3➜ curl -X PUT http://localhost:8001/api/v1/namespaces/cert-manager/finalize -H "Content-Type: application/json" --data-binary @tmp.json
4kill $PID
comments powered by Disqus