Killer Shell CKA - Apiserver Crash

1 개요

Killer Shell CKA - Apiserver Crash

• https://killercoda.com/killer-shell-cka/scenario/apiserver-crash
• Kubernetes의 kube-apiserver 매니페스트에 의도적으로 잘못된 인자를 추가하거나 YAML 문법 오류를 넣어 장애 상황을 만들어 보고 백업본으로 복구하는 실습

2 최초 상태

controlplane:~$ k get pod -A | grep api
kube-system          kube-apiserver-controlplane               1/1     Running   1 (27m ago)   31d

3 실험1

3.1 수정1

# 수정 전 원본 백업!
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.ori

kube-apiserver를 Crash 시키기 위해 매니페스트 파일에 잘못된 인수를 추가한다.

controlplane:~$ vim /etc/kubernetes/manifests/kube-apiserver.yaml

...
  - command:
    - kube-apiserver
    - --this-is-very-wrong ## 추가
    - --advertise-address=172.30.1.2

3.2 확인1

controlplane:~$ k get pod -A
The connection to the server 172.30.1.2:6443 was refused - did you specify the right host or port?

controlplane:~$ crictl ps -a | grep apiserver
c2fd5e7298b80       ee794efa53d85       1 second ago        Exited              kube-apiserver            1                   395619c032af5       kube-apiserver-controlplane               kube-system

controlplane:~$ crictl logs c2fd5e7298b80
Error: unknown flag: --this-is-very-wrong

3.3 복구1

controlplane:~$ cp ~/kube-apiserver.yaml.ori /etc/kubernetes/manifests/kube-apiserver.yaml
controlplane:~$ k get pod -A | grep api
kube-system          kube-apiserver-controlplane               1/1     Running   0               31d

4 실험2

4.1 수정2

controlplane:~$ vim /etc/kubernetes/manifests/kube-apiserver.yaml

...
    #- --etcd-servers=https://127.0.0.1:2379
    - --etcd-servers=this-is-very-wrong

4.2 확인2

controlplane:~$ k get pod -A
The connection to the server 172.30.1.2:6443 was refused - did you specify the right host or port?

controlplane:~$ crictl ps -a | grep api
3be4df346fbff       ee794efa53d85       28 seconds ago      Exited              kube-apiserver            0                   37291352696b2       kube-apiserver-controlplane               kube-system

controlplane:~$ crictl logs 3be4df346fbff
...
W0919 11:05:34.885990       1 logging.go:55] [core] [Channel #1 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "this-is-very-wrong", ServerName: "this-is-very-wrong", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: address this-is-very-wrong: missing port in address"
W0919 11:05:35.683576       1 logging.go:55] [core] [Channel #3 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "this-is-very-wrong", ServerName: "this-is-very-wrong", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: address this-is-very-wrong: missing port in address"
F0919 11:05:39.561206       1 instance.go:226] Error creating leases: error creating storage factory: context deadline exceeded

4.3 복구2

controlplane:~$ cp ~/kube-apiserver.yaml.ori /etc/kubernetes/manifests/kube-apiserver.yaml
controlplane:~$ k get pod -A | grep api
kube-system          kube-apiserver-controlplane               1/1     Running   0              31d

5 실험3

5.1 수정3

controlplane:~$ vim /etc/kubernetes/manifests/kube-apiserver.yaml

#apiVersion: v1
apiVersionTHIS IS VERY ::::: WRONG v1

5.2 확인3

controlplane:~$ k get pod -A
The connection to the server 172.30.1.2:6443 was refused - did you specify the right host or port?

controlplane:~$ crictl ps -a | grep api
controlplane:~$

controlplane:~$ tail -f /var/log/syslog | grep apiserver
...
2025-09-19T11:09:32.736629+00:00 controlplane kubelet[1554]: E0919 11:09:32.736321    1554 file.go:187] "Could not process manifest file" err="/etc/kubernetes/manifests/kube-apiserver.yaml: couldn't parse as pod(Object 'apiVersion' is missing in '{\"apiVersionTHIS IS VERY ::::\":\"WRONG v1\",\"kind\":\"Pod\",..., please check config file" path="/etc/kubernetes/manifests/kube-apiserver.yaml"

5.3 복구3

controlplane:~$ cp ~/kube-apiserver.yaml.ori /etc/kubernetes/manifests/kube-apiserver.yaml
controlplane:~$ k get pod -A | grep api
kube-system          kube-apiserver-controlplane               1/1     Running   0              31d

6 같이 보기

• Killer Shell CKA