Killer Shell CKA - Apiserver Crash

1 개요

Killer Shell CKA - Apiserver Crash

https://killercoda.com/killer-shell-cka/scenario/apiserver-crash
Kubernetes의 kube-apiserver 매니페스트에 의도적으로 잘못된 인자를 추가하거나 YAML 문법 오류를 넣어 장애 상황을 만들어 보고 백업본으로 복구하는 실습

2 최초 상태

controlplane:~$ k get pod -A | grep api
kube-system          kube-apiserver-controlplane               1/1     Running   1 (27m ago)   31d

3 수정1

# 수정 전 원본 백업!
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.ori

kube-apiserver를 Crash 시키기 위해 매니페스트 파일에 잘못된 인수를 추가한다.

vim /etc/kubernetes/manifests/kube-apiserver.yaml

# - --etcd-servers=https://127.0.0.1:2379
- --etcd-servers=this-is-very-wrong

4 확인 및 복구 1

controlplane:~$ k get pod -A
The connection to the server 172.30.1.2:6443 was refused - did you specify the right host or port?

controlplane:~$ crictl ps -a | grep apiserver
d4f2aead6efd1       ee794efa53d85       About a minute ago   Exited              kube-apiserver            5                   52e3a48665b8e       kube-apiserver-controlplane               kube-system

controlplane:~$ crictl logs d4f2aead6efd1    
...
W0918 10:29:07.635469       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "this-is-very-wrong", ServerName: "this-is-very-wrong", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: address this-is-very-wrong: missing port in address"
W0918 10:29:08.626327       1 logging.go:55] [core] [Channel #2 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "this-is-very-wrong", ServerName: "this-is-very-wrong", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: address this-is-very-wrong: missing port in address"

5 복구

controlplane:~$ cp ~/kube-apiserver.yaml.ori /etc/kubernetes/manifests/kube-apiserver.yaml
controlplane:~$ kubectl get pod -A | grep api
NAMESPACE            NAME                                      READY   STATUS    RESTARTS      AGE
kube-system          calico-kube-controllers-fdf5f5495-8jbqm   1/1     Running   1 (30m ago)   30d
kube-system          canal-rtfc5                               2/2     Running   2 (30m ago)   30d
kube-system          coredns-6ff97d97f9-2rxsf                  1/1     Running   1 (30m ago)   30d
kube-system          coredns-6ff97d97f9-85m5c                  1/1     Running   1 (30m ago)   30d
kube-system          etcd-controlplane                         1/1     Running   1 (30m ago)   30d
kube-system          kube-apiserver-controlplane               1/1     Running   1 (30m ago)   30d
kube-system          kube-controller-manager-controlplane      1/1     Running   1 (30m ago)   30d
kube-system          kube-proxy-7kdz8                          1/1     Running   1 (30m ago)   30d
kube-system          kube-scheduler-controlplane               1/1     Running   1 (30m ago)   30d
local-path-storage   local-path-provisioner-5c94487ccb-gmwjg   1/1     Running   1 (30m ago)   30d

6 같이 보기

Killer Shell CKA