Killer Shell CKA - Apiserver Crash

1 개요

killercoda CKA - Apiserver Crash

Kubernetes의 kube-apiserver 매니페스트에 의도적으로 잘못된 인자를 추가하거나 YAML 문법 오류를 넣어 장애 상황을 만드는 실습

2 최초 상태

controlplane:~$ k get pod -A
NAMESPACE            NAME                                      READY   STATUS    RESTARTS      AGE
kube-system          calico-kube-controllers-fdf5f5495-8jbqm   1/1     Running   1 (14m ago)   30d
kube-system          canal-rtfc5                               2/2     Running   2 (14m ago)   30d
kube-system          coredns-6ff97d97f9-2rxsf                  1/1     Running   1 (14m ago)   30d
kube-system          coredns-6ff97d97f9-85m5c                  1/1     Running   1 (14m ago)   30d
kube-system          etcd-controlplane                         1/1     Running   1 (14m ago)   30d
kube-system          kube-apiserver-controlplane               1/1     Running   1 (14m ago)   30d
kube-system          kube-controller-manager-controlplane      1/1     Running   1 (14m ago)   30d
kube-system          kube-proxy-7kdz8                          1/1     Running   1 (14m ago)   30d
kube-system          kube-scheduler-controlplane               1/1     Running   1 (14m ago)   30d
local-path-storage   local-path-provisioner-5c94487ccb-gmwjg   1/1     Running   1 (14m ago)   30d

3 변경

kube-apiserver를 Crash 시키기 위해 매니페스트 파일(/etc/kubernetes/manifests/kube-apiserver.yaml)에 잘못된 인수를 추가한다.

# 일단 백업!
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.ori

# 수정
vim /etc/kubernetes/manifests/kube-apiserver.yaml

# - --etcd-servers=https://127.0.0.1:2379
- --etcd-servers=this-is-very-wrong

4 확인

controlplane:~$ k -n kube-system get pod
The connection to the server 172.30.1.2:6443 was refused - did you specify the right host or port?

controlplane:~$ crictl ps -a | grep apiserver
d4f2aead6efd1       ee794efa53d85       About a minute ago   Exited              kube-apiserver            5                   52e3a48665b8e       kube-apiserver-controlplane               kube-system

controlplane:~$ crictl logs d4f2aead6efd1    
...
I0918 10:29:07.634977       1 shared_informer.go:350] "Waiting for caches to sync" controller="*generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator]"
W0918 10:29:07.635469       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "this-is-very-wrong", ServerName: "this-is-very-wrong", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: address this-is-very-wrong: missing port in address"
W0918 10:29:08.626327       1 logging.go:55] [core] [Channel #2 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "this-is-very-wrong", ServerName: "this-is-very-wrong", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: address this-is-very-wrong: missing port in address"
W0918 10:29:08.626403       1 logging.go:55] [core] [Channel #1 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "this-is-very-wrong", ServerName: "this-is-very-wrong", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: address this-is-very-wrong: missing port in address"

5 복구

controlplane:~$ cp ~/kube-apiserver.yaml.ori /etc/kubernetes/manifests/kube-apiserver.yaml
controlplane:~$ kubectl get pod -A
NAMESPACE            NAME                                      READY   STATUS    RESTARTS      AGE
kube-system          calico-kube-controllers-fdf5f5495-8jbqm   1/1     Running   1 (30m ago)   30d
kube-system          canal-rtfc5                               2/2     Running   2 (30m ago)   30d
kube-system          coredns-6ff97d97f9-2rxsf                  1/1     Running   1 (30m ago)   30d
kube-system          coredns-6ff97d97f9-85m5c                  1/1     Running   1 (30m ago)   30d
kube-system          etcd-controlplane                         1/1     Running   1 (30m ago)   30d
kube-system          kube-apiserver-controlplane               1/1     Running   1 (30m ago)   30d
kube-system          kube-controller-manager-controlplane      1/1     Running   1 (30m ago)   30d
kube-system          kube-proxy-7kdz8                          1/1     Running   1 (30m ago)   30d
kube-system          kube-scheduler-controlplane               1/1     Running   1 (30m ago)   30d
local-path-storage   local-path-provisioner-5c94487ccb-gmwjg   1/1     Running   1 (30m ago)   30d

6 같이 보기

killercoda CKA

7 참고

https://killercoda.com/killer-shell-cka/scenario/apiserver-crash