해당 웹페이지를 사용할 수 없음

✔️ CentOS, 아파치, Varnish에서 테스트하였습니다.

해당 웹페이지를 사용할 수 없음

접속 불가

Chrome에서 zetawiki.com 접속하니...

해당 웹페이지를 사용할 수 없음

확인

top 상태 확인

프로세스 상태 코드가 D^[1]인 프로세스 많음
로드 에버리지 90 이상

[root@zetawiki ~]# top
top - 12:47:27 up 41 days, 21:11,  1 user,  load average: 90.02, 107
Tasks: 275 total,   2 running, 273 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us, 13.0%sy,  0.0%ni,  0.3%id, 82.3%wa,  1.5%hi,  2.2%s
Mem:   2058540k total,  2049720k used,     8820k free,      396k buf
Swap:  4030456k total,  2171416k used,  1859040k free,    23796k cac

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  310 root      10  -5     0    0    0 D  1.3  0.0   1:57.41 kswapd0 
 8384 apache    17   0  331m  24m 5780 D  1.3  1.2   0:05.69 httpd   
 8746 apache    16   0  327m  30m 8076 D  1.3  1.5   0:03.73 httpd   
 8857 apache    17   0  329m  25m 5768 D  1.3  1.2   0:03.23 httpd   
 8888 apache    18   0  332m  21m 4928 D  1.3  1.1   0:04.22 httpd   
 8907 apache    17   0  327m  22m 5728 D  1.3  1.1   0:03.12 httpd   
15420 apache    16   0  348m  26m 9412 D  1.3  1.3   4:22.04 httpd   
19038 apache    16   0  347m  25m 9260 D  1.3  1.3   3:47.38 httpd   
 8383 apache    16   0  331m  27m 6468 D  1.0  1.4   0:05.76 httpd   
 8386 apache    17   0  329m  27m 6104 D  1.0  1.4   0:05.23 httpd   
 8571 apache    18   0  332m  22m 4892 D  1.0  1.1   0:05.25 httpd   
 8948 apache    18   0  331m  21m 4624 D  1.0  1.1   0:03.20 httpd   
17378 apache    16   0  349m  30m 8888 D  1.0  1.5   4:13.85 httpd   
 1242 apache    16   0  339m  30m 6868 D  0.7  1.5   0:43.51 httpd   
 1312 apache    16   0  338m  22m 6468 D  0.7  1.1   0:45.83 httpd   
 1325 apache    18   0  338m  26m 6276 D  0.7  1.3   0:42.61 httpd   
 8260 apache    16   0  333m  27m 9068 D  0.7  1.4   0:08.36 httpd   
 8324 apache    17   0  333m  25m 5448 D  0.7  1.3   0:06.04 httpd   
 8385 apache    17   0  324m  23m 6092 D  0.7  1.2   0:05.70 httpd   
 8413 apache    17   0  331m  24m 5252 D  0.7  1.2   0:05.57 httpd   
 8462 apache    16   0  332m  20m 6200 D  0.7  1.0   0:06.33 httpd   
 8573 apache    16   0  328m  28m 6020 D  0.7  1.4   0:04.50 httpd

top - 12:47:30 up 41 days, 21:11,  1 user,  load average: 89.30, 107
Tasks: 275 total,  13 running, 262 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us, 24.5%sy,  0.0%ni,  0.0%id, 69.6%wa,  1.9%hi,  3.5%s
Mem:   2058540k total,  2047856k used,    10684k free,      400k buf
Swap:  4030456k total,  2184016k used,  1846440k free,    23992k cac

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1256 apache    16   0  344m  30m 8556 R  2.8  1.5   0:46.73 httpd   
 1242 apache    16   0  339m  30m 6840 R  1.9  1.5   0:43.57 httpd   
 8292 apache    17   0  331m  24m 6728 R  1.9  1.2   0:07.54 httpd   
 8383 apache    17   0  331m  27m 6436 D  1.9  1.4   0:05.82 httpd   
 8573 apache    17   0  328m  28m 6000 D  1.9  1.4   0:04.56 httpd   
 8828 apache    17   0  328m  25m 6468 D  1.9  1.3   0:03.98 httpd   
 8907 apache    18   0  327m  22m 5728 D  1.9  1.1   0:03.18 httpd   
15420 apache    16   0  348m  26m 9380 D  1.9  1.3   4:22.10 httpd   
17378 apache    16   0  349m  30m 8844 D  1.9  1.5   4:13.91 httpd   
 8763 apache    17   0  329m  25m 5180 D  1.6  1.3   0:05.10 httpd   
16793 apache    16   0  394m  27m 8524 D  1.6  1.4   4:21.01 httpd   
17925 apache    17   0  397m  28m 6580 D  1.6  1.4   3:53.98 httpd   
19038 apache    16   0  347m  25m 9236 D  1.6  1.3   3:47.43 httpd   
  310 root      10  -5     0    0    0 R  1.3  0.0   1:57.45 kswapd0 
 8320 apache    16   0  331m  29m  10m D  1.3  1.5   0:05.94 httpd   
 8439 apache    17   0  331m  27m 6776 D  1.3  1.4   0:06.86 httpd   
 8535 apache    17   0  329m  27m 5844 D  1.3  1.4   0:04.76 httpd   
 8570 apache    18   0  328m  24m 4732 D  1.3  1.2   0:04.22 httpd   
 8653 apache    17   0  328m  23m 4688 D  1.3  1.1   0:04.25 httpd   
 8854 apache    17   0  331m  23m 5564 D  1.3  1.2   0:03.70 httpd   
 8962 apache    18   0  333m  20m 5180 D  1.3  1.0   0:03.40 httpd   
 8331 apache    18   0  226m  12m 3428 R  0.9  0.6   0:06.48 httpd

복구

Varnish, 아파치 재시작

[root@zetawiki ~]# service varnish stop
Stopping Varnish Cache:                                    [  OK  ]

[root@zetawiki ~]# service httpd stop
Stopping httpd:                                            [  OK  ]

[root@zetawiki ~]# service httpd start
Starting httpd:                                            [  OK  ]

[root@zetawiki ~]# service varnish start
Starting Varnish Cache:                                    [  OK  ]

확인 2

웹사이트 접속 잘됨
top 현황 정상

top - 13:02:20 up 41 days, 21:26,  2 users,  load average: 0.17, 6.7
Tasks: 185 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.9%us,  0.1%sy,  0.0%ni, 98.4%id,  0.5%wa,  0.0%hi,  0.1%s
Mem:   2058540k total,   770092k used,  1288448k free,     6920k buf
Swap:  4030456k total,   237876k used,  3792580k free,   232004k cac

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 9399 apache    15   0  341m  81m  38m S  2.3  4.1   0:11.41 httpd   
 9397 apache    16   0  335m  78m  38m S  1.7  3.9   0:10.79 httpd   
 9398 apache    16   0  336m  77m  38m S  1.0  3.9   0:12.58 httpd   
 9395 apache    16   0  392m  78m  38m S  0.7  3.9   0:10.69 httpd   
 9400 apache    15   0  336m  77m  38m S  0.7  3.9   0:11.07 httpd   
 9401 apache    16   0  336m  77m  38m S  0.7  3.8   0:10.00 httpd   
 9437 varnish   18   0 2227m  15m 8376 S  0.7  0.8   0:02.11 varnishd
10577 root      15   0 12756 1168  836 R  0.7  0.1   0:00.05 top     
 9396 apache    15   0  335m  78m  38m S  0.3  3.9   0:14.63 httpd   
    1 root      15   0 10364  488  460 S  0.0  0.0   0:02.29 init    
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:06.26 migratio
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.09 ksoftirq
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:02.19 migratio
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.08 ksoftirq
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.01 watchdog
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:02.19 migratio
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.06 ksoftirq
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.01 watchdog
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:03.15 migratio
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.10 ksoftirq
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.01 watchdog

messages 로그

avahi-daemon: Invalid response packet from host 문서를 참고하십시오.

Nov  9 04:02:35 zetawiki syslogd 1.4.1: restart.
Nov 11 00:11:50 zetawiki avahi-daemon[3803]: Invalid response packet from host 135.79.246.161.
Nov 11 12:17:19 zetawiki avahi-daemon[3803]: Invalid response packet from host 135.79.246.161.
Nov 11 03:17:19 zetawiki varnishd[4098]: Child (4102) not responding to CLI, killing it.
Nov 11 03:17:21 zetawiki varnishd[4098]: Child (4102) not responding to CLI, killing it.
Nov 11 03:17:21 zetawiki varnishd[4098]: Child (4102) died signal=3
Nov 11 03:17:26 zetawiki varnishd[4098]: child (8808) Started
Nov 11 03:17:36 zetawiki varnishd[4098]: Pushing vcls failed: CLI communication error (hdr)
Nov 11 03:17:36 zetawiki varnishd[4098]: Child (8808) said Child starts
Nov 11 03:17:43 zetawiki varnishd[4098]: Child (8808) said SMF.s0 mmap'ed 1073741824 bytes of 1073741824
Nov 11 03:17:43 zetawiki varnishd[4098]: Child (8808) said Child dies
Nov 11 03:18:07 zetawiki varnishd[4098]: Child (8808) died status=1
Nov 11 03:48:15 zetawiki varnishd[4098]: Manager got SIGINT
Nov 11 03:49:12 zetawiki varnishd[9433]: Platform: Linux,2.6.18-348.16.1.el5,x86_64,-sfile,-smalloc,-hcritbit
Nov 11 03:49:12 zetawiki varnishd[9433]: child (9437) Started
Nov 11 03:49:12 zetawiki varnishd[9433]: Child (9437) said Child starts
Nov 11 03:49:12 zetawiki varnishd[9433]: Child (9437) said SMF.s0 mmap'ed 1073741824 bytes of 1073741824

조치: ServerLimit 상향 조정

Server MPM이 Prefork임을 확인

[root@zetawiki ~]# httpd -V | grep "^Server MPM"
Server MPM:     Prefork

httpd.conf의 prefork 변수 상향 조정^[2]

<IfModule prefork.c>
#StartServers       8
#MinSpareServers    5
#MaxSpareServers   20
StartServers       16
MinSpareServers    10
MaxSpareServers   40
#ServerLimit      256
#MaxClients       256
#MaxRequestsPerChild  4000
ServerLimit      2048
MaxClients       2048
MaxRequestsPerChild  32000
</IfModule>

아파치 재시작

[root@zetawiki ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

같이 보기

주석

↑ Uninterruptible sleep ( 깨울 수 없는 잠;; )
↑ 대기 속성(StartServers 등)은 2배, 최대 속성(ServerLimit 등)은 8배로 높임

[1] Uninterruptible sleep ( 깨울 수 없는 잠;; )

[2] 대기 속성(StartServers 등)은 2배, 최대 속성(ServerLimit 등)은 8배로 높임

[1]

[2]