"echo 0 /proc/sys/kernel/hung task timeout secs" disables this message.

"echo 0 /proc/sys/kernel/hung task timeout secs" disables this message.

예시 1

Dec  9 01:57:51 jmnote varnishd[22585]: child (22588) Started
Dec  9 01:57:51 jmnote varnishd[22585]: Child (22588) said Child starts
Dec  9 01:57:51 jmnote varnishd[22585]: Child (22588) said SMF.s0 mmap'ed 1610612736 bytes of 1610612736
Dec  9 06:08:47 jmnote varnishd[22585]: Child (22588) not responding to CLI, killing it.
Dec  9 06:08:57 jmnote varnishd[22585]: Child (22588) not responding to CLI, killing it.
Dec  9 06:09:07 jmnote varnishd[22585]: Child (22588) not responding to CLI, killing it.
Dec  9 06:09:17 jmnote varnishd[22585]: Child (22588) not responding to CLI, killing it.
Dec  9 06:10:47 jmnote last message repeated 9 times
Dec  9 15:11:58 jmnote kernel: INFO: task varnishd:22588 blocked for more than 120 seconds.
Dec  9 06:12:55 jmnote varnishd[22585]: Child (22588) not responding to CLI, killing it.
Dec  9 15:13:00 jmnote kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 15:16:46 jmnote kernel: varnishd      D ffffffff8031db60     0 22588  22585         22590       (NOTLB)
Dec  9 15:16:46 jmnote kernel:  ffff810044dabc28 0000000000000082 ffff81007fd93830 ffff81005667f080
Dec  9 15:16:46 jmnote kernel:  0000c9e94060c654 0000000000000009 ffff81007fd93830 ffff81007fe2e0c0
Dec  9 15:16:46 jmnote kernel:  0000c9e94061ad9e 00000000000004aa ffff81007fd93a18 0000000500000002
Dec  9 15:16:46 jmnote kernel: Call Trace:
Dec  9 15:16:47 jmnote kernel:  [<ffffffff80063171>] wait_for_completion+0x79/0xa2
Dec  9 15:16:47 jmnote kernel:  [<ffffffff8008f455>] default_wake_function+0x0/0xe
Dec  9 15:16:47 jmnote kernel:  [<ffffffff800ebee7>] do_coredump+0x2f3/0x893

예시 2

Dec  9 19:16:27 jmnote smartd[4257]: smartd has fork()ed into background mode. New PID=4257. 
Dec  9 19:16:27 jmnote avahi-daemon[4222]: Server startup complete. Host name is jmnote.local. Local service cookie is 2192076488.
Dec  9 21:31:55 jmnote avahi-daemon[4222]: Invalid response packet from host 192.168.100.160.
Dec  9 21:43:25 jmnote kernel: INFO: task httpd:19485 blocked for more than 120 seconds.
Dec  9 21:43:25 jmnote kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 21:43:25 jmnote kernel: httpd         D ffff810061617da8     0 19485   3609         19486 19484 (NOTLB)
Dec  9 21:43:25 jmnote kernel:  ffff810063b0dce8 0000000000000082 ffff81007dc9d288 ffffffff880973a1
Dec  9 21:43:25 jmnote kernel:  ffff81007edc8000 0000000000000001 ffff810054fe17f0 ffff8100466737f0
Dec  9 21:43:25 jmnote kernel:  000007f10a4bf20a 00000000000f058d ffff810054fe19d8 000000048805c460
Dec  9 21:43:25 jmnote kernel: Call Trace:
Dec  9 21:43:25 jmnote kernel:  [<ffffffff880973a1>] :virtio_blk:do_virtblk_request+0x26d/0x2b0
Dec  9 21:43:25 jmnote kernel:  [<ffffffff8000985a>] __d_lookup+0xb0/0xff
Dec  9 21:43:25 jmnote kernel:  [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
Dec  9 21:43:25 jmnote kernel:  [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
Dec  9 21:43:25 jmnote kernel:  [<ffffffff8000d083>] do_lookup+0xf5/0x24b

조치

메모리 부족이라 판단하여 서버 증설

• MySQL + 아파치 + Varnish를 서버 1개에서 돌리던 중
• 최번시 메모리 사용률 85% 이상
• 현재 서버에는 MySQL만 남기고, 아파치 + Varnish는 새 서버에서 구동

같이 보기

• messages 로그
• hung_task_timeout_secs