mongo异常退出问题排查
mongo跑了一段时间后莫名其妙的异常退出,在mongo自己的日志里并没有留下任何遗言。
这种情况下,凭借个人的经验,猜测又是系统内存不足,触发OOM-killer,杀死了mongod。
查看 /var/log/messages文件

果然又是OOM的问题
(93条消息) Mongo崩溃crash, 报out of memory的问题分析与解决方案_飞出四季做的茧的博客-CSDN博客_mongodb 崩溃
默认的是,系统内存的一半再减1,我现在的机器是16G的,也就是最高会申请到7G。
太多了,触发oom
设置为2G,观察
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
| May 19 18:22:00 pfsmongo1 kernel: NetworkManager invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 May 19 18:22:00 pfsmongo1 kernel: NetworkManager cpuset=/ mems_allowed=0 May 19 18:22:00 pfsmongo1 kernel: CPU: 4 PID: 671 Comm: NetworkManager Kdump: loaded Tainted: G OE ------------ 3.10.0-1160.36.2.el7.x86_64 #1 May 19 18:22:00 pfsmongo1 kernel: Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014 May 19 18:22:00 pfsmongo1 kernel: Call Trace: May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94783559>] dump_stack+0x19/0x1b May 19 18:22:00 pfsmongo1 kernel: [<ffffffff9477e5f8>] dump_header+0x90/0x229 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94106972>] ? ktime_get_ts64+0x52/0xf0 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff9415dfff>] ? delayacct_end+0x8f/0xb0 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c252d>] oom_kill_process+0x2cd/0x490 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c1f1d>] ? oom_unkillable_task+0xcd/0x120 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c2c1a>] out_of_memory+0x31a/0x500 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c9834>] __alloc_pages_nodemask+0xad4/0xbe0 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94219388>] alloc_pages_current+0x98/0x110 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941bdfe7>] __page_cache_alloc+0x97/0xb0 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941c0f80>] filemap_fault+0x270/0x420 May 19 18:22:00 pfsmongo1 kernel: [<ffffffffc03f1756>] ext4_filemap_fault+0x36/0x50 [ext4] May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941ee78a>] __do_fault.isra.61+0x8a/0x100 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941eed3c>] do_read_fault.isra.63+0x4c/0x1b0 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff941f6580>] handle_mm_fault+0xa20/0xfb0 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94790653>] __do_page_fault+0x213/0x500 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff94790a26>] trace_do_page_fault+0x56/0x150 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff9478ffa2>] do_async_page_fault+0x22/0xf0 May 19 18:22:00 pfsmongo1 kernel: [<ffffffff9478c7a8>] async_page_fault+0x28/0x30 May 19 18:22:00 pfsmongo1 kernel: Mem-Info:
|
NetworkManager invoked oom-killer