API node auto restart, mysql-max segfault at ...

From: Toan. Dang Anh <toanda(at)vinagame.com.vn>
Date: Sat Sep 08 2007 - 09:23:04 EDT

Dear All,
I have build MySQL Cluster with 1 MGM, 1 API, and 2 SQL node with version 5.0.41 I have serius problem with API node.
The API node auto restart very normally with 1 hour or 30 minutes, or event less. On mysql log,
Number of processes running now: 0

070908 17:40:02 [Warning] 'user' entry 'root@PrivateData1' ignored in --skip-name-resolve mode.
070908 17:40:02 [Warning] 'user' entry '@PrivateData1' ignored in --skip-name-resolve mode.
070908 17:40:02 [Note] /usr/sbin/mysqld-max: ready for connections.

Version: '5.0.37-max' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Experimental (GPL)

Number of processes running now: 0

070908 18:56:18  mysqld restarted
070908 18:56:22 [Warning] 'user' entry 'root@PrivateData1' ignored in --skip-name-resolve mode.
070908 18:56:22 [Warning] 'user' entry '@PrivateData1' ignored in --skip-name-resolve mode.
070908 18:56:22 [Note] /usr/sbin/mysqld-max: ready for connections.

Version: '5.0.37-max' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Experimental (GPL)

I run command dmesg, show error:
Free pages: 12204kB (0kB HighMem)
Active:498002 inactive:494921 dirty:0 writeback:0 unstable:0 free:3051 slab:2563 mapped:991926 pagetables:4329 DMA free:11564kB min:4kB low:8kB high:12kB active:0kB inactive:0kB present:16384kB pages_scanned:1009 all_unreclaimable? yes protections[]: 0 0 0
Normal free:640kB min:2220kB low:4440kB high:6660kB active:1992008kB inactive:1979684kB present:4964348kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 4*64kB 0*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11564kB Normal: 156*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 640kB HighMem: empty
Swap cache: add 1105093, delete 1105093, find 11993/17667, race 0+0 Free swap: 0kB
1245183 pages of RAM
235128 reserved pages
65615 pages shared
0 pages swap cached
mysqld-max: page allocation failure. order:0, mode:0x1d2

Call Trace:<ffffffff8015dd14>{__alloc_pages+709} <ffffffff80160802>{do_page_cache_readahead+234}

<ffffffff8015aafc>{filemap_nopage+338} <ffffffff801688a1>{do_no_page+683}
<ffffffff8013fd08>{del_singleshot_timer_sync+21} <ffffffff80168c57>{handle_mm_fault+373}
<ffffffff80188786>{do_select+978} <ffffffff80110d91>{error_exit+0}
<ffffffff80123b4e>{do_page_fault+518} <ffffffff80188c25>{sys_select+1147}
<ffffffff8030b043>{thread_return+88} <ffffffff80110d91>{error_exit+0}

Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
Normal per-cpu:

cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16

HighMem per-cpu: empty

Free pages: 11628kB (0kB HighMem)
Active:496721 inactive:496408 dirty:0 writeback:0 unstable:0 free:2907 slab:2561 mapped:991931 pagetables:4329 DMA free:11564kB min:4kB low:8kB high:12kB active:0kB inactive:0kB present:16384kB pages_scanned:1009 all_unreclaimable? yes protections[]: 0 0 0
Normal free:64kB min:2220kB low:4440kB high:6660kB active:1986756kB inactive:1985632kB present:4964348kB pages_scanned:7359 all_unreclaimable? no protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 4*64kB 0*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11564kB Normal: 16*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64kB HighMem: empty
Swap cache: add 1105093, delete 1105093, find 11993/17667, race 0+0 Free swap: 0kB
1245183 pages of RAM
235128 reserved pages
65680 pages shared
0 pages swap cached

mysqld-max[19427]: segfault at 0000000040e66fe8 rip 0000003037568900 rsp 000000 040e67018 error 6
mysqld-max[19607]: segfault at 0000000040b5afe8 rip 0000003037568900 rsp 000000 040b5b018 error 6
mysqld-max[21598]: segfault at 0000000040d62fe8 rip 0000003037568900 rsp 000000 040d63018 error 6
mysqld-max[26583]: segfault at 0000000041131fe8 rip 0000003037568900 rsp 000000 041132018 error 6
mysqld-max[19417]: segfault at 0000000041131fe8 rip 0000003037568900 rsp 000000 041132018 error 6
mysqld-max[19530]: segfault at 0000000040b9bfe8 rip 0000003037568900 rsp 000000 040b9c018 error 6
mysqld-max[19586]: segfault at 0000000040b19fe8 rip 0000003037568900 rsp 000000 040b1a018 error 6
mysqld-max[9017]: segfault at 0000000040b9bfe8 rip 0000003037568900 rsp 0000000 40b9c018 error 6
mysqld-max[9318]: segfault at 0000000040fecfe8 rip 0000003037568900 rsp 0000000 40fed018 error 6

API my.cnf config:

skip-innodb
skip-name-resolve
skip-locking

#improve performance
ndb_force_send = 0
ndb_use_exact_count = 0

key_buffer_size = 512M
max_allowed_packet=128M
read_buffer_size=6M
read_rnd_buffer_size=6M
connect_timeout=25
interactive_timeout=30
wait_timeout=25
init-connect='SET NAMES utf8'

max_connections = 1024
max_user_connections = 256
max_connect_errors=128

thread_concurrency=4
ndbcluster

Server: 2 CPU, 4GB RAM. I run top command: top - 20:05:43 up 60 days, 5:10, 2 users, load average: 0.10, 0.14, 0.16

Tasks:  46 total,   1 running,  45 sleeping,   0 stopped,   0 zombie
Cpu0  :  2.7% us,  3.0% sy,  0.0% ni, 92.4% id,  0.0% wa,  0.3% hi,  1.7% si
Cpu1  :  0.0% us,  0.0% sy,  0.0% ni, 100.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   4040224k total,   872196k used,  3168028k free,   271876k buffers
Swap:  4192956k total,     2296k used,  4190660k free,   298404k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 9362 mysql     15   0  658m 194m 3980 S    6  4.9  13:08.19 mysqld-max

Please help me solve this problem.
Thank so much.
ToanDA Received on Sat Sep 8 09:29:06 2007