Re: Exploding memory!!

Answering all my own items today :)

Turns out that the problem was the mysqld daemon (which, for those who are trying to keep score at home, is on the same server as the data node). It was trying to "take over" the server and use all the memory. Of course this caused badness. I made a few changes to the my.cnf of the mysqld daemon and restarted it. Percentage of memory consumed dropped from 30% to around a steady 5% for the mysqld daemon. there is something like 300MB free RAM of the 640MB and I am a much happier camper.

For those who run into this problem..I added some queue and buffer parameters to the my.cnf file basically.

thanks,

Keith
----- Original Message -----
From: "B. Keith Murphy" <kmurphy@icontact.com> To: "cluster" <cluster@lists.mysql.com> Sent: Thursday, September 20, 2007 4:49:21 PM (GMT-0500) America/New_York Subject: Exploding memory!!

So this is fun. I have a developement cluster set up. Two servers, each server is running a sql node and a data node. The management node runs on another computer.

Around noon I restarted the cluster. I am tracking cpu/mem/packets and such through a tool called ganglia. It gives you pretty little html graphs so you can see "trends".

I have 640MB of RAM for each server. Not much I know, but we aren't really doing much on this. The my.cnf files are really straight-forward as is the config.ini file (which I will show in a sec).

Do you need help?

In addition to our free technical library, Pantek provides professional IT services to companies who utilize Open Source software: Learn more about Pantek's Technical Support offerings Place an order for 24/7/365 Technical Support here Learn more about Pantek, who we are, and what we do Chat with a Live Operator Related lists.mysql.com Links announce benchmarks bugs certification cluster commits dotnet gui-tools internals java maxdb myodbc mysql mysqldoc packagers perl plusplus replication win32 Request more information about Pantek

I have watched the total memory usage rise from around 200 MB to around 400MB in the last 4.5 hours using ganglia and top. Assuming this trend doesn't change it will reach the end of physical memory in about 4 more hours, start swapping and crash. So, this really wouldn't be a good thing. I am not finding anything significant about memory leaks or such in mysql lately so I am hoping against hope that I have something configured screwey. Which is entirely possible as I am new to this clustering thing :)

Oh, another tidbit:

2007-09-20 16:07:09 [MgmSrvr] INFO -- Node 3: Data usage is 15%(988 32K pages of total 6400) 
2007-09-20 16:07:09 [MgmSrvr] INFO -- Node 3: Index usage is 3%(260 8K pages of total 7200) 
2007-09-20 16:07:09 [MgmSrvr] INFO -- Node 3: Resource 0 min: 0 max: 639 curr: 0 

My data and index memory usage according to ndb_mgm is 15% for data and 3% for indexes. This has stayed consistent since startup.

My config.ini is following. You will see that I specify two replicas. In this situation with 2 servers and two data nodes that means I have one copy of the database on each server right? And the Datamemory and Indexmemory lines specify that I have allocated a total of 256MB of RAM per data node? Thanks for the help everyone!!

My config.ini is:

# Options affecting ndbd processes on all data nodes:

[NDBD DEFAULT]
NoOfReplicas=2 # Number of replicas
DataMemory=200M # How much memory to allocate for data storage IndexMemory=56M # How much memory to allocate for index storage

# TCP/IP options:

[TCP DEFAULT]
portnumber=2202 # This the default; however, you can use any
# port that is free for all the hosts in the cluster
# Note: It is recommended that you do not specify the
# portnumber at all and allow the default value to be
# used instead

Do you need more help?

In addition to our free technical library, Pantek provides professional IT services to companies who utilize Open Source software: Learn more about Pantek's Technical Support offerings Place an order for 24/7/365 Technical Support here Learn more about Pantek, who we are, and what we do Chat with a Live Operator Related lists.mysql.com Links announce benchmarks bugs certification cluster commits dotnet gui-tools internals java maxdb myodbc mysql mysqldoc packagers perl plusplus replication win32 Request more information about Pantek

# Management process options:

[NDB_MGMD]
hostname=xxx.xxx.xxx.117 # Hostname or IP address of MGM node datadir=/var/lib/mysql-cluster # Directory for MGM node log files

# Options for data node "A":

[NDBD]
# (one [NDBD] section per data node)

hostname=xxx.xxx.xxx.135 # Hostname or IP address datadir=/usr/local/mysql/data # Directory for this data node's data files

# Options for data node "B":

[NDBD]
hostname=xxx.xxx.xxx.136 # Hostname or IP address datadir=/usr/local/mysql/data # Directory for this data node's data files

# SQL node "A" options:

[MYSQLD]
hostname=xxx.xxx.xxx.135 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)

# SQL node "B" options:

[MYSQLD]
hostname=xxx.xxx.xxx.136 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)

-- 
B. Keith Murphy 
Database Administrator 
iContact 
2635 Meridian Parkway, 2nd Floor 
Durham, North Carolina 27713 
blog: 
http://blog.paragon-cs.com 
(o) 919-433-0786 
(c) 850-637-3877 


-- 
B. Keith Murphy 
Database Administrator 
iContact 
2635 Meridian Parkway, 2nd Floor 
Durham, North Carolina 27713 
blog: 
http://blog.paragon-cs.com 
(o) 919-433-0786 
(c) 850-637-3877