Re: cluster takes enormous room to store data

Hello,

On 7/7/07, Brian Moon <brianm@dealnews.com> wrote:
> > I was hoping this would allow to import a DB of around 1.3Gigs of Data from
> > a "classic" mysql server and am very surprised by the results.
>
> You likely have a lot of wasted space using int where smallint will do
> and using varchar(255) when varchar(20) will do. Cluster (5.0) does not
> store variable length data all that well. Also, TEXT and BLOB fields
> can use a lot of space. More than you think due to the way they are
> stored in NDB. We had to analyze our data very carefully. We saved 2GB
> by just converting varchar and numeric fields to appropriate sizes and
> dropping one TEXT field on a 200k row table.

I would review your schema and use ndb_size.pl to check for space savings. Like Brian said, its very easy to consume a lot of space in 5.0. 5.1 of course has variable sized columns so you save a little more - added with disk based data tables to save even more memory.

> Most likely you are running 32-bit versions of MySQL. 32-bit versions
> of MySQL can not allocate more than about 2.5GB of memory. Now, I am no
> NDB expert, but I know this has always been the case with other mysql
> products. I have only run NDB on 64-bit machines so I can not say for
> sure that this is an issue.

All 32bit applications have this limitation, its a memory addressing problem, 32 bit numbers only go so high - which limits each application to around 2 - 3GB of memory. If you are running a x86_64 OS then simply use a 64bit version of MySQL. If you have 32bit processors you can run multiple ndbd on each host to consume your available memory.

Adam

-- 
MySQL Cluster Mailing List
For list archives: 
http://lists.mysql.com/cluster
To unsubscribe:    
http://lists.mysql.com/cluster?unsub=lists@pantek.com