mongodb not scaling down: workaround and documentation

[Elrond]

mongodb needs 0.5 GB for a fresh install. Half of this goes to
kombu.

kombu could use redis or some other transport (see
`#322 </issues/322>`_), so fixing `#322 </issues/322>`_ will help
this issue too.

mongodb database files contain a lot of NUL bytes. So one can
easily use sparse files to save space on disk:

::

     # service mongodb stop
     # cd /var/lib/
     # cp -a --sparse=always mongodb mongodb.new
     # mv mongodb mongodb.old
     # mv mongodb.new mongodb
     # service mongodb start
     # # TEST mongodb
     # rm -rf mongodb.old

Maybe later versions of mongodb do this already internally. The
above was needed on mongodb from debian/stable.

Documenting the "make sparse" possibility will also help this
issue.

[Elrond]

Ahh, caring for old bugs...

CELERY\_ALWAYS\_EAGER¶
======================

Using CELERY\_ALWAYS\_EAGER (which is already documented) shrinks
the mongodb size (on a fresh install or after removing
kombu\_default) by 50%. Which is already great.

The docs mentions that CELERY\_ALWAYS\_EAGER is not for production.
But on small systems (freedombox) it might still be acceptable as a
production setting to save space?

That said: This bug can be set to "40% done". :-)

mongodb in general¶
===================

cwebber found
`http://groups.google.com/group/mongodb-user/browse\_thread/thread/6c32836a742270aa <http://groups.google.com/group/mongodb-user/browse_thread/thread/6c32836a742270aa>`_
which lists some options to shrink mongo's space usage. I will test
them. If they help, the mediagoblin docs should point to this
posting (or the parts of the mongo docs).
Relevant mongo docs:
`http://www.mongodb.org/display/DOCS/Command+Line+Parameters <http://www.mongodb.org/display/DOCS/Command+Line+Parameters>`_
and
`http://www.mongodb.org/display/DOCS/File+Based+Configuration <http://www.mongodb.org/display/DOCS/File+Based+Configuration>`_

If the "sparse files" trick (as shown above) is not documented or
integrated into mongo by now, it should at least be documented
somewehre.

[Caleb Davis]

Elrond says:
1. Those files on disk contain a a lot NUL chunks. If one converts
them to sparse files, things are already much better (see the bug,
it's in there).


2. Putting "noprealloc = true" in mongodb.conf: mongodb usually
   allocates a fresh, empty file for mmapping. So it has fresh space,
   just in case. That takes up a lot of space on disk.

3. "nssize = 1" (MB). This is the namespcae file size. I don't know
   actually, what is the worse effect of this. It might break
   everything. ;o)

4. Running GMG/celery in always-eager-mode (see bug; This is the
   default for lazyserver.sh currently) also saves half the mongo
   size, as kombu (messaging whatnot behind celery) isn't used. Worse
   effect: Any processing is synchronous. So you have to wait for the
   server to process your uploaded media, while sitting at your desk
   and watching your browser.


5.


-  There seems to be a "smallfiles" option. I have not tested it.
   It should make the first mmaped file much smaller. Only makes
   sense, if you intend to have a small amount of data in your db
   anyway.
-  This isn't available in the config, but only as a cmdline param.
   :-\|

-  Most mongo options only take effect before even starting GMG, as
   they must be set before the "mediagoblin" database is created on
   the db-server.


7.


-  Most options only affect on disk space. As mmaping even a 1 TB
   file (on a 64bit machine) doesn't load that file into RAM. The OS
   only loads stuff into RAM actually used. And the OS can "swap"
   (write to the file on disk) the stuff anyway, if needed. The real
   problem is a large db, because it will basicly be either fully in
   RAM, or any operation on it will swap like mad.
-  celery\_always\_eager will actually save RAM, as it means a
   complete database not being created/used! Caleb swears to:
-  At least put this in some nice form on the wiki. :-)
-  Including some links to the docs from the bug, etc.
-  docs: See "mongodb in general" paragraph at
   `http://bugs.foocorp.net/issues/323#note-1 <http://bugs.foocorp.net/issues/323#note-1>`_
   There are some links to official mongo docs. Those should be put in
   the wiki (or the deployment docs).
-  put a link to the wiki page on the bug.
-  [BONUS] - Write a cool "Using GMG on limited hardware (read
   freedombox)" chapter for the non existent deployment docs.

Elrond conludes:
So, if someone comes in and says "I want to run GMG on my
freedombox, what should I take care of", we can point them to the
wiki and say "What we know, is there. It's not ready for the
official docs, but all the info is there." :-))

[Elrond]

Thanks Caleb for taking care of this!!

Also for the "See also" section of the upcoming wiki page:


-  `https://jira.mongodb.org/browse/SERVER-1378 <https://jira.mongodb.org/browse/SERVER-1378>`_
   (found by and also comments by cwebber)

So that people know where to go and help. ;)

And:

\* 8. UNTESTED: Backing up and restoring the complete db might safe
some space/RAM, as it might remove some fragmentation. \* Maybe
someone more familiar with mongodb could look over this finally.

[Caleb Davis]

I'm closing this ticket because these efforts are now documented in
the wiki!

`http://wiki.mediagoblin.org/Scaling\_Down <http://wiki.mediagoblin.org/Scaling_Down>`_

[Caleb Davis]

hm, maybe just let Elrond decide whether to close the ticket.

[Elrond]

Great! Thanks!

[Christopher Allan Webber]

I'm marking this as closed. We can continue to update these docs as
we find more info.

Thanks Caleb / Elrond for your efforts on this ticket!

[Will Kahn-Greene]

The original url for this bug was ​http://bugs.foocorp.net/issues/323 .
Relations:
#22: related, #49: related