Opened 13 years ago
Last modified 13 years ago
#50 closed task (FIXED)
mongodb not scaling down: workaround and documentation
Reported by: | Elrond | Owned by: | Elrond |
---|---|---|---|
Priority: | minor | Milestone: | 0.0.5 |
Component: | documentation | Keywords: | |
Cc: | Parent Tickets: |
Description
mongodb needs 0.5 GB for a fresh install. Half of this goes to kombu. kombu could use redis or some other transport (see `#322 </issues/322>`_), so fixing `#322 </issues/322>`_ will help this issue too. mongodb database files contain a lot of NUL bytes. So one can easily use sparse files to save space on disk: :: # service mongodb stop # cd /var/lib/ # cp -a --sparse=always mongodb mongodb.new # mv mongodb mongodb.old # mv mongodb.new mongodb # service mongodb start # # TEST mongodb # rm -rf mongodb.old Maybe later versions of mongodb do this already internally. The above was needed on mongodb from debian/stable. Documenting the "make sparse" possibility will also help this issue.
Change History (8)
comment:2 by , 13 years ago
Elrond says: 1. Those files on disk contain a a lot NUL chunks. If one converts them to sparse files, things are already much better (see the bug, it's in there). 2. Putting "noprealloc = true" in mongodb.conf: mongodb usually allocates a fresh, empty file for mmapping. So it has fresh space, just in case. That takes up a lot of space on disk. 3. "nssize = 1" (MB). This is the namespcae file size. I don't know actually, what is the worse effect of this. It might break everything. ;o) 4. Running GMG/celery in always-eager-mode (see bug; This is the default for lazyserver.sh currently) also saves half the mongo size, as kombu (messaging whatnot behind celery) isn't used. Worse effect: Any processing is synchronous. So you have to wait for the server to process your uploaded media, while sitting at your desk and watching your browser. 5. - There seems to be a "smallfiles" option. I have not tested it. It should make the first mmaped file much smaller. Only makes sense, if you intend to have a small amount of data in your db anyway. - This isn't available in the config, but only as a cmdline param. :-\| - Most mongo options only take effect before even starting GMG, as they must be set before the "mediagoblin" database is created on the db-server. 7. - Most options only affect on disk space. As mmaping even a 1 TB file (on a 64bit machine) doesn't load that file into RAM. The OS only loads stuff into RAM actually used. And the OS can "swap" (write to the file on disk) the stuff anyway, if needed. The real problem is a large db, because it will basicly be either fully in RAM, or any operation on it will swap like mad. - celery\_always\_eager will actually save RAM, as it means a complete database not being created/used! Caleb swears to: - At least put this in some nice form on the wiki. :-) - Including some links to the docs from the bug, etc. - docs: See "mongodb in general" paragraph at `http://bugs.foocorp.net/issues/323#note-1 <http://bugs.foocorp.net/issues/323#note-1>`_ There are some links to official mongo docs. Those should be put in the wiki (or the deployment docs). - put a link to the wiki page on the bug. - [BONUS] - Write a cool "Using GMG on limited hardware (read freedombox)" chapter for the non existent deployment docs. Elrond conludes: So, if someone comes in and says "I want to run GMG on my freedombox, what should I take care of", we can point them to the wiki and say "What we know, is there. It's not ready for the official docs, but all the info is there." :-))
comment:3 by , 13 years ago
Thanks Caleb for taking care of this!! Also for the "See also" section of the upcoming wiki page: - `https://jira.mongodb.org/browse/SERVER-1378 <https://jira.mongodb.org/browse/SERVER-1378>`_ (found by and also comments by cwebber) So that people know where to go and help. ;) And: \* 8. UNTESTED: Backing up and restoring the complete db might safe some space/RAM, as it might remove some fragmentation. \* Maybe someone more familiar with mongodb could look over this finally.
comment:4 by , 13 years ago
Status: | New → Closed |
---|
I'm closing this ticket because these efforts are now documented in the wiki! `http://wiki.mediagoblin.org/Scaling\_Down <http://wiki.mediagoblin.org/Scaling_Down>`_
comment:5 by , 13 years ago
Owner: | set to |
---|---|
Status: | Closed → Feedback |
hm, maybe just let Elrond decide whether to close the ticket.
comment:7 by , 13 years ago
Milestone: | → 0.0.5 |
---|---|
Status: | Feedback → Closed |
I'm marking this as closed. We can continue to update these docs as we find more info. Thanks Caleb / Elrond for your efforts on this ticket!
comment:8 by , 13 years ago
The original url for this bug was http://bugs.foocorp.net/issues/323 .
Relations:
#22: related, #49: related
Note:
See TracTickets
for help on using tickets.