Smarter extlib installs

[Christopher Allan Webber]

Unfortunately, we have a last minute yak to shave for 0.8.0: if we want to fix #918, we should do it right, and that means no more committing (often minified or otherwise compiled) javascript libraries into our git tree.

But how to do that? System packagers can take advantage of other system packages of javascript and the like (though admittedly we might not make this easy at present). In the meanwhile, if we want to make development easy for contributors and *not* have them battle about 5 language-specific package managers that may or may not cover all our needs to fill out extlib, it would be nice if there was a way to just script out "fetch all this stuff and unpack it here"

I've been trying to model out what the data needs of such a file would look like, and this looks maybe about right-ish:

EXTLIBS = [
    Extlib(
        "lato",
        verify=check_md5("ab516ef108"),
        install=fetch_and_unpack(
            "http://example.org/lato-1.7.tar.gz",
            move_from="lato-1.7"),
        # complete and corresponding source
        c_and_cs=fetch_and_unpack("http://example.org/lato-source.git")
    ),
    Extlib(
        "pdf.js",
        verify=check_md5("adc95ed383ec00cf4342bf4dd343e9bb"),
        install=fetch_and_unpack(
            "http://example.org/pdf.js-8.7.tar.gz",
            move_from="pdfjs-8.7"),
        # complete and corresponding source
        c_and_cs=print_instructions("git clone git://gitpit.example/pdfjs/")
    )
]

A script would still have to be written and carried alongside it. It might be nice to give it a more lax license than the rest of our code but I am somewhat loose about that.

[Christopher Allan Webber]

A conversation with Kuno / warp about a better extlib solution

[warp]

I have some further thoughts and quick experiments in an extlib-experiments branch here:

​https://github.com/warpr/mediagoblin/blob/extlib-experiments/extlib/notes.txt

I later noticed that for some dependencies the not-including-the-library-in-the-source-tree thing is solved by using git submodules, how feasable is it to just do that for everything else in extlib?

[Asheesh Laroia]

Chris and I are considering doing something like:

• Make npm be a dependency for development, but

• Distribute a tarball of the dependencies with the release, so "end users" (people installing MediaGoblin on their own systems) don't have to know we use npm, and

• Also distribute a tarball without the dependencies embedded to satisfy nice people in Debian who want a pristine tarball of just MediaGoblin so they can fetch their own dependencies.

[Christopher Allan Webber]

So, paulproteus started a branch related to this, and I've been extending it and pushing up to my personal repo. I think the code is now approximately good enough for merge (??):

​https://gitorious.org/mediagoblin/cwebbers-mediagoblin/commits/npm-or-bower

It needs some testing. I've verified that most things work. I need to check that the workflow works right for users. If anyone wants to volunteer... :)

[Christopher Allan Webber]

Okay, I've tested things hopefully good enough! ;) This is now merged into git master.