Implement Cloud Files public storage backend

[joar]

Use the StorageInterface to make a new public storage backend.

[joar]

Cloud Files public store support is working on my local copy.

OpenStack Object Store confirmed by creiht of Freenode's #openstack-dev:

04/00:01.55 Does Swift fully support the python-cloudfiles bindings? [...] 04/00:16.46 joar: the python-cloudfiles bindings should work with swift [...] 04/00:22.21 thanks creiht! 04/00:22.41 Is there any service out there using openstack currently? 04/00:24.26 joar: swift is what backs cloud files 04/00:24.37 at rackspace 04/00:24.41 Ok, that's great. 04/00:25.13 there are several other deployments of swift 04/00:25.54 Internap, Nephoscale, Korea Telecom off the top of my head 04/00:26.03 Can't speak for Nova 04/00:27.06 Wikimedia was experimenting with it to store images for wikipedia, but I never heard if they turned it all the way on 04/00:27.50 creiht: I was actually wondering because I needed to confirm that my development will also work with Swift setups. 04/00:28.10 ahh cool 04/00:28.37 it should then 04/00:28.38 But if Cloud Files are powered by Swift, I've already done so. 04/00:28.43 :) 04/00:28.48 cool, may I ask which tool? 04/00:28.54 MediaGoblin 04/00:29.02 or GNU MediaGoblin 04/00:29.03 The only main difference between the two, is to make the auth endpoint url configurable 04/00:29.13 Yes, that's implemented :) 04/00:29.20 also servicenet.

[joar]

Created merge request; https://gitorious.org/mediagoblin/mediagoblin/merge_requests/11

[Christopher Allan Webber]

So exciting!

But, okay, a few things:

def _resolve_filepath(self, filepath):
        return '-'.join(
            clean_listy_filepath(filepath))

Would '/'.join make more sense here? Sounds like that's totally valid: http://www.rackspace.com/cloud/blog/2010/01/26/nested-folders-in-rackspace-cloud-files/

Also the indentation is deeper than it should be in this method.

Also, would it make sense that instead of specifying the container, the first element of the path is the container? So in ['dir1', 'dir2', 'dir3', 'file.jpg'], the container is 'dir1'. Maybe that isn't safe... I guess we haven't guaranteed that len(file_path) > 1, which would otherwise be necessary (though currently that's our usage). So I guess this does make sense. (Maybe put another way, is there anything bad about having an insane number of files in one huge container? I'm guessing probably not... hopefully not anyway :) But it seems like the design was to allow for insane numbers of files in one container, so it's probably fine.)

2 more nitpicks:

• I'd rather the cloud storage class appear after BasicFileStore... BasicFileStore is the reference implementation and so should be nearer the top of the file!
• I'd rather that documentation inside the config file on how to use this go on the wiki than in the config file itself? Though I'd be fine with merging it as-is, it's probably TMI for someone just setting up the most basic site.

But yes, looking great, stellar, amazing, fantastic. I'm stoked.

[joar]

• mediagoblin.storage is now fully PEP-8 compliant.
• mediagoblin.storage:BaseFileStorage precedes CloudFilesStorage
• I have removed all the cloudfiles-specific options from mediagoblin.ini, it's now in the wiki at http://wiki.mediagoblin.org/ConfigureMediaGoblin#Setting_up_Cloud_Files_public_storage
• I have NOT changed the container to be set automatically after dir1. As you say, I don't think that's the thought with containers, they are not so much directories as they are some kind of pedagogical separation when handling multiple projects on a single Cloud Files account.

Will post link to repository as soon as Gitorious is back up.

[joar]

New merge request; https://gitorious.org/mediagoblin/mediagoblin/merge_requests/11

[Christopher Allan Webber]

Soooo awesome. Will merge tonight.

[Christopher Allan Webber]

Annnnnd merged!

That was a crazy awesome discussion/implementation/merge turnaround time!

[Will Kahn-Greene]

The original url for this bug was ​http://bugs.foocorp.net/issues/477 .