Make MediaGoblin work with latest celery, kombu

[Christopher Allan Webber]

MediaGoblin is currently locked to very specific versions of Celery & Kombu in its setup.py requirements field. We should remove this restriction and make MediaGoblin work with the latest versions

[bigjust]

pinning to specific versions is the only way to sanely manage deployments. Unless there is a specific feature that only the latest version has, and mediagoblin needs, then it would make since to upgrade.

[Christopher Allan Webber]

Spaetz has a branch here: ​https://gitorious.org/~spaetz/mediagoblin/spaetz-mediagoblin/commits/488_update_celery

I'm reviewing it now.

[Christopher Allan Webber]

• There was a reason we were setting the request id ourselves *before* the task started: this way we could retrieve information about the task later in something like the processing panel. With this change that looks like it is no longer possible? Is there any reason we can't do this anymore?

• One of these things says:

     Don't create a task-id directory per upload. Just store queued uploads
     in a single directory (this is the most controversial change and might
     need discussion!!!)

indeed, I think we need to discuss it :)

There are two important reasons for this:

1. This prevents naming conflicts. If we don't do this, both you and I might upload pony.jpg, and there would be a conflict indeed!
2. The plan was that this way plugins and etc which might pass along more files for the processing stage have a clean way of putting them somewhere that was entirely encapsulated with the main media file. Nothing is doing that yet, but it might?

• We probably do need to release note BROKER_HOST -> BROKER_URL, yeah :( Do you think we should provide a deprecated fallback with warning? Or does it have a conflicting meaning now?

Okay, that's it. Other than the above, I'm very excited about removing the celery restriction!...

[spaetz]

Hi Chris,

no it will still be possible to access the task-id even after we send off a celery task. We just need to store the AsyncResult in our MediaData when processing. It has access to the task ID and other task related things. This will also be a better place to store failure error messages rather than in the MediaEntry object itself.

As for the uploads and conflicts,... we are ensuring name uniqueness afaik, and we store processed files in their own directory later on anyway. But sure, if we want a task-id directory, than we should keep that, no problem.

spaetz

[Christopher Allan Webber]

I think given Spaetz and I's comments above, this might need some more work till it can be merged? Though I'm not sure. I'm moving this off the review queue for now... if someone wants to help me get this merged I'd be happy to work with them on that.

[rodney757]

So I used the celery stuff out of this branch and addressed cwebber's concerns. Now mediagoblin works with the latest celery.