Opened 12 years ago

Closed 11 years ago

#488 closed task (fixed)

Make MediaGoblin work with latest celery, kombu

Reported by: Christopher Allan Webber Owned by:
Priority: major Milestone: 0.5.0
Component: programming Keywords: review
Cc: Will Kahn-Greene, spaetz Parent Tickets:


MediaGoblin is currently locked to very specific versions of Celery & Kombu in its requirements field. We should remove this restriction and make MediaGoblin work with the latest versions

Change History (9)

comment:1 by Christopher Allan Webber, 12 years ago

Type: defecttask

comment:2 by bigjust, 11 years ago

pinning to specific versions is the only way to sanely manage deployments. Unless there is a specific feature that only the latest version has, and mediagoblin needs, then it would make since to upgrade.

comment:3 by Christopher Allan Webber, 11 years ago

Cc: spaetz added

comment:4 by Christopher Allan Webber, 11 years ago

  • There was a reason we were setting the request id ourselves *before* the task started: this way we could retrieve information about the task later in something like the processing panel. With this change that looks like it is no longer possible? Is there any reason we can't do this anymore?
  • One of these things says:
     Don't create a task-id directory per upload. Just store queued uploads
     in a single directory (this is the most controversial change and might
     need discussion!!!)

indeed, I think we need to discuss it :)

There are two important reasons for this:

  1. This prevents naming conflicts. If we don't do this, both you and I might upload pony.jpg, and there would be a conflict indeed!
  2. The plan was that this way plugins and etc which might pass along more files for the processing stage have a clean way of putting them somewhere that was entirely encapsulated with the main media file. Nothing is doing that yet, but it might?
  • We probably do need to release note BROKER_HOST -> BROKER_URL, yeah :( Do you think we should provide a deprecated fallback with warning? Or does it have a conflicting meaning now?

Okay, that's it. Other than the above, I'm very excited about removing the celery restriction!...

comment:5 by spaetz, 11 years ago

Hi Chris,

no it will still be possible to access the task-id even after we send off a celery task. We just need to store the AsyncResult in our MediaData when processing. It has access to the task ID and other task related things. This will also be a better place to store failure error messages rather than in the MediaEntry object itself.

As for the uploads and conflicts,... we are ensuring name uniqueness afaik, and we store processed files in their own directory later on anyway. But sure, if we want a task-id directory, than we should keep that, no problem.


comment:6 by Christopher Allan Webber, 11 years ago

Keywords: review added

comment:7 by Christopher Allan Webber, 11 years ago

Status: newreview

comment:8 by Christopher Allan Webber, 11 years ago

Status: reviewaccepted

I think given Spaetz and I's comments above, this might need some more work till it can be merged? Though I'm not sure. I'm moving this off the review queue for now... if someone wants to help me get this merged I'd be happy to work with them on that.

comment:9 by rodney757, 11 years ago

Milestone: 0.5.0
Resolution: fixed
Status: acceptedclosed

So I used the celery stuff out of this branch and addressed cwebber's concerns. Now mediagoblin works with the latest celery.

Note: See TracTickets for help on using tickets.