celery crash

[davidak]

my setup:

virtual machine with 2 cores and 4 gb ram
debian 7
postgresql
paster
nginx
mediagoblin 0.6.2.dev
​init script from joar

i uploaded some videos. they get transcoded, but after some time it stops.
new uploaded media don't appear on the website.

also stopping of celery don't work.

root@mediagoblin:~# /etc/init.d/mediagoblin-celeryd stop
Stopping GNU MediaGoblin Celery task queue: mediagoblin-celerydFailed to shut down after 20 seconds. Aborting.
 failed!

very often the same lines in the celery log:

Traceback (most recent call last):
  File "/var/www/mediagoblin/mediagoblin/media_types/video/transcoders.py", line 687, in _on_message
    self._progress_callback(data.get('percent'))
  File "/var/www/mediagoblin/mediagoblin/processing/__init__.py", line 43, in __call__
    self.entry.save()
  File "/var/www/mediagoblin/mediagoblin/db/base.py", line 39, in save
    sess.commit()
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 721, in commit
    self.transaction.commit()
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 352, in commit
    self._assert_active(prepared_ok=True)
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 194, in _assert_active
    % self._rollback_exception
sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: This result object does not return rows. It has been closed automatically.
/var/www/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.10-py2.7.egg/celery/app/trace.py:348: RuntimeWarning: Exception raised outside body: InvalidRequestError("This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: This result object does not return rows. It has been closed automatically.",):
Traceback (most recent call last):
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.10-py2.7.egg/celery/app/trace.py", line 253, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.10-py2.7.egg/celery/app/trace.py", line 201, in on_error
    R = I.handle_error_state(task, eager=eager)
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.10-py2.7.egg/celery/app/trace.py", line 85, in handle_error_state
    }[self.state](task, store_errors=store_errors)
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.10-py2.7.egg/celery/app/trace.py", line 120, in handle_failure
    task.on_failure(exc, req.id, req.args, req.kwargs, einfo)
  File "/var/www/mediagoblin/mediagoblin/processing/task.py", line 154, in on_failure
    mark_entry_failed(entry_id, exc)
  File "/var/www/mediagoblin/mediagoblin/processing/__init__.py", line 334, in mark_entry_failed
    u'fail_metadata': {}})
  File "/var/www/mediagoblin/mediagoblin/db/util.py", line 31, in atomic_update
    synchronize_session=False)
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2710, in update
    update_op.exec_()
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 829, in exec_
    self._do_pre()
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 851, in _do_pre
    session._autoflush()
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1139, in _autoflush
    self.flush()
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1824, in flush
    self._flush(objects)
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1902, in _flush
    subtransactions=True)
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 645, in begin
    nested=nested)
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 215, in _begin
    self._assert_active()
  File "/var/www/mediagoblin/local/lib/python2.7/site-packages/SQLAlchemy-0.8.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 194, in _assert_active
    % self._rollback_exception
InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: This result object does not return rows. It has been closed automatically.

[ayleph]

This issue was addressed in the v0.7.1 release. From the release notes [1]:

This release is especially useful if you have been running postgres and have been receiving seemingly random database transaction errors.

I am closing this ticket as the issue described has been addressed. If someone still experiences the same issue after upgrading to v0.7.1 or later, then please re-open the ticket with details.

[1] ​http://mediagoblin.readthedocs.org/en/v0.7.1/siteadmin/relnotes.html