lazycelery raises IndexError

[Boris Bobrov]

I have default mg 0.8.1 setup and trying to run ./lazycelery doesn't work for me with error:

("\nThe 'celeryd' command is deprecated, please use 'celery worker' instead:\n\n$ celery worker -B\n\n",)
WARNING:mediagoblin.init.config:When setting up config section, could not import 'mediagoblin.media_types.image'
WARNING:mediagoblin.init.config:When setting up config section, could not import 'mediagoblin.media_types.video'
[2016-02-25 02:07:17,246: WARNING/MainProcess] /home/mediagoblin/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/apps/worker.py:162: CDeprecationWarning: 
Starting from version 3.2 Celery will refuse to accept pickle by default.

The pickle serializer is a security concern as it may give attackers
the ability to execute any command.  It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::

    CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.


  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
 
 -------------- celery@picnic v3.1.20 (Cipater)
---- **** ----- 
--- * ***  * -- Linux-3.16.0-4-686-pae-i686-with-debian-8.3
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         default:0xb6c3bb4c (.default.Loader)
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- 
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery
                

[2016-02-25 02:07:17,250: ERROR/MainProcess] Unrecoverable error: IndexError('list index out of range',)
Traceback (most recent call last):
  File "/home/mediagoblin/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/home/mediagoblin/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/bootsteps.py", line 119, in start
    self.on_start()
  File "/home/mediagoblin/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/apps/worker.py", line 173, in on_start
    self.set_process_status('-active-')
  File "/home/mediagoblin/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/apps/worker.py", line 277, in set_process_status
    info='{0} ({1})'.format(info, platforms.strargv(sys.argv)),
  File "/home/mediagoblin/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/platforms.py", line 718, in strargv
    arg_start = 2 if 'manage' in argv[0] else 1
IndexError: list index out of range

Right now I am debugging another issue. Chris, could you please take a look if the bug is still in 0.9.0?

[ayleph]

No problems on 0.8.1 or 0.9.0 for me using celery-3.1.13 inside py2 virtualenv.

 ~/mediagoblin-dev $ bin/pip show celery
---
Name: celery
Version: 3.1.13
Location: /srv/mediagoblin/mediagoblin-beta/lib/python2.7/site-packages/celery-3.1.13-py2.7.egg
Requires: pytz, billiard, kombu

I'll try upgrading celery and report back.

[ayleph]

Force-upgraded to 3.1.20 and still no problems in 0.8.1 or 0.9.0.

$ bin/pip show celery
---
Name: celery
Version: 3.1.20
Location: /srv/mediagoblin/mediagoblin-beta/lib/python2.7/site-packages
Requires: kombu, billiard, pytz

[ayleph]

Sample output from my machine. Note that I've set CELERYD_CONCURRENCY = 2 in my mediagoblin.ini.

  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))

 -------------- celery@goblinrefuge v3.1.20 (Cipater)
---- **** -----
--- * ***  * -- Linux-4.2.5-1-ARCH-x86_64-with-glibc2.2.5
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         default:0x7feca07b1e10 (.default.Loader)
- ** ---------- .> transport:   sqlite:////srv/mediagoblin/mediagoblin-beta/kombu.db
- ** ---------- .> results:     database
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . mediagoblin.notifications.task.EmailNotificationTask
  . mediagoblin.processing.task.ProcessMedia
  . mediagoblin.processing.task.handle_push_urls
  . mediagoblin.submit.task.collect_garbage

2016-02-24 20:05:07,719 INFO    [celery.beat] beat: Starting...
2016-02-24 20:05:07,750 INFO    [celery.worker.consumer] Connected to sqlite:////srv/mediagoblin/mediagoblin-beta/kombu.db
2016-02-24 20:05:07,756 DEBUG   [celery.beat] Current schedule:
<Entry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)>
<Entry: garbage-collection mediagoblin.submit.task.collect_garbage() <freq: 1.00 hour>
2016-02-24 20:05:07,756 DEBUG   [celery.beat] beat: Ticking with max interval->5.00 minutes
2016-02-24 20:05:07,787 DEBUG   [celery.beat] beat: Waking up in 5.00 minutes.

[Boris Bobrov]

This happened because the virtualenv was created without system-site packages. To fix this you need to create the venv with --system-site-packages