Opened 12 years ago

Closed 12 years ago

Last modified 9 years ago

#438 closed defect (fixed)

Celeryd Segmentation Fault

Reported by: Jorge Araya Navarro Owned by: Jorge Araya Navarro
Priority: blocker Milestone:
Component: programming Keywords:
Cc: jorgean@… Parent Tickets:

Description

since the last code update of mediagoblin in goblinartists.net, I upload a music or video file but it doesn't appears on the site, so I did something to catch some debug info and this is what I got:

[goblinartists@goblinartists mediagoblin]$ CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd -f celeryd.log -l debug --autoscale=20,3
Segmentation fault
[goblinartists@goblinartists mediagoblin]$ cat celeryd.log 
Using celeryd config: mediagoblin_local.ini
Using ./bin/celeryd
No CloudFiles host URL specified, defaulting to Rackspace US
[2012-04-19 22:28:47,816: WARNING/MainProcess] -------------- celery@goblinartists.net v2.5.1
---- **** -----
--- * ***  * -- [Configuration]
-- * - **** ---   . broker:      sqlalchemy://sqlite:////srv/http/goblinartists.net/mediagoblin/kombu.db//
- ** ----------   . loader:      celery.loaders.default.Loader
- ** ----------   . logfile:     celeryd.log@DEBUG
- ** ----------   . concurrency: {min=3, max=20}
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        OFF
-- ******* ----
--- ***** ----- [Queues]
 --------------   . celery:      exchange:celery (direct) binding:celery
                  

[Tasks]
  . celery.backend_cleanup
  . celery.chord
  . celery.chord_unlock
  . mediagoblin.processing.task.ProcessMedia
[2012-04-19 22:28:47,817: DEBUG/MainProcess] [Worker] Loading modules.
[2012-04-19 22:28:47,830: DEBUG/MainProcess] [Worker] Claiming components.
[2012-04-19 22:28:47,831: DEBUG/MainProcess] [Worker] Building boot step graph.
[2012-04-19 22:28:47,832: DEBUG/MainProcess] [Worker] New boot order: ['queues', 'pool', 'mediator', 'beat', 'autoreloader', 'timers', 'state-db', 'autoscaler', 'consumer']
[2012-04-19 22:28:47,834: DEBUG/MainProcess] Starting celery.concurrency.processes.TaskPool...
[2012-04-19 22:28:47,859: DEBUG/MainProcess] created semlock with handle 140224292536320
[2012-04-19 22:28:47,860: DEBUG/MainProcess] created semlock with handle 140224292532224
[2012-04-19 22:28:47,860: DEBUG/MainProcess] created semlock with handle 140224292528128
[2012-04-19 22:28:47,860: DEBUG/MainProcess] created semlock with handle 140224292524032
[2012-04-19 22:28:47,860: DEBUG/MainProcess] created semlock with handle 140224292519936
[2012-04-19 22:28:47,860: DEBUG/MainProcess] created semlock with handle 140224292515840
[2012-04-19 22:28:47,860: DEBUG/MainProcess] created semlock with handle 140224292511744
[2012-04-19 22:28:47,861: DEBUG/MainProcess] created semlock with handle 140224292507648
[2012-04-19 22:28:47,861: DEBUG/MainProcess] created semlock with handle 140224292503552
[2012-04-19 22:28:47,873: DEBUG/MainProcess] created semlock with handle 140224292499456
[2012-04-19 22:28:47,874: DEBUG/MainProcess] created semlock with handle 140224292495360
[2012-04-19 22:28:47,875: DEBUG/MainProcess] created semlock with handle 140224292491264
[2012-04-19 22:28:47,875: DEBUG/MainProcess] created semlock with handle 140224292487168
[2012-04-19 22:28:47,875: DEBUG/MainProcess] created semlock with handle 140224292483072
[2012-04-19 22:28:47,875: INFO/PoolWorker-1] child process calling self.run()
[2012-04-19 22:28:47,886: DEBUG/MainProcess] created semlock with handle 140224292478976
[2012-04-19 22:28:47,887: DEBUG/MainProcess] created semlock with handle 140224292474880
[2012-04-19 22:28:47,888: DEBUG/MainProcess] created semlock with handle 140224291917824
[2012-04-19 22:28:47,887: INFO/PoolWorker-2] child process calling self.run()
[2012-04-19 22:28:47,888: DEBUG/MainProcess] created semlock with handle 140224291913728
[2012-04-19 22:28:47,888: DEBUG/MainProcess] created semlock with handle 140224291909632
[2012-04-19 22:28:47,901: DEBUG/MainProcess] worker handler starting
[2012-04-19 22:28:47,901: INFO/PoolWorker-3] child process calling self.run()
[2012-04-19 22:28:47,903: DEBUG/MainProcess] celery.concurrency.processes.TaskPool OK!
[2012-04-19 22:28:47,903: DEBUG/MainProcess] Starting celery.worker.mediator.Mediator...
[2012-04-19 22:28:47,903: DEBUG/MainProcess] result handler starting
[2012-04-19 22:28:47,904: DEBUG/MainProcess] celery.worker.mediator.Mediator OK!
[2012-04-19 22:28:47,904: DEBUG/MainProcess] Starting celery.worker.autoscale.Autoscaler...
[2012-04-19 22:28:47,904: DEBUG/MainProcess] celery.worker.autoscale.Autoscaler OK!
[2012-04-19 22:28:47,905: DEBUG/MainProcess] Starting celery.worker.consumer.Consumer...
[2012-04-19 22:28:47,905: WARNING/MainProcess] celery@goblinartists.net has started.
[2012-04-19 22:28:47,905: DEBUG/MainProcess] Consumer: Re-establishing connection to the broker...
[2012-04-19 22:28:47,906: DEBUG/MainProcess] Consumer: Connection established.
[2012-04-19 22:28:47,989: DEBUG/MainProcess] basic.qos: prefetch_count->8
[2012-04-19 22:28:48,002: DEBUG/MainProcess] Consumer: Starting message consumer...
[2012-04-19 22:28:48,003: DEBUG/MainProcess] Consumer: Ready to accept tasks!
[2012-04-19 22:29:11,291: INFO/MainProcess] Got task from broker: mediagoblin.processing.task.ProcessMedia[c680cb6d-e17d-4d61-8729-55dae31b89b9]
[2012-04-19 22:29:11,326: DEBUG/MainProcess] Mediator: Running callback for task: mediagoblin.processing.task.ProcessMedia[c680cb6d-e17d-4d61-8729-55dae31b89b9]
[2012-04-19 22:29:11,326: DEBUG/MainProcess] TaskPool: Apply <function execute_and_trace at 0x3a2fb18> (args:('mediagoblin.processing.task.ProcessMedia', u'c680cb6d-e17d-4d61-8729-55dae31b89b9', [u'38'], {}) kwargs:{'hostname': 'goblinartists.net', 'request': {'retries': 0, 'task': 'mediagoblin.processing.task.ProcessMedia', 'utc': False, 'loglevel': 10, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'args': [u'38'], 'expires': None, 'is_eager': False, 'eta': None, 'hostname': 'goblinartists.net', 'kwargs': {}, 'logfile': 'celeryd.log', 'id': u'c680cb6d-e17d-4d61-8729-55dae31b89b9'}})
[2012-04-19 22:29:11,328: DEBUG/MainProcess] Task accepted: mediagoblin.processing.task.ProcessMedia[c680cb6d-e17d-4d61-8729-55dae31b89b9] pid:1572
[2012-04-19 22:29:11,414: WARNING/PoolWorker-3] No idea what happened here, but it failed: NoResultFound('No row was found for one()',)
[2012-04-19 22:29:11,421: DEBUG/PoolWorker-2] worker got EOFError or IOError -- exiting
[2012-04-19 22:29:11,421: DEBUG/PoolWorker-1] worker got EOFError or IOError -- exiting
[2012-04-19 22:29:11,421: DEBUG/PoolWorker-2] worker exiting after 0 tasks
[2012-04-19 22:29:11,421: DEBUG/PoolWorker-1] worker exiting after 0 tasks
[2012-04-19 22:29:11,421: INFO/PoolWorker-2] process shutting down
[2012-04-19 22:29:11,421: INFO/PoolWorker-1] process shutting down
[2012-04-19 22:29:11,421: DEBUG/PoolWorker-2] running all "atexit" finalizers with priority >= 0
[2012-04-19 22:29:11,421: DEBUG/PoolWorker-1] running all "atexit" finalizers with priority >= 0
[2012-04-19 22:29:11,421: DEBUG/PoolWorker-2] running the remaining "atexit" finalizers
[2012-04-19 22:29:11,421: DEBUG/PoolWorker-1] running the remaining "atexit" finalizers
[2012-04-19 22:29:11,421: INFO/PoolWorker-2] process exiting with exitcode 0
[2012-04-19 22:29:11,421: INFO/PoolWorker-1] process exiting with exitcode 0
[2012-04-19 22:29:11,420: DEBUG/PoolWorker-3] worker got EOFError or IOError -- exiting
[2012-04-19 22:29:11,424: DEBUG/PoolWorker-3] worker exiting after 1 tasks
[2012-04-19 22:29:11,424: INFO/PoolWorker-3] process shutting down
[2012-04-19 22:29:11,424: DEBUG/PoolWorker-3] running all "atexit" finalizers with priority >= 0
[2012-04-19 22:29:11,424: DEBUG/PoolWorker-3] running the remaining "atexit" finalizers
[2012-04-19 22:29:11,424: INFO/PoolWorker-3] process exiting with exitcode 0

looks like [2012-04-19 22:29:11,414: WARNING/PoolWorker-3] No idea what happened here, but it failed: NoResultFound('No row was found for one()',) is some sqlalchemy problem, maybe my mediagoblin_local.ini should help to find out if this is a bug or not.

[goblinartists@goblinartists mediagoblin]$ cat mediagoblin_local.ini 
# If you want to make changes to this file, first copy it to
# mediagoblin_local.ini, then make the changes there.

[mediagoblin]
direct_remote_path = /mgoblin_static/
email_sender_address = "noreply@goblinartists.net"

## Uncomment and change to your DB's appropiate setting.
## Default is a local sqlite db "mediagoblin.db".
sql_engine = postgresql:///mediagoblin

# set to false to enable sending notices
email_debug_mode = false

# Set to false to disable registrations
allow_registration = true

## Uncomment this to turn on video or enable other media types
media_types = mediagoblin.media_types.image, mediagoblin.media_types.video, mediagoblin.media_types.ascii, mediagoblin.media_types.audio
## Uncomment this to put some user-overriding templates here
#local_templates = %(here)s/user_dev/templates/

[storage:queuestore]
base_dir = %(here)s/user_dev/media/queue

[storage:publicstore]
#base_dir = %(here)s/user_dev/media/public
#base_url = /mgoblin_media/
storage_class = mediagoblin.storage.cloudfiles:CloudFilesStorage
#storage_class = mediagoblin.storage.cloudfiles
cloudfiles_user = shackra
cloudfiles_api_key = asdasdasdasdasdasdasdasdasdads
cloudfiles_container = mediagoblin
cloudfiles_use_servicenet = false

[celery]
[goblinartists@goblinartists mediagoblin]$

and no, gmg dbupdate doesn't show any special output

[goblinartists@goblinartists mediagoblin]$ ./bin/gmg dbupdate
** Message: pygobject_register_sinkfunc is deprecated (GstObject)
/srv/http/goblinartists.net/mediagoblin/lib/python2.7/site-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py:473: SAWarning: Unicode type received non-unicode bind param value.
[goblinartists@goblinartists mediagoblin]$

Change History (3)

comment:1 by Christopher Allan Webber, 12 years ago

You said you were going to re-checkout... after you have that working, and since #439 was fixed, do you still have this error?

comment:2 by Jorge Araya Navarro, 12 years ago

Resolution: → fixed
Status: new → closed

just update kombu to 2.1.5
./bin/easy_install kombu==2.1.5 and this bug should don't appears anymore

in reply to:  1 comment:3 by Jorge Araya Navarro, 12 years ago

Replying to cwebber:

You said you were going to re-checkout... after you have that working, and since #439 was fixed, do you still have this error?

Yes, the bug disappears just updating kombu, but I don't updated my code yet (so, idk if #439 is solved for me until I upgrade)

Note: See TracTickets for help on using tickets.