Opened 4 years ago

Closed 4 years ago

#5609 closed defect (fixed)

Celery connection to RabbitMQ errors at start or after running for a few minutes

Reported by: Fernando Gutierrez Owned by:
Priority: major Milestone:
Component: programming Keywords:
Cc: Parent Tickets:


When Celery is starting or after it has run for a few minutes I get many "connection reset" errors in the log.

After this errors Celery does not recover and appears to be running but does not process any tasks.

There are two root causes for this problem:

1) The systemd service file provided as example in the deployment guide does not specify that celery must be started after RabbitMQ. The OS is free to start them in any order and when celery starts before RabbitMQ it will not be able to connect

2) In mediagoblin/init/celery/ the BROKER_HEARTBEAT is set to 1 second. In slower machines or under heavy load this setting causes missed hearbeats and a eventual connection reset.
I think this code had the intention to cause a more granular task scheduling but from what I understand from celery docs it has nothing to do with task scheduling. The CELERYBEAT_SCHEDULE is just a periodic task and has nothing to do with BROKER_HEARTBEAT.

Attachments (1)

fix_celery_connections.diff (897 bytes ) - added by Fernando Gutierrez 4 years ago.
Proposed fix

Download all attachments as: .zip

Change History (2)

by Fernando Gutierrez, 4 years ago

Attachment: fix_celery_connections.diff added

Proposed fix

comment:1 by Ben Sturmfels, 4 years ago

Resolution: fixed
Status: newclosed

Thanks very much Fernando, merged!

We really appreciate the work you've done here to investigate this issue.

Note: See TracTickets for help on using tickets.