UnicodeEncodeError with non-ASCII characters in comment emails

[ayleph]

Adding a comment with a non-ASCII character results in a server error when sending a notification email. Running py2 with close-to-latest master code. Eg, trying to add something like "テストコメントです！" results in the error below.

2016-03-28 20:45:19,114 INFO    [mediagoblin.notifications.task] Sending notification email about <Notification #7: <LocalUser #13 verified user "ayleph">: None (unseen)>
===== Email =====
From address: mediagoblin@goblinrefuge.com
To addresses: ayleph@thisshitistemp.com
Subject: Goblin Refuge Beta - machalus commented on your post
-- Body: --
Error - <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode characters in position 130-139: ordinal not in range(128)
URL: http://pumpdev.goblinrefuge.com/u/ayleph/m/43/comment/add/
File '/path/to/mediagoblin/lib/python2.7/site-packages/Paste-1.7.5.1-py2.7.egg/paste/exceptions/errormiddleware.py', line 144 in __call__
  app_iter = self.application(environ, sr_checker)
File '/path/to/mediagoblin/mediagoblin/app.py', line 342 in __call__
  return self.call_backend(environ, start_response)
File '/path/to/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.9.6-py2.7.egg/werkzeug/wsgi.py', line 588 in __call__
  return self.app(environ, start_response)
File '/path/to/mediagoblin/mediagoblin/app.py', line 276 in call_backend
  return self._finish_call_backend(request, environ, start_response)
File '/path/to/mediagoblin/mediagoblin/app.py', line 318 in _finish_call_backend
  response = controller(request)
File '/path/to/mediagoblin/mediagoblin/decorators.py', line 281 in wrapper
  return controller(request, media=media, *args, **kwargs)
File '/path/to/mediagoblin/mediagoblin/decorators.py', line 47 in wrapper
  return controller(request, *args, **kwargs)
File '/path/to/mediagoblin/mediagoblin/decorators.py', line 74 in new_controller_func
  return controller(request, *args, **kwargs)
File '/path/to/mediagoblin/mediagoblin/decorators.py', line 103 in wrapper
  return controller(request, *args, **kwargs)
File '/path/to/mediagoblin/mediagoblin/user_pages/views.py', line 210 in media_post_comment
  trigger_notification(link, media, request)
File '/path/to/mediagoblin/mediagoblin/notifications/__init__.py', line 63 in trigger_notification
  email_notification_task.apply_async([cn.id, message])
File '/path/to/mediagoblin/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/task.py', line 557 in apply_async
  link=link, link_error=link_error, **options)
File '/path/to/mediagoblin/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/task.py', line 755 in apply
  request=request, propagate=throw)
File '/path/to/mediagoblin/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/trace.py', line 355 in eager_trace_task
  uuid, args, kwargs, request)
File '/path/to/mediagoblin/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/trace.py', line 253 in trace_task
  I, R, state, retval = on_error(task_request, exc, uuid)
File '/path/to/mediagoblin/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/trace.py', line 240 in trace_task
  R = retval = fun(*args, **kwargs)
File '/path/to/mediagoblin/mediagoblin/notifications/task.py', line 44 in run
  message['body'])
File '/path/to/mediagoblin/mediagoblin/tools/mail.py', line 140 in send_email
  print(message_body)
File '/path/to/mediagoblin/lib/python2.7/site-packages/PasteScript-1.7.5-py2.7.egg/paste/script/serve.py', line 530 in write
  fileobj.write(text)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 130-139: ordinal not in range(128)

[ayleph]

Turns out this is only an issue when email_debug_mode = True. Dropping severity drastically.

[jsandoval]

Is this ticket still active?. I can't seem to be able to replicate it.

[ayleph]

Yes, this still occurs for me on v0.9.0 with python2.7.

[Dusan Ristic]

Please review..
​https://github.com/dusan87/mediagoblin/tree/dusan_bug_5449

I think it's caused by encode() method, since we already get encoded Chinese chars which are tried to be encoded again.

[Boris Bobrov]

Although this looks good, could you please add unit tests? Examples can be found in mediagoblin/tests/test_util.py

[Ben Sturmfels]

Hi Dusan,

Thanks very much for spending time with us at the PyConAU sprints!

I'm sure you showed me some tests for this patch at the event. Did you forget to add and commit the test file? Essentially what we need is for this test to demonstrate the existence of the bug before your patch is applied.

Regards,
Ben

[Ben Sturmfels]

Hi Ayleph,

Replying to ayleph:

Yes, this still occurs for me on v0.9.0 with python2.7.

I've just had a quick look at this old bug report and tried to reproduce it, but it seems like MediaGoblin isn't sending comment notification emails at all. We have a function send_comment_email, but it doesn't appear to be called anywhere in the codebase. I'll create a separate report for that.

Given we're now Python 3-only, there's a good chance this unicode problem is now a non-issue, so I'll close this bug report.

Regards,
Ben