Media processing fails on video with year only

[Ben Sturmfels]

Re-processing a video that failed to be processed gives the following traceback:

$ bin/gmg reprocess --celery initial
DEBUG:mediagoblin.processing.task:Processing <MediaEntry 19: rock-your-emacs>

(python:3524): GStreamer-CRITICAL **: gst_date_time_get_month: assertion 'gst_date_time_has_month (datetime)' failed

(python:3524): GStreamer-CRITICAL **: gst_date_time_get_day: assertion 'gst_date_time_has_day (datetime)' failed

(python:3524): GStreamer-CRITICAL **: gst_date_time_get_hour: assertion 'gst_date_time_has_time (datetime)' failed

(python:3524): GStreamer-CRITICAL **: gst_date_time_get_minute: assertion 'gst_date_time_has_time (datetime)' failed

(python:3524): GStreamer-CRITICAL **: gst_date_time_get_second: assertion 'gst_date_time_has_second (datetime)' failed

(python:3524): GStreamer-CRITICAL **: gst_date_time_get_microsecond: assertion 'gst_date_time_has_second (datetime)' failed
ERROR:mediagoblin.processing.task:An unhandled exception was raised while processing <MediaEntry 19: rock-your-emacs>
WARNING:mediagoblin.processing:No idea what happened here, but it failed: ValueError('month must be in 1..12',)
WARNING:mediagoblin.processing:No idea what happened here, but it failed: ValueError('month must be in 1..12',)
Traceback (most recent call last):
  File "/home/web/local/mediagoblin/bin/gmg", line 9, in <module>
    load_entry_point('mediagoblin==0.8.0', 'console_scripts', 'gmg')()
  File "/home/web/local/mediagoblin/mediagoblin/gmg_commands/__init__.py", line 142, in main_cli
    args.func(args)
  File "/home/web/local/mediagoblin/mediagoblin/gmg_commands/reprocess.py", line 305, in reprocess
    initial(args)
  File "/home/web/local/mediagoblin/mediagoblin/gmg_commands/reprocess.py", line 280, in initial
    reprocess_action='initial')
  File "/home/web/local/mediagoblin/mediagoblin/submit/lib.py", line 257, in run_process_media
    task_id=entry.queued_task_id)
  File "/home/web/local/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/task.py", line 557, in apply_async
    link=link, link_error=link_error, **options)
  File "/home/web/local/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/task.py", line 755, in apply
    request=request, propagate=throw)
  File "/home/web/local/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/trace.py", line 355, in eager_trace_task
    uuid, args, kwargs, request)
  File "/home/web/local/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/trace.py", line 253, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/home/web/local/mediagoblin/local/lib/python2.7/site-packages/celery-3.1.20-py2.7.egg/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/web/local/mediagoblin/mediagoblin/processing/task.py", line 101, in run
    processor.process(**reprocess_info)
  File "/home/web/local/mediagoblin/mediagoblin/media_types/video/processing.py", line 393, in process
    vp8_threads=vp8_threads, vorbis_quality=vorbis_quality)
  File "/home/web/local/mediagoblin/mediagoblin/media_types/video/processing.py", line 255, in transcode
    store_metadata(self.entry, metadata)
  File "/home/web/local/mediagoblin/mediagoblin/media_types/video/processing.py", line 136, in store_metadata
    'tags': get_tags(audio_info)
  File "/home/web/local/mediagoblin/mediagoblin/media_types/video/processing.py", line 112, in get_tags
    dt.get_microsecond()).isoformat()
ValueError: month must be in 1..12

I'll need to prepare a smaller video that has similar metadata, but the video I was uploading on is here:

​http://www.sturm.com.au/2015/talks/rock-your-emacs-libreplanet/rock-your-emacs.webm

I'm assuming that the issue is that if there is any date information, MediaGoblin assumes the full date is supplied. This video only has a year specified I think, so the get_month() returns zero.

[ayleph]

I've got a similar issue in #5401 where the code assumes the video object has a property set without first checking against null.

[Ben Sturmfels]

The best I can do for a test video is to cut down the original talk video in Pitivi (where it was originally edited) and set the metadata through "Project settings". Resulting video is 19K (attached).

I was hoping to be able to provide a GStreamer one-liner to reproducibly create a test video, but after trying all sorts of incantations, was not able to force it to use a year-only date. This is where I got to with GStreamer. This is a single frame VP8/WebM video using GStreamer's test pattern, no audio. Unfortunately, it produces a warning and for setting a year-only date:

gst-launch-1.0 videotestsrc num-buffers=1 ! taginject tags="date=2009" ! video/x-raw,width=100,height=100 ! vp8enc ! webmmux ! filesink location=file.webm

Also tried using avconv to add metadata to the above video, but that metadata didn't show up it Totem or VLC either:

avconv -i file.webm -metadata date=2009 -metadata title=foo out.webm

This makes me wonder how Pitivi manages this trick, given it's GStreamer based. Perhaps it's cheating somehow and writing invalid data... I don't know. Even if it is though, MediaGoblin should probably deal sensibly with the invalid data.

[Ben Sturmfels]

Tiny snippet of original video, rendered by Pitivi, including year-only date metadata.

[Ben Sturmfels]

Is it appropriate it include the test video in MediaGoblin and fire up the whole media processing pipeline in the test case? If we really wanted to focus on testing just MediaGoblin, rather than also GStreamer, maybe the better option would be to mock the media object, providing invalid metadata?

[Ben Sturmfels]

I've just had a go at providing a mock stream_info object to mediagoblin.media_types.video.processing.get_tags(), but it gets complicated quickly where taglist.foreach takes a function.

To write a test, I need to make dt.get_year() return, say, 2016, while the rest of the dt.get_...() return 0. Any advice on how best to structure this test?

[Ian Kelling]

FSF would still like to get this fixed or figure out a workaround.

[ayleph]

I tested the example video provided by Sturm, and it processed fine on my relatively up-to-date instance. iank, are you still affected by this issue? Are you running recent-ish code?

Edit: no, sorry, I was wrong. It looks like I had already hacked in a workaround for this issue on my instance. When I tried again from master, the file failed to process. Here's the change I'm running in my modified instance.

--- mediagoblin/media_types/video/processing.py
+++ mediagoblin/media_types/video/processing-modified.py
@@ -116,10 +116,13 @@
         # TODO: handle timezone info; gst.get_time_zone_offset +
         # python's tzinfo should help
         dt = tags['datetime']
-        tags['datetime'] = datetime.datetime(
-            dt.get_year(), dt.get_month(), dt.get_day(), dt.get_hour(),
-            dt.get_minute(), dt.get_second(),
-            dt.get_microsecond()).isoformat()
+        try:
+            tags['datetime'] = datetime.datetime(
+                dt.get_year(), dt.get_month(), dt.get_day(), dt.get_hour(),
+                dt.get_minute(), dt.get_second(),
+                dt.get_microsecond()).isoformat()
+        except:
+            tags['datetime'] = None
     for k, v in tags.copy().items():
         # types below are accepted by json; others must not present
         if not isinstance(v, (dict, list, six.string_types, int, float, bool,

[ayleph]

It looks like I've been running this workaround in production since November 2015. Haven't noticed an issue with it in that time. I'll push this to master.

[ayleph]

This has been merged to master in ​a35007c.