Handling languages Babel currently doesn't (Na'vi, Lojban)

[Christopher Allan Webber]

Currently, Babel doesn't support languages like Na'vi, Lojban. The unfortunate side effect of this is that when our language tooling hits these languages, it explodes.

Looking into how to fix this, it looks like Babel ​downloads the ​CLDR data on languages, and this is used to populate a big ol' dictionary which is stored in ​babel.localedata:_cache. Unfortunately, while we *can* monkeypatch this variable (simply ​load('en_US') once then append 'jbo' or etc to the dictionary), the value we would have to supply is a very, very complex data structure, and I doubt we could provide it correctly. This value is extracted by Babel during the ​CLDR data downloading/importing process and it is a non-trivial exercise.

I ​filed a bug looking for a workaround for unidentified languages, but I think a workaround will be hard given the large amount Babel relies on this data. So, probably, we will need to supply CLDR data, but that data is too complex to just write by hand.

One way or another (monkeypatching or not), in order to supply these, we need the proper CLDR data. I spoke to someone involved in the process, and found two things:

• They are welcome to contributions of Na'vi and Lojban to their catalog, and there is a ​new contributor page. We should (and likely will) do this. Probably, the process for getting things officially will take some time.
• In the meanwhile, while going through the process to generate those, we should be able to get the appropriate files to run through the necessary tooling to generate the data we need.

[Boris Bobrov]

I'm getting this now:

breton@breton-pc ~/src/mediagoblin (master*) $ make
./devtools/update_extlib.sh
./devtools/compile_translations.sh
compiling catalog 'mediagoblin/i18n/de/mediagoblin.po' to 'mediagoblin/i18n/de/LC_MESSAGES/mediagoblin.mo'
compiling catalog 'mediagoblin/i18n/hi/mediagoblin.po' to 'mediagoblin/i18n/hi/LC_MESSAGES/mediagoblin.mo'
compiling catalog 'mediagoblin/i18n/ja/mediagoblin.po' to 'mediagoblin/i18n/ja/LC_MESSAGES/mediagoblin.mo'
compiling catalog 'mediagoblin/i18n/nn_NO/mediagoblin.po' to 'mediagoblin/i18n/nn_NO/LC_MESSAGES/mediagoblin.mo'
compiling catalog 'mediagoblin/i18n/hu/mediagoblin.po' to 'mediagoblin/i18n/hu/LC_MESSAGES/mediagoblin.mo'
compiling catalog 'mediagoblin/i18n/gl/mediagoblin.po' to 'mediagoblin/i18n/gl/LC_MESSAGES/mediagoblin.mo'
compiling catalog 'mediagoblin/i18n/ca@valencia/mediagoblin.po' to 'mediagoblin/i18n/ca@valencia/LC_MESSAGES/mediagoblin.mo'
Traceback (most recent call last):
  File "./bin/pybabel", line 11, in <module>
    sys.exit(main())
  File "/home/breton/src/mediagoblin/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 1151, in main
    return CommandLineInterface().run(sys.argv)
  File "/home/breton/src/mediagoblin/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 665, in run
    return getattr(self, cmdname)(args[1:])
  File "/home/breton/src/mediagoblin/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 765, in compile
    catalog = read_po(infile, locale)
  File "/home/breton/src/mediagoblin/local/lib/python2.7/site-packages/babel/messages/pofile.py", line 122, in read_po
    catalog = Catalog(locale=locale, domain=domain, charset=charset)
  File "/home/breton/src/mediagoblin/local/lib/python2.7/site-packages/babel/messages/catalog.py", line 279, in __init__
    locale = Locale.parse(locale)
  File "/home/breton/src/mediagoblin/local/lib/python2.7/site-packages/babel/core.py", line 315, in parse
    raise UnknownLocaleError(input_id)
babel.core.UnknownLocaleError: unknown locale 'ia'
Makefile:118: recipe for target 'i18n' failed
make: *** [i18n] Error 1

[ayleph]

Could we ignore Interlingua errors like we ignore Lojban errors in devtools/compile_translations.sh?

 for file in `find mediagoblin/i18n/ -name "*.po"`; do
     if [ "$file" != "mediagoblin/i18n/jbo/mediagoblin.po" ] && \
+       [ "$file" != "mediagoblin/i18n/ia/mediagoblin.po" ] && \
        [ "$file" != "mediagoblin/i18n/templates/en/mediagoblin.po" ]; then
         mkdir -p `dirname $file`/LC_MESSAGES/;
         $PYBABEL compile -i $file \
               -o `dirname $file`/LC_MESSAGES/mediagoblin.mo \
               -l `echo $file | awk -F / '{ print $3 }'`;
    else
        echo "Skipping $file which pybabel can't compile :(";
    fi;
done

[Christopher Allan Webber]

Yes, I think that's the right call. Looks like you pushed to master, thanks!

[Christopher Allan Webber]

(Note that that was the workaround for 'ia' locale, not for the general issue itself)

[Boris Bobrov]

a very interesting workaround was suggested in #5384. Maybe we could work out the proper way of getting locale data from the workaround?

[Christopher Allan Webber]

Interesting suggestion indeed in #5384. What is in the python-babel-localedata package anyway? I wonder why it wouldn't be installed by default if it includes useful locales.

I'm not sure how we'd find the path and link in locales from it though generally, on non-Debian systems. Any idea?

(Also, will it work with Na'vi and Lojban?)