Techno Barje

Jetpack localization using YAML format

2011-11-17T00:00:00+00:00

In a previous post, I’ve described my first proposal for localization support in jetpack addons. I’ve decided to change locale files format for YAML instead of JSON. During MozCamp event, folks helped me identifying some pitfalls with JSON:

No multiline string support. Firefox parser allows multiline but it is not officialy supported! So that it will disallow third party tools to work properly.
No easy way to add comments. It is mandatory for localizers to have context description in comments next to keys to translate. As there is no way to add comments in JSON, it will end up complexifying a lot locale format.

Example

French locale file in YAML format

# You can add comments with `#`…
Hello %s: Bonjour %s         # almost …
hello_key: Bonjour %s        # wherever you want!

# For multiline, you need to indent your string with spaces
multiline:
  "Bonjour
   %s"

# Plural forms.
# we use a nested object with attributes that depends on the target language
# in english, we only have 'one' (for 1) and 'other' (for everything but 1)
# in french, it is the same except that 'one' match 0 and 1
# in some language like Polish, there is 4 forms and 6 in arabic
#
# So that having a structured format like YAML,
# help us writing these translations!
pluralString:
  one: "%s telechargement"
  other: "%s telechargements"

# I need to enclode these strings with `"` because of %. See note after.

Addon code

// Get a reference to `_` gettext method with:
const _ = require("l10n").get;

// These three forms end up returning the same string.
// We can still use a locale string in code, or use a key.
// And multiline string gets its `\n` removed. (there is a way to keep them)
_("Hello %s", "alex") == _("hello_key", "alex") == _("multiline", "alex")

// Example of non-naive l10n feature, plurals:
_("pluralString", 0) == "0 telechargement"
_("pluralString", 1) == "1 telechargement"
_("pluralString", 10) == "10 telechargements"

Advantages of YAML

Multiline strings are supported nicely / easy to read. You do not need to add a final \ on all lines. As mulitiline is easier, localizers can use them more often and it will surely improve readability of locale files!
Structured data format. we can use this power whenever it is needed. For example, when we need to implement complex l10n features like plural forms or any feature that goes beyond simple 1-1 localization. The cool thing if we compare to JSON is that even if we define structures, we keep a really simple format with no noise (like {, }, “, …).

As nothing comes without any issues, here is what I’ve found around YAML:

This format is not a Web standard. I don’t think it makes much sense to avoid using it because of that. We are clearly missing a standardized format for localization in the web world.
You may hit some issues when you do not enclose your strings with " or ’. For example, you can’t start a string with %, nor having a : in middle of your string without enclosing it.
Even if YAML is not a web standard, it has been formaly specified. And unfortunately, a handy feature becomes a pitfall for our purpose! Some strings are automatically converted. Yes, True, False, … are automatically converted to a boolean value. We can work around this in multiple ways, either by documenting it, or modifying the parser. The same solution apply here, you need to enclose your string with quotes.

Again, feedback is welcomed on this group thread and you can follow this work in bug 691782.

Jetpack localization

2011-10-31T00:00:00+00:00

I’m going to describe the first proposal of localization support for Jetpack. This approach uses gettext pattern and json files for locales. It is the first step of multiple iterations. This one only allows retrieving localized string in javascript code. We are going to give ways to translate files, mainly HTML files, through another iteration. And we are about to offer an online tool to ease addon localization (like babelzilla website).

Let’s start by looking at a concrete example, then I’ll justify our different choices.

French locale file

{
  "Hello %s": "Bonjour %s",
  "hello_user": "Bonjour %s"
}

Addon code

// Retrieve a dynamic reference to `_` gettext method with:
const _ = require("l10n").get;

// Then print to the console a localized string:
console.log(_("Hello %s", "alex"));
// => Prints "Bonjour alex" in french.

// Or, if we don't want to use localized string in addon code:
console.log(_("hello_user", "alex"));

Why gettext?

It gives a way to automatically fetch localizable strings or ids from source code by searching for _( ) pattern.
It allows to use either strings or IDs as value to translate. It is obviously better to use IDs. Because locales will broke each time addon developer fix a typo in the main language hard coded in the code.

But we should not forget that the high level APIs is trying to simplify addon development. So that it has to be really easy to translate a simple addon that has only 2 JS files and less than 50 lines of code! And the simple fact to mandatory require a locale file for the default language appears like a big burden for such small addon.

Having said that, I’m really happy that gettext approach doesn’t discourage, nor makes it harder to use IDs, and so, if an addon developer build a big addon or just want to take more time to use better pratice, he still can do it, easily!

Why JSON for locales?

We could have used properties files, like XUL addons. But this format has some limitations that are not compatible with gettext pattern. Keys can’t contain spaces and are limited to ASCII or something alike, so that we can’t put text in a key.

So instead of using yet another specific format, I’m suggesting here to use JSON. JSON is really easy to parse and generate from both client and server side, and I’m convinced that it is simple enough to be edited with a text editor. On top of that we can build a small web application to ease localization.

In my very first proposal, I used a complex JSON object with nested attributes. But it ends up complexifying the whole story without real advantage. So I’m suggesting now to use the most simple JSON file we can require: one big object with keys being strings or id to translate and values being translated strings. Then we will be able to use JSON features to implement complex localization features, like plurals handling. So that values may be an array of plurals forms.

The big picture

Everything starts with one addon developer or one of its contributor. If one of them want to make the addon localizable, they have to use this new localization module.

const _ = require("l10n").get;

There is already multiple choices that has been made here:

_ is not a magic global. We need to explicitely require it. This choice will simplify compatibility with other CommonJS environnements, like NodeJS.
The name of the module itself is l10n instead of localization in order to ease the use of it.
This module expose _ function on get attribute in order to be able to expose another methods. I’m quite confident we will need some functions for plurals or files localization.

Then, they need to use _ on localizable strings:

var cm = require("context-menu");
cm.Item({
  label: _("My Menu Item"),
  context: cm.URLContext("*.mozilla.org")
});

Now, they have two choices:

use a string written in their prefered language, like here. So that they don’t have to create a locale file.
use an ID. Instead of _("My Menu Item"), we will use: _("contextMenuLabel"). But it forces to create a localization file in order to map contextMenuLabel to My Menu Item.

Then, either a developer or a localizer can generate or modify locales files. Each jetpack package can have its own locale folder. This folder contains one JSON file per supported language. Here is how looks like a jetpack addon:

* my-addon/
  * package.json   # manifest file with addon name, description, version, …
  * data/          # folder for all static files
    * images, 
    * html files, 
    * …
  * lib /          # folder that contains all JS modules:
    * main.js      # main module to execute on startup
    * my-module.js # custom module that may use localization module
    * …
  * locale/       # our main interest!
    * en-US.json
    * fr-FR.json
    * en-GB.json
    * …

The next iteration will add a new feature to our command line tool, that is going to generate or update a locale file for a given language by fetching localization strings from source code. For example, the following command will generate my-addon/locale/fr-FR.json file:

$ cfx fetch-locales fr-FR

my-addon/locale/fr-FR.json

{
  "My Menu Item": "My Menu Item"
}

Finally, we need to replace right side values with the localized strings:

{
  "My Menu Item": "Mon menu"
}

And build the final addon XPI file with:

$ cfx xpi

Any kind of feedback would be highly appreciated on this group thread.

If you want to follow this work, subscribe to bug 691782.