MozLCDB: Mozilla Locale Database

MozLCDB: What is it?

MozLCDB is a tool made for localization of various Mozilla (Gecko) based products. It is designed to resolve issues introduced since Mozilla Firefox 1.0 branch, and a replacement for MozillaTranslator.

The author and "Why a new tool?"

  This is Hung-Te Lin, maintainer of Traditional Chinese Mozilla Localization, and the famous "mozip" (a tool to make Win32 localized builds).

  I have been using MozillaTranslator for years (whether the official builds or my own tweaked versions) since Mozilla M14 (Milestone 14, not 1.4. The M series are older than Mozilla 0.6). During these years MT works O.K although more and more extra works have to be done because Mozilla locale structure changes all the time.

  Since the landing of Mozilla Firefox 1.0PR, localizers of Mozilla* suffered from the huge changes in Mozilla locale structure and new CVS requirements. Unfortunately, MozillaTranslator does not work anymore. Although MT has released new versions to solve issues, after trying MT5.04 I think it's time to develop a new tool which is more robust or I have to manually do many extra work for all versions. With my experience in localization, I proposed what I need and made a new tool, named MozLCDB. This name is a little weird because the better name 'MLDB' is already taken by others. Hmmm. Maybe MLD?

  I'm not an perl guru so this script is not perfect nor elegant now, but it helped me to generate a working Firefox 1.0PR locale without too much effort (in comparision to fight with MozillaTranslator).

Features of MozLCDB

MozLCDB is focused to Mozilla Localization. In comparsion to MozillaTranslator, you'll notice that:

Requirements

You need these stuff:

Download

Get these files:

Background Knowledge

This section explains the basic idea about MozLCDB. It helps you to understand it more although you may use it without knowing anything explained here. Yes, you can go to "Usage:" sections if you are in a hurry. However it is highly recommended to get things clear before you start.
Workflow
   I'm trying to explain the workflow of MozLCDB here. Don't worry about the terms quoted, they will be explained in following subsections.

   Basically, MozLCDB works in this manner: MozLCDB tries to 'import' from a 'prepared directory' (pulled from CVS or ZIP language pack), then save and merge into a '(glossary) database'. If any new locale string or updated ones are found, these entries will be written to an 'input (editing) table'. You can then edit the input table to do localization. After everything is done, you have to 'update' the '(glossary) database' with your 'input (editing) table'. Then you can 'export' the locales to another 'prepared directory'. Finally, the 'prepared directory' will contain all the localized stuff.

File Format and Default Filenames
   MozLCDB tries to record everything in a text file named 'mozlcdb.txt'. It is saved in perl Dump format, however it is still readble ( if you know a little perl) and also called as the 'glossary' (due to MT) or 'database'. And there's another format of input that MozLCDB can read: the input (editing) table. It is more human readble and looks like DOS/WIN INI or GNU PO format. The default file name of this kind file is 'current.txt'. When importing from and exporting to Mozilla locale directories, currently MozLCDB recognizes only DTD and ".properties" files.
Import, Export, and Update
import means to read locales from Mozilla files into the database.
export means to write locales from database into real Mozilla files.
update means to read locales from an input (editing table) into the database.
Directory Structure
   Mozilla locale is stored in a hierarchical directory structure, also refered as chrome node. So you will see a very long path like "locale/en-US/global/global.dtd" in the directory you pulled from CVS or extracted from a language pack. MozLCDB mostly runs with the level just after where 'en-US' sits, so whether you're working with CVS or language pack, you must move directories (or move mozlcdb program and files) and assign correct ROOT path to fit thie rule. Then it can be called as a 'prepared directory'. (P.S: Actually, you can run in any level of directories. But you may get problems if you work in that way.)
ROOT
   What's the ROOT? It's a PATH to help MozLCDB find corresponding locale path. For example, if my current directory is D:\Mozilla\LC (Win) or /home/moz/lc (UNIX) and my locale is in full path of D:\Mozilla\LC\locale\en-US\global\... or /home/moz/lc/locale/en-US/global/..., then you have to use ROOT as locale/en-US.
The procedure of 'Import'and Multiple Product
   What really happened when you 'import'? MozLCDB will scan and walk through all files inside the 'prepared directory' assigned by ROOT. Currently, only *.dtd and *.properties will be processed. All other files will be ignored.

   When one DTD or properties file was found, MozLCDB first reads its content for the 'key's and original locales. If a key was not found (never seen), it will be inserted into database as a 'new' entry. If a key was found, MozLCDB will determine what to do depending on the original (en-US) locale message. If it is exactly same as some version recorded in database, it is considered a 'same' entry with no new entries introduced. If no entries with same original could be found, MozLCDB will append one entry with timestamp into database. This is the magic to make multiple product possible. We keep all versions of original messages.

The procedure of 'Export'
   What really happened when you 'export'? MozLCDB will scan and walk through all files inside the 'prepared directory' assigned by ROOT. Currently, only *.dtd and *.properties will be processed. All other files will be ignored.

   When one DTD or properties file was found, MozLCDB first reads its content for the 'key's and original locales, then try to lookup database to decide what localized string to put in. If you export with '-x', MozLCDB assume you are working with a clean original copy exactly just what you have imported, so it will check for existence of keys and original locale message. However, you may want to export to a working (trial) copy. In that usage, -x will warn and stop because original message is filled with your localized strings. In this case you may use '-X' which will not check for keys.

Usage: Syntax Overview

Usage: Import from MozillaTranslator

If you don't need to import from MozillaTranslator, skip this section.
You must have python working before you start the procedure.


I know this is weird: why python, not perl? Because this python script is modified by another python script I wrote years ago and I'm tired of making another perl version.

Usage: Import from an already localized language pack

If you have an already working language pack, say aa-BB, you must get its matching version of en-US package or importing may be useless.

Usage: Import from CVS directory

Remember to backup your glossary database (mozlcdb.txt) before you import because you can rollback if anything goes wrong.

Usage: Import from ZIP(JAR) language pack

Remember to backup your glossary database (mozlcdb.txt) before you import because you can rollback if anything goes wrong.

Usage: Working with input(editing) table

The input(editing) table is named 'current.txt'. You should see it after you imported something (except importing from MozillaTranslator or already localized language pack).
The format is very simple. Take this for example: Keep in mind that you should use UTF-8 file encoding.

Usage: Update database with an input table

When you finished editing, you have to update database. This is the default action if you give no arguments to MozLCDB.

Usage: Export database to CVS

Before you export, you have to request for CVS account. Although you may already checked in your locale module, for example aa-BB, but we're not going to use it. Please review the section of 'Background Knowledge' for 'The procedure of Export'. You have to use the original dtd and properties files from en-US.

Usage: Export database to a standalone language pack, or JAR files

Before you start, there's something you have to know. Currently we don't have a really good method for changing locale in Firefox (maybe we will never do), and the bug of "using extensions without corresponding locale leads to crashing" is still there (maybe will be solved in 1.0 release), so trying to provide language packs for end user is really a mess.

The better way is to export to CVS files and let mozilla.org build a official localized build. And if you want to make end user happy, due to the issues I mentioned above, here I described is kind of 'hacked' language pack. I'll export files, and make them with en-US chrome node path, and save to an 'en-US.jar' to replace the original en-US.jar.

This is NOT GOOD, but seems like making most end user happy. If you don't want to make such evil language packs, you have to go manually edit every *.rdf in directories, and prepare install.rdf or install.js.

OK... But if you are happy with evil packs, or you're just trying your newly localized stuff (this is mainly what I use this feature now for) , here's how to make it work:

Usage: Things left

Some commands are not explained in previous sections.

Usage: Tips

Epilogue

This program is still under development. I used it sucessfully to make a working language pack (well, the en-US.jar) for Mozilla Firefox 1.0PR, Traditional Chinese. I'm not sure whether it is easy enough for most people or not, but it is designed for all what I need. So here I present it to all who needs to localize new Mozilla products but could not find any working tools to go on.

If you have any suggestions/problems, feel free to contact me. You can also use mozilla-l10n mailing list.

Copyright

MozLCDB is licensed under MPL 1.1 . This program and document is made by Hung-Te Lin. Allrights reserved.