i18nchecker (Part 1)

i18nchecker is a tool resulting from the discussion inspired by a trip to Experian in Monaco (where a risk management system on the NetBeans Platform is created) "One Big Bundle File For Internationalization" in this blog, from 1 December 2010. In that discussion, Tim Dudgeon from ChemAxon, where the same kind of requirements exist for Instant JChem, a NetBeans Platform application, comments:

 

We have worked things so that we can generate an Excel file with all the strings in that we can generate, send for translation, and then re-incorporate the translations back into the codebase. The code is quite involved, but we'd be happy to share if there is interest.

And now, as hoped for, the code referred to above has been separated by Petr Hamernik, who is from Tim Dudgeon's team, and can be found at:

https://github.com/phamernik/i18nchecker.

The project is a small subset from Instant JChem, focused on 4 specific tasks: print all i18n errors (e.g., unused bundle keys), produce a csv file for translation, integrate a translated csv file back into the application, and provide unit tests that prevent i18n regressions.

When you access everything from the git repo, you'll find you'll have, together with the sources and related libs, a 'playground':

The playground is a demo environment that you can use to try out the i18nchecker.

The build.xml file that you see in the screenshot above has these targets, which have been set up to work with the playground environment:

<target name="i18n-consistency-check"  description="Verification of localized strings in source code" depends="jar">
    <property file="nbproject/private/private.properties"/>
    <property name="i18n.modulefilter" value=""/>
    <echo>I18N Consistency Check - use modulefilter property if you want to see errors only from one module</echo>
    <taskdef classname="org.i18nchecker.I18nChecker" name="i18nConsistencyCheck" classpath="lib/anttasks.jar:lib/antlr-runtime-3.2.jar:dist/i18nchecker.jar"/>
    <i18nConsistencyCheck srcdir=".." topdirs="i18nchecker/playground,i18nchecker/playground/PaintApp" modulefilter="${i18n.modulefilter}"/>
</target>

<!-- TODO: improve following tasks - reuse taskdef etc. -->
<target name="i18n-prepare-japanese" description="Prepare CSV file for translation to Japanese">
    <taskdef classname="org.i18nchecker.I18nChecker" name="i18nConsistencyCheck" classpath="lib/anttasks.jar:lib/antlr-runtime-3.2.jar:dist/i18nchecker.jar"/>
    <mkdir dir="build/i18n"/>
    <i18nConsistencyCheck srcdir=".." topdirs="i18nchecker/playground,i18nchecker/playground/PaintApp" language="ja" exportto="build/i18n/japanese.csv"/>
</target>

<target name="i18n-apply-japanese" description="Apply Japanese translation to projects resource bundles">
    <taskdef classname="org.i18nchecker.I18nChecker" name="i18nConsistencyCheck" classpath="lib/anttasks.jar:lib/antlr-runtime-3.2.jar:dist/i18nchecker.jar"/>
    <i18nConsistencyCheck srcdir=".." topdirs="i18nchecker/playground,i18nchecker/playground/PaintApp" language="ja" importfrom="translations/japanese.csv"/>
</target>

Note 1. The "japanese" references above could be anything. In this case, the targets are named "japanese" since that is the language to which Instant JChem is localized.

Note 2. The "playground" is the top level dir for "module1" and "module2", while "PaintApp" is the top level dir for "Paint" and "ColorChooser".

Note 3. The "modulefilter" attribute is where you can, optionally, put a module name (or part of the name). Then only the modules with names where the filter is a substring of the module name are processed. It is useful for when you have thousands of errors in many modules and you want to fix errors only in one module. Put e.g., "modules/DIF" into the module filter property (in private.properties) and then the consistency check task prints only the errors for this particular module. Now you can quickly iterate, i.e., run task | see output | click on errors in Output window | open sources | fix | save and then in the next round the fixed errors disappear from the output.

When I run the first target, the Output window shows me this (the lines in bold are made bold by me, to highlight the type of error checking that is done):

Module: C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\ColorChooser
Scanned 0 Java sources, 1 primary and 0 translated resource bundles. Found 1 potential problems.
Module's resource bundle specified in manifest.mf
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\ColorChooser\\src\\org\\netbeans\\swing\\colorchooser\\Bundle.properties:1: Missing OpenIDE-Module-Display-Category NetBeans module bundle

Module: C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint
Scanned 2 Java sources, 1 primary and 1 translated resource bundles. Found 12 potential problems.
Probably missing key in resource bundle or string should be marked with // NOI18N
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:107: print.printable
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:191: .png
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:192: .png
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:219: png
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:234: version
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:234: 1.0
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:236: color
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:237: size
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:241: version
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:243: color
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\PaintTopComponent.java:244: size
Module's resource bundle specified in manifest.mf
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\PaintApp\\Paint\\src\\org\\netbeans\\paint\\Bundle.properties:1: Missing OpenIDE-Module-Display-Category NetBeans module bundle

Module: C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\module1
Scanned 1 Java sources, 1 primary and 1 translated resource bundles. Found 3 potential problems.
Very likely missing key in resource bundle
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\module1\\src\\org\\i18nchecker\\test1\\TestA.java:19: Key_missing
Probably missing key in resource bundle or string should be marked with // NOI18N
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\module1\\src\\org\\i18nchecker\\test1\\TestA.java:9: should be localized
Probably unused resource bundle
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\module1\\src\\org\\i18nchecker\\test1\\Bundle.properties:8: KEY_NotUsed

Module: C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\module2
Scanned 1 Java sources, 0 primary and 0 translated resource bundles. Found 1 potential problems.
Probably missing key in resource bundle or string should be marked with // NOI18N
C:\\Documents and Settings\\gwielenga\\My Documents\\NetBeansProjects\\I18N\\i18nchecker\\i18nchecker\\playground\\module2\\src\\org\\i18nchecker\\test2\\TestB.java:9: should be localized
Summary:

PaintApp/ColorChooser=1
PaintApp/Paint=12
playground/module1=3
playground/module2=1

total=17

When I run the second target, a csv file with this content is created in "build/i18n/japanese.csv" (the first line below shows the headers for the subsequent lines, hence, for clarity I have highlighted the first line):

"Variable name","English","Translation","Module name","Package"
"OpenIDE-Module-Name","ColorChooser",,"PaintApp/ColorChooser","org/netbeans/swing/colorchooser"
"CTL_NewCanvasAction","New Canvas",,"PaintApp/Paint","org/netbeans/paint"
"LBL_BrushSize"," Brush Size",,"PaintApp/Paint","org/netbeans/paint"
"LBL_Clear","Clear",,"PaintApp/Paint","org/netbeans/paint"
"LBL_Foreground"," Foreground",,"PaintApp/Paint","org/netbeans/paint"
"MSG_Overwrite"," {0} exists.  Overwrite?",,"PaintApp/Paint","org/netbeans/paint"
"MSG_SaveFailed"," Could not write to file {0}",,"PaintApp/Paint","org/netbeans/paint"
"MSG_Saved"," Saved image to {0}",,"PaintApp/Paint","org/netbeans/paint"
"OpenIDE-Module-Name","Paint",,"PaintApp/Paint","org/netbeans/paint"
"TTL_SAVE_DIALOG","Save",,"PaintApp/Paint","org/netbeans/paint"
"UnsavedImageNameFormat","Image {0}",,"PaintApp/Paint","org/netbeans/paint"
"KEY_NotUsed","This string is not used in Java sources",,"playground/module1","org/i18nchecker/test1"
"Key_correct","This string is in resource bundle",,"playground/module1","org/i18nchecker/test1"
"NAME_key","Another string in resource bundle",,"playground/module1","org/i18nchecker/test1"

In other words, now there is one big file with all of the keys in the application, ready to be translated.

Now, I imagine that the csv file has been translated. (Petr used Google translate for the below. Remember, again, that "Japanese" is just an example here.)

"Variable name","English","Translation","Module name","Package"
"OpenIDE-Module-Name","ColorChooser",,"PaintApp/ColorChooser","org/netbeans/swing/colorchooser"
"CTL_NewCanvasAction","New Canvas",,"PaintApp/Paint","org/netbeans/paint"
"LBL_BrushSize"," Brush Size",,"PaintApp/Paint","org/netbeans/paint"
"LBL_Clear","Clear","クリア","PaintApp/Paint","org/netbeans/paint"
"LBL_Foreground"," Foreground","前景","PaintApp/Paint","org/netbeans/paint"
"MSG_Overwrite"," {0} exists.  Overwrite?",,"PaintApp/Paint","org/netbeans/paint"
"MSG_SaveFailed"," Could not write to file {0}",,"PaintApp/Paint","org/netbeans/paint"
"MSG_Saved"," Saved image to {0}",,"PaintApp/Paint","org/netbeans/paint"
"OpenIDE-Module-Name","Paint",,"PaintApp/Paint","org/netbeans/paint"
"TTL_SAVE_DIALOG","Save",,"PaintApp/Paint","org/netbeans/paint"
"UnsavedImageNameFormat","Image {0}","画像{0}","PaintApp/Paint","org/netbeans/paint"
"KEY_NotUsed","This string is not used in Java sources",,"playground/module1","org/i18nchecker/test1"
"Key_correct","This string is in resource bundle","この文字列は、リソースにバンドルされ","playground/module1","org/i18nchecker/test1"
"NAME_key","Another string in resource bundle",,"playground/module1","org/i18nchecker/test1"

The above file is in "translations/japanese.csv". When I run the third target, I get a "Bundle_ja.properties" file in each module where a "Bundle.properties" file with keys for translation was found, which (in the case of the Paint module) has this content:

LBL_Clear=クリア
LBL_Foreground=前景
UnsavedImageNameFormat=画像{0}

Finally, the i8nchecker provides a JUnit test. The test checks how many i18n errors you have in your code, compares it with a predefined number, and fails if there are more errors than the predefined number. About this, Petr writes:

 

We started with several thousand i18n errors half a year ago. I just wanted to ensure that we were not adding new i18n errors. So I created a golden file which contains current numbers, which means current count of i18n errors per module. This was executed as part of each build on Hudson, so whenever someone added a new string which should be localized, and NOI18N comment was missing, the test failed. About once a week I was regenerating this "golden file" with current numbers. If someone fixed a few i18n errors, the file was regenerated with lower numbers. Thanks to this, we went slowly down with errors. In other words, improvements were allowed, but any regressions were causing the unit test to fail. In January, we put in some effort there and so now the i18n error numbers went to zero, which means that now we don't have any known i18n error in our sources and the golden file contains just total=0. The unit test does exactly the same as the "consistency check" target, i.e., it collects the number of errors but then it compares that number with a "golden file" and fails if the current numbers are worse (higher) than in the golden file.

To use the above functionality, go to the Test Packages folder and take note of the "i18n_known_error.properties" file, which has this example content:

PaintApp/ColorChooser=1
PaintApp/Paint=12
playground/module1=3
playground/module2=1

total=17

About the above file, Petr says:

 

The unit test compares the numbers per module, which means that when someone fixes one known i18n error in one module and introduces a new error in another module, the test will fail anyway. The total=17 above is mostly for our info, since the unit test works per module.

Now, isn't this a great project. There's also a JAR you can download, if you don't want the sources. Of course, not for NetBeans Platform applications only, though it is tailored for the Instant JChem sources which have a "modules/src" structure, while everything that is not under "src" is ignored. However, since you have the sources available, you can tweak them to fit your project's needs.

Feedback welcome, of course, and it would be nice if anyone using this project would mention that (on Twitter or in their blog or somewhere). Thanks Petr and Tim, fantastic resource that I know several NetBeans Platform developers (at least) are going to be very happy about.

Comments:

We have created a similiar solution, a custom maven plugin that can dump out all bundle files from a large multi module maven project into a mirrored project structure that can then be imported into OmegaT for translation.

OmegaT is pretty smart about translation and keeps already translated phrases so that a translation is helped by previous translations, even when bundle texts are not an exact match.

When a translation has been made we have another goal on the same maven plugin that can import the bundle texts back into our maven project.

In addition to this we also have got a goal for checking that there are no empty bundle strings, that a list of required locales does have bundle strings for all keys in the the default bundles.

The plugin is inhouse only right now, but if anyone is interested we might release it to the public some way.

Posted by Johan Andrén on March 01, 2011 at 04:18 PM PST #

Hi Johan, yes, that would be great, if you could share that Maven plugin with the world in some way! I'd definitely like to try it.

Posted by Geertjan on March 02, 2011 at 03:32 PM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Geertjan Wielenga (@geertjanw) is a Principal Product Manager in the Oracle Developer Tools group living & working in Amsterdam. He is a Java technology enthusiast, evangelist, trainer, speaker, and writer. He blogs here daily.

The focus of this blog is mostly on NetBeans (a development tool primarily for Java programmers), with an occasional reference to NetBeans, and sometimes diverging to topics relating to NetBeans. And then there are days when NetBeans is mentioned, just for a change.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
12
13
14
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today