Tips on internationalization of software

Introduction

Internationalization or globalization of software is the process of developing a program core whose feature design and code design don't make assumptions based on a single language or locale and whose source code base simplifies the creation of different language editions of a program. The aim of this article is to provide solution for internationalization of existing software. The solution uses Windows 2000 as the platform, Visual Studio and Visual C++ for implementing the solution.

Keywords

Character set: Each written language uses a specific set of symbols. Character set is an encoding of one or more symbol sets into numbers that can be manipulated by computer hardware and software. Examples of character set are EBCDIC, ASCII.
Code page: An ordered set of characters in which a numeric index (code point value) is associated with each character. Example of code pages is Code Page 437 or CP437

SBCS: SBCS is 8 - bit character encoding, which is sufficient to represent ASCII character set.

MBCS: MBCS also called double byte character (DBCS). MBCS is a variable bit character encoding. Each character in MBCS is either one byte or two bytes. In character set some range of byte is set aside as lead byte. A lead byte specifies that it and following trail byte comprise a single two byte wide character.

Unicode: Unicode is 16 bit character encoding.

Locale: A locale is a set of rules-and-data specific to a given language and geographic area. These rules and data include information on:

Character classification,

Date and time formatting,

Numeric, currency, weight and measure conventions, and

Sorting rules.

Overview

Change the project setting so that the entry point of the program is changed and

Unicode

is defined. Collect string literals from the source code and add the strings that need to be localized to the string table in the resource of the project. The strings that do not need to be localized are handled in different way. Use macros defined in

tchar.h

file to handle string variables. Replace functions like

strcmp

by

_tcscmp

. The functions starting with

_tcs

are macros, which gets mapped to the functions starting with

str

if

_UNICODE

and

_MBCS

is not defined or gets mapped to functions starting with

_mbs

if

_MBCS

is defined or gets mapped to functions starting with

wcs

if

_UNICODE

is defined. To localize user interface make a copy of the resource change the English text to the localized text by using cut and paste mechanism. Then localized date, time and number formats. Short cut keys are also to be localized because of changing keyboard patterns for different locale.

Changing the project setting

Set

wWinMainCRTStartup

as the Entry Point symbol in the Output category of the Link tab in the Project Settings dialog box. This will make

wWinMain

as entry point as MFC Unicode applications use

wWinMain

as the entry point.



Set

_UNICODE

and

UNICODE

as the preprocessor definitions in the General category of the C/C++ tab in the Project Settings dialog box. This will define

UNICODE

and

_UNICODE

in the project

Handling string literals

The Visual C++ compiler interprets a literal string coded as

L"this is a literal string"

to mean a string of Unicode characters. Use the

_T

macro to code literal strings generically, so they compile as Unicode strings under Unicode or as ANSI strings (including MBCS) without Unicode. For example, instead of:

pWnd->SetWindowText( “Hello” );

use:

pWnd->SetWindowText( _T(“Hello”) );

With

_UNICODE

defined,

_T

translates the literal string to the L-prefixed form; otherwise,

_T

translates the string without the L prefix.

String and number literals that need to be localized are stored in

STRINGTABLE

resource by adding them to the applications

.rc

file. For each locale a copy of

STRINGTABLE

is made and the caption of the strings is changed to contain the localized or translated string. Then using LoadString function the string literal is loaded into memory. The value of string loaded depends on the thread locale.

Handling string variables

Use macros defined in

tchar.h

that will expand depending upon

_UNICODE

and

_MBCS

symbols. Following is the list of macros and their expansion when

_UNICODE

and

_MBCS

is defined.

Macro	Meaning	Expansion when _MBCS is defined	Expansion when_UNICODE defined
_TCHAR	Character	Char	wchar_t
_TSCHAR	Signed character	signed char	wchar_t
_TUCHAR	Unsigned character	unsigned char	wchar_t
_TXCHAR	Unsigned character	unsigned char	wchar_t
_TINT	Integer	unsigned int	wint_t
_TEOF	End of file	EOF	WEOF

Use “portable” run-time functions.

TCHAR.H

defines macros prefixed with

_tcs

, which, with the correct preprocessor definitions, map to

str, _mbs, or wcs

functions as appropriate.

To check upper or lower case character use

IsCharLower

and

IsCharUpper.

To convert from lower case to upper case use

CharUpper

and

CharUpperBuff.

To convert from upper case to lower case use

CharLower

and

CharLowerBuff.

For string comparison use

CompareString.

For string conversion use

MultiByteToWideChar, WideCharToMultiByte, LCMapString and FoldString.

Localization of user interface resource

Edit

.rc

file of the project by adding a copy of a resource for particular language and mention the proper code page number. Following is the extract from a resource file for English resource.

/////////////////////////////////////////////////////////////////////////////

// English (U.S.) resources

#if !defined(AFX_RESOURCE_DLL) || defined(AFX_TARG_ENU)

#ifdef _WIN32

LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_US

#pragma code_page(1252)

#endif //_WIN32

Each resource is placed in a different language section of the resource database, using

LANGUAGE

keyword. The

pragma

statements specify the code pages that the resource compiler should use to convert the string into

Unicode

. This code page is used by the run time system to display the characters.
Using cut and paste mechanism change the title of the window and controls in the resource editor.

Localization of date

The

GetDateFormat

function formats a date as a date string for a specified locale. The function formats either a specified date or the local system date.

int GetDateFormat(
LCID Locale,               // locale for which date is to be formatted
DWORD dwFlags,             // flags specifying function options
CONST SYSTEMTIME *lpDate,  // date to be formatted
LPCTSTR lpFormat,          // date format string
LPTSTR lpDateStr,          // buffer for storing formatted string
int cchDate                // size of buffer
);

The first argument specifies the locale for which the date string is to be formatted. If

lpFormat

is

NULL

, the function formats the string according to the date format for this locale. If

lpFormat

is not

NULL

, the function uses the locale only for information not specified in the format picture string

GetDateFormat

accepts date information as a

SYSTEMTIME

structure whose pointer is passed as the third argument. If

NULL

is passed as third argument the function use the current system date.

The second argument specifies the formatting style using symbols defined in

winnls.h

. These are

DATE_SHORTDATE, DATE_LONGDATE and DATE_YEARMONTH.

Sometimes you need to deviate from the default date formats. In those cases, you must either fetch an alternative format string from the locale database or construct a format string and pass it as fourth parameter. This format string is often called a picture string. It can contain the codes listed below:

Picture code	Description
d	Day of month as digits, with no leading zero for single digit days
dd	Day of month as digits with leading zero for single digit days.
ddd	Day of week as three letter abbreviation
dddd	Day of week as full name
M	Month as digits with no leading zero for single digit months.
MM	Month as digits with leading zero for single digit months
MMM	Month as three letter abbreviation
MMMM	Month as full name
y	Year as last two digits with no leading zero for years less than 10
yy	Year as last two digits with leading zero for years less than 10
yyyy	Year as four digits
gg	Era or period string

Localization of time

Win32 provides two functions for working with time formats:

GetTimeFormat

and

EnumTimeFormats.

The

GetTimeFormat

function formats a time as a time string for a specified locale. The function formats either a specified time or the local system time.

intGetTimeFormat(
LCID Locale,       // locale for which time is to be formatted
DWORD dwFlags,             // flags specifying function options
CONST SYSTEMTIME *lpTime,  // time to be formatted
LPCTSTR lpFormat,          // time format string
LPTSTR lpTimeStr,          // buffer for storing formatted string
int cchTime                // size, in bytes or characters, of the buffer
);

The first parameter

Locale

Specifies the locale for which the time string is to be formatted. If

lpFormat

is

NULL

, the function formats the string according to the time format for this locale. If

lpFormat

is not

NULL

, the function uses the locale only for information not specified in the format picture string.
The fourth argument of

GetTimeFormat

is an optional picture string using the codes shown below:

Picture code	Description
h	Hours with no leading zero for single digit hours; 12 hour clock
hh	Hours with leading zero for single digit hours; 12 hour clock
H	Hours with no leading zero for single digit hours; 24 hour clock
HH	Hours with leading zero for single digit hours; 24 hour clock
m	Minutes with no leading zero for single digit minutes
mm	Minutes with leading zero for single digit minutes
s	Seconds with no leading zero for single digit seconds
ss	Seconds with leading zero for single digit seconds
t	Single character time marker string such as A or P
tt	Multi character time marker string such as AM or PM

If you supply null pointer the function uses the

LOCALE_STIMEFORMAT

item from specified locale.

GetTimeFormat

also enables you to omit or adjust parts of the formatted output by specifying the appropriate flags in its second argument as shown below:

Style	Description
TIME_NOMINUTESORSECONDS	No minutes or seconds
TIME_NOSECONDS	No seconds
TIME_NOTIMEMARKER	No AM/PM marker
TIME_FORCE24HOURFORMAT	24-hour time format

The third argument is pointer to a

SYSTEMTIME

structure that contains the time information to be formatted. If this pointer is

NULL

, the function uses the current local system time

Localization of number format

The

GetNumberFormat

function formats a number string as a number string customized for a specified locale.

int GetNumberFormat(
LCID Locale,                // locale for which string is to be formatted
DWORD dwFlags,              // bit flag that controls the function's operation
LPCTSTR lpValue,            // pointer to input number string
CONST NUMBERFMT *lpFormat,  // pointer to a formatting information structure
LPTSTR lpNumberStr,         // pointer to output buffer
int cchNumber               // size of output buffer
);

The

GetNumberFormat

function punctuates a numeric string according to the rules of a specific locale. The first argument specifies the locale for which the number string is to be formatted. If

lpFormat

is

NULL

, the function formats the string according to the number format for this locale.
The second argument contains a bit flag that controls the operation of the function. If

lpFormat

is

non-NULL

, this parameter must be zero. If

lpFormat

is

NULL

, you can specify the

LOCALE_NOUSEROVERRIDE

flag to format the string using the system default number format for the specified locale; or you can specify zero to format the string using any user overrides to the locale's default number format.

The third argument is a pointer to input string. The input string should contain digit characters from 0 through 9 with a leading minus sign if the number is negative and a single decimal point if the number has a fractional part. The presence of any other characters causes the function to return 0, in which case you can call

GetLastError

to get more details. If the function succeeds, it returns the number of characters produced in the output buffer.

GetNumberFormat

replaces the minus sign and decimal point with characters specified by the locale database. It also rounds the fractional part to the appropriate number of places and inserts grouping characters in the integer part.
If you want to use a format other than the locales default you must construct a

NUMBERFMT

structure which has following definition:

typedef struct _numberfmt
{
UINT NumDigits; //From LOCALE_IDIGITS
UINT LeadingZero; // From LOCALE_ILZERO
UINT Grouping; //From LOCALE_SGROUPING
LPTSTR lpDecimalSep; //From LOCALE_SDECIMAL
LPTSTR lpThousandSep; //From LOCALE_STHOUSAND
UINT NegativeOrder; //From LOCALE_INEGNUMBER
}NUMBERFMT;

The structure members correspond directly to the locale items mentioned in the comments, except for the grouping member, which should be a value from 0 through 9 if all groups are the same size.

Handling shortcut keys

Shortcut keys also called accelerators provide keyboard alternatives for menu and toolbar actions. For instance, Ctrl + A and Ctrl + Z are the conventional shortcuts for Select All and Undo commands, respectively. For instance, if you compare the U.S. and French keyboards you will see that the A and Z keys switch places with Q and W.
The solution for handling shortcuts keys is as follows:

During initialization, call

GetUserDefaultUILanguage

to obtain the default UI language for the current user. Still during initialization call

LoadKeyboardLayout

to get the HKL for the default input locale.

Within the message loop, check whether the current message is from the keyboard. If so, get the current input locale. If the input locale is using a different language than the default UI call

MapVirtualKeyEx

to map the virtual key code back to the default input locale.

Call

TransalateAccelerator

in the normal way. If it returns

FALSE

, restore the original virtual key code in the message so that normal keys are processed in the current language.

MFC and similar Windows programming environments usually bury the main message loop deep within the library, but they also provide “hook” so that you can perform special operation, such as UI sensitive shortcut mapping. MFC offers two virtual functions named PreCreateWindow and PreTranslateMessage for this purpose. Use PreCreateWindow function to contain step 1 and PreTranslateMessage to contain step 2 and 3.

Handling message resource

FormatMessage

API function converts error codes into localized messages for display to the user. The win32 messages are stored in multilingual resources attached to the various system components or in system resource DLLs. Use message compiler to define a repertoire of message codes and their localized strings. This compiler is a Visual Studio utility program. MC accepts a message script file and produces header file, binary file and single resource script. The header file contains the

#define

symbols for the message codes. The BIN files contain the tables that correlate message codes to text strings for the various languages.
The

FormatMessage

function formats a message string. The function requires a message definition as input. The message definition can come from a buffer passed into the function. It can come from a message table resource in an already-loaded module. Or the caller can ask the function to search the system's message table resource(s) for the message definition. The function finds the message definition in a message table resource based on a message identifier and a language identifier. The function copies the formatted message text to an output buffer, processing any embedded insert sequences if requested.

DWORD FormatMessage(
DWORD dwFlags,      // source and processing options
LPCVOID lpSource,   // pointer to  message source
DWORD dwMessageId,  // requested message identifier
DWORD dwLanguageId, // language identifier for requested message
LPTSTR lpBuffer,    // pointer to message buffer
DWORD nSize,        // maximum size of message buffer
va_list *Arguments  // pointer to array of message inserts
);

Working of software

The locale settings of the operating system can be change from control panel. Open control panel click on Regional options open General tab and then click on Set default button on General tab. select the system locale. Click on Apply button and restart the computer. If necessary change the input locale. To change the input locale click on Input locales tab. Select one of the installed input locale and the click on Set as default button. Click on Apply button.
The dialog and string table resource in different languages are embedded in the image of the executable. When the software is started depending upon the locale settings of the operating system the operating system loads the particular resource and displays the resource. The number formatting, date and time formatting also depends upon the locale settings of the thread, which can be changed at run time.

Pros

The localized strings reside in the image of the executable so there is no need to maintain an external database.

Operating system is responsible for loading the language specific resource

Cons

Cut and paste mechanism is used to changed to text of windows and controls this may introduce errors. Also, the programmer who will do cut and paste may not have the knowledge of the language, so he/she will not able to correct the errors.

When new language needs to introduce, there is a massive change in the source code. The programmer will have to change the title of the windows and controls and make changes in the string table.

The choice of the language is done through the locale setting of the operating system. There is no way to change the language setting during runtime.

Unable to localize common dialogs like File dialog, color dialog, find replace dialog, font dialog and print dialog. These dialogs are part of the Windows common dialog library (COMMDLG.DLL). The solution to this problem is to develop our own dialogs which shall support Unicode and which are able to localize.

Milind Shingade

Click here to view Milind Shingade's online profile.