Typing around the world

Chinese Comes to Keyman Desktop 8

We've just re-released our Simplified Chinese keyboard (aka CS Pinyin). It's now 64-bit compliant for Keyman Desktop 8.  We also have a version of this keyboard that can be used in your web browser or embedded into any website using KeymanWeb!

The Many Features of Simplified Chinese

If you're a Chinese scholar or student and you haven't tried Simplified Chinese yet, you should. Several years ago, in collaboration with Peter Hauer of Linguasoft, we designed the Simplified Chinese keyboard as an innovative multi-method input mapping solution to make Chinese simple for scholars.  It is now updated for Keyman Desktop 8.0, with support for Windows 7 and 64-bit.

The keyboard is packed with features including:

Input Features

• Four full Chinese input methods, with mapping for over a 100,000 single or multi-character sequences —
  1. Pinyin
  2. RAD-RSC (Radical/Rest Stroke Count)
  3. FCI (Four Corner Index)
  4. Unicode code point
• English direct input of more than 550 common names for historically important people and places (e.g. Beethoven › 贝多芬, Kuwait › 科威特).
• An input dictionary of over 440 Chinese family names.
• Two-letter abreviation input for key Chinese provinces, regions, and cities.
• Optional input of tone digits (1-4). 
• Wild-card input matching.

IMX Features

• Four separate character selection methods —
  1. Keyboard numerals
  2. Keypad
  3. Function keys
  4. Mouse
• Vertical or horizontal resizable IMX window.
• Optional cursor tracking.

Output Features

• Three distinct output methods —
  
  1. Han
  2. Pinyin
  3. Han + Pinyin
• Easy output of any single character in a multi-character sequence.
• Optional automatic output of unique matches.
• Full and half-width punctuation.

The Four Input Methods

Pinyin

It's easy to enter standard pinyin. Just start typing, with or without tone digits. Press enter to select the first character. Type a number, use the arrow keys, or point-and-click to enter a later character.

You can do the same for multi-character strings.

RAD-RSC

To enter characters by radical-rest stroke count, type '$' followed by the traditional (214) radical number followed by a '.' followed by the remaining stroke count. For example, to find and enter 央, you would enter '$37' for the radical 37 (大) + '.2' for the two additional strokes.

If you don't know the traditional number for the radical, the keyboard includes a simple look-up. Enter the number of strokes in the radical followed by a question mark to pull up the full list of radicals with that stroke count.

Four Corner

Use '#' to enter characters by the four corner method, typing the digits in z shaped order.

Unicode

Finally, any Unicode character, Chinese or otherwise, can be typed with the keyboard using the format 'U+XXXX' or 'U+XXXXX'. This greatly simplifies entry of obscure supplementary plane characters.

Get the keyboard

Download the Simplified Chinese keyboard for Keyman Desktop.

Use the Simplified Chinese keyboard in your web browser with KeymanWeb.

Upgrading the Tavultesoft website infrastructure to improve its Unicode support

Since 2004, the Tavultesoft website has had a reasonable degree of Unicode support.  It’s been a bit of a mixed bag, with limitations in technology choices at the time making it hard for us to achieve the level of Unicode support we wanted.  In particular, our database backend (Microsoft SQL Server) was not using Unicode columns, because our front end (PHP) did not have any stable drivers available for SQL Server that supported Unicode.  So we compromised, and stored our data in UTF-8 on what were supposed to be Windows-1252 “ANSI” columns.  This is not ideal.  But it got us over the hurdles and provided a level of Unicode support at the time.

The big risk with storing UTF-8 in a column intended for Windows-1252 codepage is that there are 5 codepoints unassigned in Windows-1252: 0x81, 0x8D, 0x8F, 0x90 and 0x9D – use of these codepoints could result in undefined behaviour.  But our tests showed that SQL Server didn’t care and just stored those bytes anyway.  Good enough for beggars.

Roll forward to today.  Microsoft now support a PHP SQL Server driver which, since version 1.1, has Unicode support via UTF-8 translation at the driver level.  Fantastic!  So how could we take advantage of this?  It turned out to be a lot less painful than we expected!

Step 1. Locate all the non-Unicode text columns

On our test server, we took a recent backup of the live database and started with that.  We analysed our database tables to find all VARCHAR, CHAR and TEXT columns.  Once we found these we deleted the indexes on these columns, including full text indexes.  There are many ways to find all the text columns in your database; probably the easiest is to query sys.columns for types 35, 167 and 175 (TEXT, VARCHAR and CHAR respectively).

Step 2. Convert the columns to Unicode equivalents

At this point we simply altered each column to its UTF-16 equivalent NVARCHAR, NCHAR or NVARCHAR(MAX).  Where a VARCHAR column had a size greater than 4000 characters, we opted to convert these into NVARCHAR(MAX) as 4000 characters is the maximum fixed column size for Unicode columns, as opposed to 8000 characters for non-Unicode columns.  For example, given a column customer.given_name VARCHAR(128) NOT NULL, we issued the following statement:

ALTER TABLE customer ALTER COLUMN given_name NVARCHAR(128) NOT NULL

This took the Windows-1252 data in the column and converted it to Unicode.  SQL Server of course did the job properly, for example, converting Windows-1252 0x80 (€ Euro sign) to the Unicode equivalent U+20AC.  Importantly, for us, with those 5 invalid bytes, SQL Server just zero-extended them and left them there.  These are considered control codes in Unicode, but we didn’t care, because we were about to fix them for good.

We did identify a number of columns that did not need to be widened, such as currency or country identifiers.  These we left as VARCHAR or CHAR columns accordingly.  The conversion process ran for 2 hours on the test server, but ran smoothly.

Step 3. Convert UTF-8 content into UTF-16

Now our data was in NVARCHAR columns, but it wasn’t right yet -- it was stored as UTF-8 via Windows-1252 in the UTF-16 native type (do you follow?  I'm not sure I do...).  Each UTF-16 string needed to be converted back to Windows-1252, and from there, read as UTF-8 (without conversion!), and then converting that UTF-8 back into UTF-16.  This was theoretically possible within SQL Server with some clever T-SQL code, but it turned out to be faster and easier to write a small app to do this transformation for each column.  We used a messy LIKE clause (imagine all the characters between 128 and 255 in a row) to reduce the number of records that needed editing. This was not perfect as LIKE does not do binary matches (and hence matched more rows than it needed to) but cut the conversion time dramatically -- many records could be excluded because they only contained ASCII characters.

At this point, we recreated the indexes.  Again, all this went smoothly on our test server.  Total time: 2 hours.

Step 4. Update T-SQL

We now needed to convert all our T-SQL data references into Unicode.  We dumped the full set of stored procedures, triggers, functions and more into a single massive text file, and used search and replace to find all instances of VARCHAR, TEXT, CHAR and also CHAR() and ASCII() function calls.  This covered all the cases that we used.  Despite seeming a huge task when we started, in the end it only took a couple of hours of careful replacing and checking to produce a monster T-SQL script.

When we ran this script on the test server, it all worked wonderfully and surprisingly quickly.  Total time: 2 hours of legwork, and 30 seconds of execution.

Step 5. Update PHP client code

OK, so our database was now all in Unicode, but PHP was still reading the data out and assuming it was UTF-8, which it no longer was.  So a change was needed to the sqlsrv_connect call:  Add the “CharacterSet” => “UTF-8” parameter.  That’s all -- everything else was already assuming UTF-8.  Total time: negligible.

With that we were done.  Running various tests on the test server showed everything working great!

Step 6. Dry run on live infrastructure

Time to do a dry run on the live server.  Unfortunately this didn’t go so well.  We ran the process on a copy of the live database, on the live infrastructure, as moving the database onto alternative infrastructure was going to mean a lot of downtime.  First problem: we ran out of memory.  Our live server actually has less memory than our test server!  Nothing crashed, but constant swapping meant that the conversion was going to take forever.  Then we started to run out of disk space.  Our 5GB test database blew out to 25GB during the conversion!

Of course, this is why we did a dry run.  During this process, the Tavultesoft website ran slightly slower than normal, but not so much that it was be a problem.  Our database server is normally quite lightly loaded…

It turned out that one particular log table was huge and also did not really need urgent conversion to Unicode.  So we opted to skip that table and ran the dry run again.  This time, things went a bit better and the whole process (steps 2, 3 and 4 really) completed in under 90 minutes.

Step 6. The real thing

Time to run live.  We took a backup of the database, and then switched off the front end of the site with a friendly message telling our users we’d be back soon, and started the process.  80 minutes later, it was all done.  We had to rebuild the full text indexes, and the upgrade was done!

Two (+ ½) New Ways to Type Korean

Today we’re proud to announce two new Korean packages for Keyman Desktop 8. 

Korean KORDA

The Korean KORDA keyboards are built to type Korean using the transliteration system KORDA (Korean Romanization for Data Application). The KORDA system is based on Korean pronunciation and follows a strict 1-to-1 principle of transliteration.

Because it's unambiguous and phonetic, the KORDA keyboard is an excellent choice for non-native speakers and new students of Korean. Unlike other common input methods (including Microsoft's Korean Standard Input), KORDA's phonetic input system makes learning to type Korean easy for new typists.

The Korean KORDA package comes with two keyboards which behave in the same way. The Jamo keyboard creates Korean hangeul from jamo each time you press spacebar. The Latin keyboard does the same thing with Latin letters. They have the same layout.

Download the Korean KORDA keyboards for Keyman Desktop 8.0

Korean Morse

The Korean Morse keyboard is built for the Morse transliteration system. The Morse system is not based on Korean pronunciation. It is an historical system for specialists, which pairs each Korean jamo with the Latin letter that uses the same Morse code value.

The Morse system has been popular among government and military organisations because it offers 100% data accuracy. However, the KORDA system also offers 100% data accuracy, and we recommend the KORDA system unless you have specific training in Latin-Korean Morse transliteration.

Download the Korean Morse keyboard for Keyman Desktop 8.0    

Command line arguments for Keyman Configuration

Keyman Desktop Configuration can be run with command line arguments to simply automate a number of tasks.  It is also possible to use the Keyman Desktop API to accomplish all these tasks, but some are more easily accessed via the command line.

The command line program is kmshell.exe, and if run with no arguments kmshell will start Keyman Desktop.  Otherwise, the following command line arguments are available:

Argument	Description
-c	Start Keyman Configuration
-i filename [-s]	Install a keyboard (.kmx) or package (.kmp) file (-s = silent)
-uk filename.kmx [-s]	Uninstall a keyboard (.kmx file) (-s = silent)
-up filename.kmp [-s]	Uninstall a package (.kmp file) (-s = silent)
-h	Open Keyman Desktop help contents
-w	Start Keyman Desktop tutorial
-t	Start Keyman Desktop text editor
-ouc	Check for updates online and prompt user if updates are found
-kw package	Show help for the installed keyboard package
-kp keyboard	Start print wizard for on screen keyboard for the keyboard
-activate key [-s]	Activate Keyman Desktop (-s = silent)

If Keyman Configuration is run as an elevated command, then the command will run in local machine context, otherwise it will affect only the current user.

KMShell.exe may be found in %ProgramFiles%\Tavultesoft\Keyman Desktop 8.0. On x64 operating systems, it may be found in %ProgramFiles(x86)%\Tavultesoft\Keyman Desktop 8.0.

Introducing KeymanWeb 2.0 Beta

KeymanWeb 2.0 today entered beta! In this new version of KeymanWeb we have put a big effort into redesigning the internals, restructuring and designing the code according to modern JavaScript principles, and using many of the techniques that other major JavaScript libraries use today. As well as this, we have created 4 user interface designs that are cleanly integrated into KeymanWeb, and even provided the flexibility for you to design your own user interface if you want!

So what has changed in version 2.0 beta? The key changes are:

• Complete refactoring of the KeymanWeb objects, splitting the user interface, the on screen keyboard, the core keystroke processing, and the various mapping tables.
• A new, consistent and clear API for accessing the KeymanWeb functionality from a web page.
• Bringing the user interfaces developed for KeymanWeb Subscriptions into the KeymanWeb Professional feature set.
• Exposing Language, Region, Language Code and Region Code properties for each keyboard registration.
• A consistent approach to user interface integration and making the feature set consistent between each user interface.
• Providing the API hooks to create your own KeymanWeb user interface.

As well as the major rework in the points above, we have done a lot of work with:

• Compatibility for various browsers – each browser has different limitations and issues for input
• Bug fixes

The most visible changes for web site owners looking to upgrade to the new KeymanWeb are around the API. We made the difficult decision to terminate support for the version 1.0 API rather than try to provide backward compatibility support. This is primarily because the original API grew organically as the product developed, and we could see many limitations and structural issues with continuing to maintain it. A clean break means we can provide a consistent, straightforward API now that can grow in the future.

KeymanWeb User Interfaces

KeymanWeb 2.0 makes available 4 standardized user interfaces. You can use any of these or implement your own.

Toggle User Interface

This UI appears to the right of the focused control; hovering over the button shows the drop down list as in the image above.

Float User Interface

This interface is much the same as the original KeymanWeb user interface, although some small bugs have been corrected, and support for multiple languages for each keyboard has been added.

Button User Interface

This control appears as a static button on the page, in a location of your choice; hovering over the button allows the user to select a keyboard and change the visibility of the On Screen Keyboard.

Toolbar User Interface

This user interface is the most powerful and flexible, and is best suited to a web page where many keyboards may be used by your site visitors. It groups the keyboard layouts by region and by language, and where more than one keyboard for a given language is available, allows easy selection of the most appropriate keyboard for the language.

KeymanWeb API Changes

In the remainder of this blog post I'll cover some of the key API changes.

Getting started with KeymanWeb

The original KeymanWeb added a number of global variables, actually expandos of the window object. These were prefixed with KeymanWeb_, but this was somewhat ugly. KeymanWeb 2.0 now defines single global object, tavultesoft, and the object tavultesoft.keymanweb contains all the functionality of KeymanWeb.

This means that initializing KeymanWeb is now much clearer as shown in the example below. Additionally, you can now specify the user interface that you want to use with KeymanWeb during initialization.

Initializing KeymanWeb 1.0

<script type="text/javascript">
var KeymanWeb_Key='your_key_here';
var KeymanWeb_Root = '';
</script>
<script src="keymanweb.js" type="text/javascript"></script>
<script type='text/javascript'>
window.onload = function()
{
KeymanWeb.Init();
}
</script>

Initializing KeymanWeb 2.0

<script type="text/javascript" src="kmw-release.js"></script>
<script type="text/javascript">
tavultesoft.keymanweb.init({
key: 'your-KeymanWeb-key',
ui: 'toggle',
root: ''
});
</script>

Registering keyboards

KeymanWeb 2.0 adds a number of properties to the keyboard registration, including language and region. This enables keyboards to be registered for more than one language, which is particularly helpful for keyboards such as European Latin, which supports over 100 different languages. Additionally, the region information is used to group the languages by continent for the Toolbar user interface. The example code below shows the old and new keyboard registration functions.

Note: a tavultesoft.keymanweb.registerKeyboard function is on the roadmap for release. This function will use more clearly named properties such as 'name', 'language', 'languageCode', etc. However, this change was not ready for the beta release.

Loading Keyboards – KeymanWeb 1.0

KeymanWeb.KRS(new Stub_Keyboard_european());
function Stub_Keyboard_european()
{
this.KF="european.js";
this.KI="Keyboard_european";
this.KN="European Latin";
}

Loading Keyboards – KeymanWeb 2.0

tavultesoft.keymanweb.KRS({
KN: 'European Latin',
KI: 'Keyboard_european',
KL: 'French',
KLC:'fra',
KR: 'Europe',
KRC:'eu',
KF: 'european.js'
});

Selecting Keyboards

This call is similar between the two versions of KeymanWeb.

Selecting a Keyboard – KeymanWeb 1.0

KeymanWeb.SetActiveKeyboard("Keyboard_european");

Selecting a Keyboard – KeymanWeb 2.0

tavultesoft.keymanweb.setActiveKeyboard("Keyboard_european");

Controlling the On Screen Keyboard

As well as providing much more control over the On Screen Keyboard, KeymanWeb 2.0 makes available a number of events that allow the web page to take action when something happens within KeymanWeb, for example, when the On Screen Keyboard is shown, moved or hidden.

KeymanWeb 1.0

// displays the On Screen Keyboard
KeymanWeb.ShowHelp();

KeymanWeb 2.0

// displays the On Screen Keyboard
tavultesoft.keymanweb.osk.show(true);

// calls my_adjustOSK whenever the On Screen Keyboard is shown
tavultesoft.keymanweb.osk.addEventListener('show', function(x, y, userLocated) { my_adjustOSK(x, y, userLocated); });

How to get the Beta

You can access downloads and live demos at the KeymanWeb Beta Home Page.

Three Cheers — Classical Greek Now Fixed on European Keyboards

Over the years, we've received a lot of support cases that look like this:

How do I type iota-subscript on your Greek keyboard? My hardware doesn't have a ¦ key.

If you've written in with this question before, you know our standard response.

The third-party Greek keyboards we offer were designed for US QWERTY hardware. For best results, you should associate your Greek keyboard with a US layout:
    http://help.keymandesktop.com/docs/start_configure_computer.php

Then, you'll find the iota-subscript using Shift+ the key to the left of the Enter key. That's  the "~" key for UK QWERTY, the " ' " key for DE QWERTZ, the "µ" key for FR AZERTY, the …

Sounds a bit complicated, no? Iota-subscript is pretty darned important to Classical Greek. It shouldn't be so hard to type on non-US hardware.

Starting today, it isn't. Cue the cheering masses: Hooray! Huzzah! Hip hip!

Starting today, if you use one of the following hardware keyboards, it'll no longer be a headache to type iota-subscript.

• AZERTY French
• QWERTY Dutch
• QWERTY Italian
• QWERTY Norwegian
• QWERTY Spanish
• QWERTY Swedish
• QWERTY UK
• QWERTY US
• QWERTZ German

Why? Because we've optimised Galaxie BibleScript Greek for European keyboards using the new improvements to mnemonic layouts in Keyman Desktop 8. If your keyboard has a "§" key, a "ñ" key, or a "¦" key, you can now type iota-subscript with nary a care, no matter where those keys are on your hardware keyboard. And this works even if your keyboard is not one of the ones officially optimised.

Better still, it's not just iota-subscript which has gotten easier to type. We've mapped all of the keys on your European layout meaningfully. French users will appreciate that keys like é and è produce έ and ὲ; Germans typing Ü will get Ϋ; and the Spanish ç key now gives ς.

You can try out the new mnemonic version of the Galaxie BibleScript keyboard here:
    http://keymankeyboards.com/?id=629

And if you decide you like the old method better, it's still online here:
    http://keymankeyboards.com/?id=413

Finally, for more details on all the optimised goodies in Galaxie BibleScript Mnemonic, check out:
    Galaxie BibleScript Mnemonic Help

Keyman Desktop 8.0.332.0 Beta Release

Keyman Desktop 8.0.332.0 is being released today as a Beta version.  This release addresses two reported issues with Keyman Desktop:

• Keyman Desktop 8.0 is not working with legacy applications that require WM_UNICHAR for Unicode input (regression from 7.1)

This affects a number of legacy applications, as well as programs that use the RichEdit control in a non-Unicode mode.  Basically, Keyman Desktop 8.0 would recognise that the program did not support Unicode input, but would then not switch into its legacy support mode.  If you have not experienced this problem, then you do not need to upgrade at this stage.

Applications known to be affected by this problem: Paratext 6.

• If a non-Latin script keyboard or IME is the default input method for Windows, then Keyman Desktop keyboard layouts may not operate correctly 

This issue had two primary symptoms:

1. If using mnemonic layouts, then the non-Latin script base keyboard would mean that no mnemonic layouts would operate, as the basic A-Z Latin script letters would be missing.
2. If the base layout was an IME (e.g. Japanese, Amharic, Korean, Chinese), then some hotkeys could interfere with the operation of any Keyman Desktop keyboard.

The fix now identifies the base keyboard script, and if it is a non-Latin base keyboard, will use US English as the base keyboard for Keyman Desktop keyboards to avoid this issue.

This release can be downloaded from http://www.tavultesoft.com/beta/

Editing Right-to-Left Text With Keyman Desktop – Hints and Tricks

Keyman Desktop works hard to make Right-to-left (RTL) languages such as Arabic and Hebrew work as seamlessly as possible in Windows. But even with all the work Keyman Desktop does, there are a few things that you need to know in order to effectively mix Right-to-left and Left-to-right (LTR) text.

Configuration

Where Keyman Desktop does most of the work for you is in the initial configuration: ticking the boxes and installing the files that Windows and Office need in order to work with Right-to-left languages.

All our modern Keyman Desktop keyboards will automatically configure your system to support Hebrew, Arabic, Persian, Pashto, or most of the other RTL languages out there. So what does Keyman Desktop do?

1. It makes sure Windows has the RTL language support files installed (important for Windows XP)
2. It installs the correct RTL language in Windows, and links it to the Keyman Desktop keyboard.
3. Keyman Desktop takes care of managing complexities around various European keyboard layouts.
4. It configures Microsoft Office to support your RTL language.

However, a few of the older Keyman Desktop keyboards do need a helping hand in the initial configuration. For that reason, full configuration instructions are documented in a Tavultesoft Knowledge Base Article at http://www.tavultesoft.com/kb/?id=41

Hint: It is often better to turn off Automatic Keyboard Switching in Word, as it can get very confusing when working in RTL and LTR languages in the same document – and it may not always select the right language. The Microsoft Office website shows you how.

Handy hint: Turn off Ctrl+Shift to switch keyboards, as it conflicts with the hotkey for switching paragraph direction. Microsoft have some instructions; see the note on this page.

Day-to-day usage

Each application has its own little tips and tricks on how to work in RTL and LTR languages. I'll start with some basic applications included with Windows.

Select your Keyman Desktop keyboard (we'll use Hebrew for this example).

Notepad

You'll find that even after you have selected your Hebrew keyboard, text still appears left aligned. Press Ctrl+Right Shift to make the whole document right aligned. In Notepad, this setting applies to the whole document. Right-clicking in Notepad allows you to insert various control codes, such as Left-to-right mark and Right-to-left mark, which control how punctuation, numbers and spaces will be displayed when mixing RTL and LTR text. More information on how these work can be found in this page in the Windows XP Documentation.

Wordpad

Like Notepad, Wordpad lefts you press Ctrl+Right Shift for right-aligned text, and Ctrl+Left Shift for left-aligned text. However, Wordpad allows you to do this per-paragraph, rather than for the whole document. Wordpad does try to guess the language you want to work with when you press these hotkeys, in Windows 7, and this may mean that you have to select your language and keyboard again after switching to RTL or LTR alignment.

Microsoft Word

Text Direction

In Word, it is important to use the Text Direction buttons as opposed to the paragraph alignment buttons. If you use the paragraph alignment buttons, you'll see strange things like words reordering or punctuation appearing in the wrong place. Pressing a Text Direction button will also update the paragraph alignment.

Paragraph Alignment: Don't use these buttons for RTL and LTR text

 

Text Direction: Do use these buttons for mixing RTL and LTR text

 

If you are writing in Blog mode, you'll find the Text Direction buttons do not appear in the ribbon by default. The Ctrl+Right Shift and Ctrl+Left Shift hotkeys will still work to switch text direction.

Mixing RTL and LTR text

Mixing Right to Left and Left to Right text in a paragraph can be pretty non-intuitive and frustrating at first. Here are some hints to make it a little easier.

• Think about paragraph direction: is the primary text English or Arabic? Or French or Farsi? The primary paragraph direction is controlled with the Text Direction buttons. Text with the opposite directionality can be entered in the middle of the paragraph, by switching languages – don't change the paragraph text direction.
• Spaces should be typed at the end of the primary language run, before you switch to the secondary language. Then switch back to the primary language at the end of the text run before pressing space again. For example: "The Arabic word حياة means life", you would type word<SPACE> before switching to Arabic to type حياة, and then type the following space after switching back to English.
• Arrow keys are somewhat unintuitive when working with RTL text. A bit of a shift in thinking helps: think about the Right Arrow in terms of Next Letter, and Left Arrow in terms of Previous Letter. Then Home becomes "First Character in Line" and End becomes "Last Character in Line", concepts which stick more easily.
• If you are not getting what you expect, check the selected language at the bottom of the screen, e.g. or . But note, when you switch languages, this information will not always be updated until you type another letter.

Troubleshooting

Problem: I get square boxes for Arabic or Hebrew text

Solution: On Win XP, turn on RTL support in Control Panel/Languages (or get Keyman to do it for you)

 

Problem: Word chooses the wrong font for my RTL text, or changes fonts on me unexpectedly

Solution 1: Make sure your keyboard is linked to the correct language (http://www.tavultesoft.com/kb/?id=41)

Solution 2: Make sure the Office Language Settings include your language (http://www.tavultesoft.com/kb/?id=41)

Solution 3: Make sure your chosen font includes full support for your language with the Keyman Desktop Font Helper (http://help.keymandesktop.com/docs/basic_fonthelper.php)

Solution 4 (from Andrew Cunningham): Word can have multiple 'active' fonts - one for mostly European scripts, one for 'complex' and right-to-left scripts, and one for East Asian scripts (Chinese, Japanese, Korean).  Press Ctrl+D to set the font for all three script classes.

 

Problem: When I press spacebar, words change order.

Solution 1: Make sure you are using the Text Direction buttons, not the paragraph alignment buttons (see above)

Solution 2: Make sure your keyboard is linked to the correct language (http://www.tavultesoft.com/kb/?id=41)

Solution 3: Make sure the Office Language Settings include your language (http://www.tavultesoft.com/kb/?id=41)

 

Problem: When I save my document and reload it, Word messes up the text order

Solution 1: Make sure your keyboard is linked to the correct language (this is unfortunately very hard to correct afterwards, but copying and pasting into Notepad and then copying and pasting back can help) (http://www.tavultesoft.com/kb/?id=41)

Solution 2: Make sure the Office Language Settings include your language (http://www.tavultesoft.com/kb/?id=41)

 

Problem: I cannot type the letters ý (U+00FD) or þ (U+00FE) in Word.

Solution 1: This is because Word recognises these letters as control codes to switch text direction. You can use Insert Symbol to add these letters as a once-off, or associate your keyboard with the Icelandic language. (http://scripts.sil.org/cms/scripts/page.php?item_id=FontFAQ_Display#6d0c5068)

 

Problem: I cannot find the Text Direction buttons.

Solution 1: Make sure Office Langauge Settings include your RTL language: if you do not have a RTL language configured, Word will hide these buttons.

Solution 2: In Blog Post mode, Word does not display these buttons, so use Ctrl+Right Shift and Ctrl+Left Shift instead.

 

Problem: When I press Ctrl+Left Shift or Ctrl+Right Shift, Windows changes the keyboard layout unexpectedly

Solution: Turn off the Ctrl+Shift keyboard switch hotkey in the Language Bar Key Settings. (http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/langbar_keystroke_shortcuts.mspx?mfr=true)

Working with unsupported RTL languages

When working with a language such as N'Ko, use Arabic as the base language, and make sure you turn off all proofing and autocorrect options.

Update (2 Sep 2011): Andrew Cunningham made several useful suggestions: Word has multiple default fonts which can be confusing when working with different scripts.  You can see all the different default fonts currently set by pressing Ctrl+D to open the Font dialog box.

Also, the following Word help topic goes into a lot more detail on how to work with RTL text: http://office.microsoft.com/en-us/word-help/right-to-left-language-features-HA010351021.aspx

Fixed link to font helper.

Keyman 8.1 Keyboard Language Proposal: Extensions to Keyboard Options

Introduction

Keyboard options were introduced in Keyman Desktop 8.0 in response to a common issue – many keyboard packages include multiple keyboards to cater for minor configuration changes. For instance a Tigrigna package has both Eritrean and Ethiopian keyboard layouts, with only a single character difference – the question mark symbol which differs between the two countries. Or Greek keyboards which differ in how the diacritics are typed.

With keyboard options, the keyboard developer can present the user with a dialog box to let them choose how they wish to use the keyboard. This is both less confusing and more flexible than having multiple keyboards.

Now when reviewing this feature a couple of weeks ago, we realized that with some small tweaks, it would become even more useful.

Proposed New Features

Three small updates to the keyboard options feature are proposed:

• Testing for base layout. This would enable you to have rules conditional for a French base layout, for example. This, combined with the improvements to mnemonic layouts in Keyman Desktop 8.0, should make it possible to cater for all European base layouts with a single keyboard.
• Allowing for On Screen Keyboard-specific rules. This would back up the functionality described in my previous blog post Keyman 8.1 Keyboard Language Proposal: Dynamic On Screen Keyboard and make the dynamic on screen keyboard more flexible.
• Add feedback on option changes via bubble and icon. While this needn't be limited to notification of option changes, some sort of feedback is really pretty essential for keyboard options to be most useful.

Testing for Base Layout

The test for base layout could be implemented with a special system store, for example:

if(&baselayout = 'FR') + 'q' > 'that key next to Caps Lock'
if(&baselayout = 'EN-US') + 'q' > 'that key next to Tab'

Determining how to identify the base layout has not yet been discussed.

Testing for On Screen Keyboard-specific rules

Again, this could be implemented with a special system store. In the following example, the 'Q' key would always show with a capital Q, even though the output would be lower case:

if(&osk = '1') + 'q' > 'Q'
if(&osk = '0') + 'q' > 'q' c not technically necessary but shown for completeness

Giving feedback via balloon or icon change

The details of how additional icons could be stored in the keyboard (and the preferred image formats) are not settled. However, in principle it could work as per this example:

store(happy-face-icon) "happy-face.png"
store(happy-message) "Keyman is now happy"
store(sad-face-icon) "sad-face.png"
store(sad-message) "Keyman is now sad"

store(sad-option) "0"

+ 'a' > seticon(happy-face-icon) showhint(happy-message) set(sad-option='0')
+ 'b' > seticon(sad-face-icon) showhint(sad-message) set(sad-option='1')

Determining the icon to use when the keyboard is initially selected would also need to be covered. At this stage, it is not anticipated that the icon would change in the menus or in the configuration screens, but feel free to ask for this – now's a good time to ask!

Keyman 8.1 Keyboard Language Proposal: Dynamic On Screen Keyboard

Introduction

Keyman 8.0 improved mnemonic keyboards by remapping the On Screen Keyboard to the underlying hardware layout. Previous versions of Keyman did not do this well. This has been a big improvement and makes mnemonic keyboards much more useful with European hardware layouts.

However, apart from this one change, we had to postpone some of the design features that we wanted to add to the On Screen Keyboard. A couple of weeks ago we had a great discussion on how we could improve the On Screen Keyboard by making it more dynamic. While dynamic updates to the keyboard have been on the future feature list for some time, details of how this could work had never been properly explored.

We also linked this discussion to some tweaks to the keyboard options feature introduced in Keyman Desktop 8.0. These were tweaks primarily around testing for base layouts and providing feedback to users when options are selected. The extensions to keyboard options have some relationship to a dynamic On Screen Keyboard but will be expanded on further in a separate blog.

Dynamic On Screen Keyboard

The big breakthrough we made in our discussion was in how we could instruct the On Screen Keyboard to dynamically update. Initially we just thought we could use the existing rules in the keyboard source code to tell the keyboard how to dynamically update. This seemed logical, but the more we looked at it, the more we realised that this would end up being confusing for the end user. There were a few reasons:

1. Keyman Desktop keyboard rules have to deal not only with basic mapping but also with character reordering and normalisation. These rules would probably be confusing when displayed on the keys.
2. Some rules make changes to multiple characters in the context but displaying the full output would not be helpful to the end user.
3. Some rules may input an invisible character but the On Screen Keyboard should instead show a hint for that letter

So we came up with the idea of a new begin statement to cater for the On Screen Keyboard (OSK). This could point to the same group as the normal begin unicode statement, or to a separate group designed to update the On Screen Keyboard. This group could use all the same context matching and finesse that the normal groups could, but the output would only be going to the On Screen Keyboard. So one trick to remember is that the output of the On Screen Keyboard rules will not affect the context for the next keystroke. The new OSK begin rule would be written as follows:

begin OSK > use(osk)

group(osk) using keys

The processing of any groups fired by the begin OSK statement have an important nuance: every matching context rule will be processed in order to render the On Screen Keyboard, for each keystroke. This means that effectively the On Screen Keyboard would run the group 47 or 48 times when updating. This shouldn't have a significant performance cost on computers today. If no rule matches for a given key, then Keyman would render the default output based on either the underlying layout or the US base key (depending on whether the keyboard is mnemonic or positional).

Keeping this in mind, we realised that colour, highlighting, graphics and font hints could be added to the output with a few additional simple statements. These additional formatting hints would be ignored by the keystroke processor, and would just be used by the On Screen Keyboard engine.

• bg(colour | default), or background(colour | default)
  
  This would set the background colour for the whole key. The last colour set in the execution of the statements would be the one that would take effect. The colour reference will probably be either a #rrggbb triplet or a standard web colour name.
  
  
• bg(colour | default), or background(colour | default)
  
  This would set the foreground colour for the subsequent letters to be printed on the key cap. In some scenarios, for example with combining diacritics, this may not be possible, and needs to be investigated.
  
  
• font(name, size, style)
  
  There are some issues with specifying the size due to the resizeability of the on screen keyboard. It may be specified as a percentage of default size.
  
  
• image(source-image)
  
  Source-image would refer to a store name which would then cause the image to be embedded in the keyboard file for use in the On Screen Keyboard. Suggested file format would be PNG to allow for alpha transparency. There are issues about sizing and mixing images and text that would need to be resolved.
  
  
• hint(text)
  
  If the user hovered the mouse cursor over the key, the hint text would be displayed in a balloon. There may be some visual indication on the key of the availability of a hint.

As there may no longer be a static On Screen Keyboard, we may need to specify font and other preferences such as 101 or 102 key layout as well. This is also a topic for further discussion.

Example Dynamic On Screen Keyboard Source File

The following straightforward keyboard will be used as an example of how we envisage dynamic on screen keyboards working.

store(&VERSION) '8.0'
store(&NAME) 'Dynamic OSK Example'

begin Unicode > use(main)
begin OSK > use(main)

store(vowel) 'aeiouAEIOU'
store(diaK) "`" "[" "]" ";" "'" "."
store(diaO) U+0300 U+030F U+030B U+0308 U+0301 U+0307
store(ZWNJ) U+200C

group(main) using keys

+ any(diaK) > $ZWNJ index(diaO, 1)

$ZWNJ any(diaO) + any(vowel) > index(vowel, 3) context(2)

c nomatch -> use default output

For simplicity in this example I have not tried to use any pre-composed letters but just left the combining diacritics in place. You'll note that we are using the same rule set for the Unicode rules as for the OSK rules, despite the discussion above. This is because this example is demonstrating the concepts rather than being a comprehensive and complete keyboard layout.

Given the keyboard source above, the Dynamic On Screen Keyboard would display initially as the image below. There isn't anything extraordinary there. The key difference is that there is no static definition for the On Screen Keyboard; rather, Keyman uses the begin OSK statement to define the display of the On Screen Keyboard. Deadkeys (as per the dk(diac) statements) would not be highlighted in any way on the On Screen Keyboard by default. This is because the deadkey statement in Keyman keyboards tends to be used for state management and not purely as a deadkey flags.

Let's look at how the keyboard could be modified to highlight deadkeys. The formatting hints given the example above would be ignored by Keyman Engine for keystroke processing, so let's keep the rules together for now.

store(&VERSION) '8.0'
store(&NAME) 'Dynamic OSK Example'

begin Unicode > use(main)
begin OSK > use(main)

store(vowel) 'aeiouAEIOU'
store(diaK) "`" "[" "]" ";" "'" "."
store(diaO) U+0300 U+030F U+030B U+0308 U+0301 U+0307
store(ZWNJ) U+200C

group(main) using keys

+ any(diaK) > $ZWNJ index(diaO, 1) bg(yellow)

$ZWNJ any(diaO) + any(vowel) > index(vowel, 3) context(2)

c nomatch -> use default output

Now with an empty context the keyboard would display something like the following mockup:

Finally, let's tweak the keyboard to make the dynamic keys with context highlighted so they are more visible. I'm not going to make the example comprehensive – it'll only cover the grave accent key.

store(&VERSION) '8.0'
store(&NAME) 'Dynamic OSK Example'

begin Unicode > use(main)
begin OSK > use(main)

store(vowel) 'aeiouAEIOU'

store(diaK) "`" "[" "]" ";" "'" "."
store(diaO) U+0300 U+030F U+030B U+0308 U+0301 U+0307
store(ZWNJ) U+200C

group(main) using keys

+ any(diaK) > $ZWNJ index(diaO, 1) bg(yellow)

$ZWNJ any(diaO) + any(vowel) > index(vowel, 3) context(2) bg(yellow)

$ZWNJ U+0300 + '`' > '`' bg(green)
$ZWNJ U+0300 + any(diaK) > context(1) index(diaO, 3) bg(grey)

c nomatch -> use default output

This would render as per the following example.

Conclusion

The concepts outlined here present a simple model of how on screen keyboards could give feedback to the end user and help make keyboards more self-documenting. I'd be very interested in any feedback or suggestions you may have – just put them in a comment on the blog.

In the next blog post, I'll discuss how the tweaks to options we discussed can also be used for integration of the On Screen Keyboard.

Updated 23 Aug 2011: In examples, replaced any(diaK) with any(diaO) in context, and replaced deadkey usage with ZWNJ to avoid side-effects with typing diacritic after another letter.