[sip-comm-dev] Re: Desktop sharing and keyboard interactions

Sebastien Vincent seb at sip-communicator.org
Fri Aug 6 17:55:32 CEST 2010


When receiving an RFB's KeyEvent message (with the X11 keysym), here 
basically the behavior of VNC servers:

- X11 based: generate keycode with XKeysymToKeycode();

- OSX: find directly the CGKeycode in a map of Keysym/CGKeycode 
correspondance. => US only so no character like à, ù, ...

- Windows VNC: Look at some acute/circumflex/diaresis character keysym 
(à, ù, â, Â, ...) and generate two characters (à = XK_grave + XK_A), if 
this is an latin ascii character like a, b, c it is passed to VkScanCode 
to find the keycode and then regenerate via keybd_event. Nothing about 
cyrillic or arabic specific code.

The VNC Java Client converts the ascii/unicode directly to X11 keysym 
(see rfb/UnicodeToKeysym.java class).

For information, current implementation of the SIP Communicator's 
desktop sharing on Windows (server side aka sharer) works fine since all 
character from my keyboard are printed correctly (including the 
characters like à, ù, ...). So basically a Linux/Windows controller 
_with_ same keyboard layout (or at least same character is obtained by 
one key press or same key combination for both layout) can interact 
perfectly with a Windows desktop.

As I said Linux/Mac desktop sharer (with french keyboard layout) could 
not display some acute/grave/circumflex/diaresis character. To solve 
this problem, we could do as VNC, that is to have a map of ascii 
character and key or key combination _but_ we should do it for all 
keyboard layouts to be complete. I am not sure there is a programmatic 
way to find the keycode combination (or not) to obtain a specific keys 
(A => SHIFT + A).
Having a map can also solve the problem of having different key 
combinations or not for same key (french Mac OS X keyboard layout and 
standard Windows/Linux ones) and Windows code should also have its own 
ascii characters/keycode(s)

What do you think ?


Le 06/08/2010 09:30, Sebastien Vincent a écrit :
> Hi,
> Yesterday I study a little bit what is done in RFB protocol (in other 
> word VNC). The specification is here 
> http://www.realvnc.com/docs/rfbproto.pdf.
> The data that is transmitted in KeyEvent message are X11 keysym (note 
> that it does not correspond to a keycode) and the X11/keysymdef.h file 
> contains all the mapping from latin (1, 2, 4, 8, 9, ...), Arabic, 
> Cyrillic, ... to their respective keysims. The RealVNC Java viewer has 
> a class that can translate from ascii/unicode to keysyms (based on a 
> 742 elements map). Note that the standard latin 1 character (a, b, 
> ...) keysyms are the same as there ascii code.
> I have to look more closely on Windows RealVNC server and Mac OS X 
> ChickenOnVNC to see how they convert X11 keysyms to their own keycode.
> In page 23 of the specification, it also raise up the problem of 
> character that requires "shift" character to be print out correctly on 
> some keyboard mapping and not on others. They propose to fake the 
> additionnal key (for example a "shift"). Again I need to look at 
> server implementation to see how they know (if they could) the 
> combination to do to obtain the right character.
> I wonder if we should do like VNC (transfer X11 keysyms and then 
> perform X11 keysym to native keycode). Maybe we should be able to 
> print the characters that cannot return valid keysym using 
> XStringToKeysym() in current implementation. Again we should test if 
> non latin character works with the OS native function we used (X11's 
> XStringToKeysym() and Windows's VkKeyScan) for example with a cyrillic 
> ones. If it does not work we should probably used a solution based on 
> X11 keysym (at least it should work on Unix-based system).
> What do you think ?
> Regards,
> -- 
> Seb
> Le 04/08/2010 09:08, Sebastien Vincent a écrit :
>> Hello all,
>> I am currently working on a desktop sharing feature (with remote 
>> control) with SIP (and later with XMPP when JIngle will be ready). 
>> The mouse interaction is working fine (move, click, dragged and 
>> wheel) but I face some problems with keyboards.
>> In the first implementation I use java.awt.Robot to do all the stuff 
>> (mouse and keyboards). The keyPress and keyRelease methods use a 
>> keycode and all available keycodes are located in KeyEvent classes 
>> such as VK_A, VK_B, VK_BACK_SPACE, ... (note that this keycode can be 
>> different from native keycode of the OS).
>> So I sent the keycode (that are returned by KeyEvent.getKeyCode()) in 
>> the messages to control remote desktop. It works fine... except for 
>> some key such as @, /, ... that has no keycode returned by 
>> KeyEvent.getKeyCode().
>> And if keyboard mapping are different between server (desktop sharer) 
>> and client (remote desktop controller) some keycode may be map to 
>> different letter. Typically on a Mac OS X that act as server (french 
>> keyboard) and a Linux client, if I type "a" on Linux, "q" will be 
>> displayed on Mac OS X.
>> Another problem is that AltGr key (that is found on some european 
>> keyboards) is not properly generated. Other keys combinations or 
>> other special keys work fine like CTRL+C, CTRL+D, backspace, enter, ...
>> Now I try another solution. It is to sent directly the character 
>> typed if it has a printable ones (i.e. KeyEvent.getKeyChar() != 
>> KeyEvent.CHAR_UNDEFINED except some keys like ENTER, BACK_SPACE, ...) 
>> and I use a JNI to do the regeneration of keys. This solution can be 
>> used (I think) to display latin character and also cyrillic or other. 
>> For the other keys (CTRL, SHIFT, ...) we still sent the keycode (as I 
>> read somewhere it will not changed accross keyboard mapping) and 
>> regenerates key with Robot. For AltGr key we handle it in the JNI 
>> directly in the JNI code.
>> Technical details of native functions used:
>> - On Windows, use of VkKeyScan to find the Virtual key from a letter, 
>> MapVirtualKey to find the scancode and keybd_event to regenerate key 
>> press or release. AltGr is obtained by CTRL+ALT keys press/release;
>> - On Linux, use of XStringToKeysym()/XKeysymToKeycode to find the X11 
>> keycode from a letter and XTestFakeKeyEvent to regenerate key press 
>> or release. For special character (like @, /, \, ...) a map is used 
>> with the X11 string representation (@ => "at",  # => "numbersign", 
>> "altgr" => "ISO_Level3_Shift");
>> - Mac OS X, use of code from 
>> http://stackoverflow.com/questions/1918841/how-to-convert-ascii-character-to-cgkeycode, 
>> CGEventCreateKeyboardEvent to create the key press/release event and 
>> CGPostEvent to regenerate key press or release.
>> Now some problems with this solution:
>> - Some character like "à", "ù", ... on X11 and Mac OS X does not have 
>> a native keycode, Typically NoSymbol is returned from Unix's 
>> XStringToKeysym() (and thus JNI cannot regenerates key press/release) 
>> but Windows does.
>> - A problem related with keyboard mapping. Some letters may be 
>> generated with a key combination  on a specific mapping ("/" can be 
>> obtained by SHIFT+: on a french keyboard) and others may not. 
>> Typically on a french Mac OS X keyboard mapping "#" is obtained by 
>> SHIFT+@ and on Linux it is obtained by AltGr+". So in this case the 
>> character "#" will be sent to Mac OS X, the JNI will find the right 
>> key but as no SHIFT has been pressed, "@" will be displayed instead 
>> of "#".
>> - Some key combinations like CTRL+C, CTRL+D, CTRL+Z, enter, 
>> backspace... produce a printable character and have to be process as 
>> if they do not.
>> Has someone another idea on how to handle keyboard interactions in a 
>> desktop sharing sessions or solutions that can work accross keyboard 
>> mapping ?
>> Regards,
>> -- 
>> Seb

To unsubscribe, e-mail: dev-unsubscribe at sip-communicator.dev.java.net
For additional commands, e-mail: dev-help at sip-communicator.dev.java.net

More information about the dev mailing list