GDIHooking library for NVDA

[aleksey_s]

i am implementing a library, with which nvda can to have access to the text, which is written by GDI functions directly. this is the delphi sources, as well as precompiled binaries. GDIHook.dll is an in-process dll which making hooks to api, GDIHookHandler.dll is manager for it. GDIHookTest.exe is an test application which works with GDIHookHandler exported functions.
note, that all stuff can have bugs and might crash your system!

[pvagner]

To test the library the following code is implemented in the patch attached to this ticked
1) ability to read owner-drawn menuItems (such as those found in the open with submenu in the windows explorer context menu)
2) ability to read owner-drawn ListBoxItems? (tested with miranda-im chatrooms)
3) test script bound to NVDA+f4 which can be used to speak all the text drawn to the client rect of the current navigator object.
4) some more class mappings in the iaccessible which I am using; so I did not want to split my testing copy to two branches.

Some notes
This is really a smal preview for testing purposes or for those wishing to help in the development. We do need to come up with better logic when and where to hook into. Current implementation might crash any application. Most likely it will crash applications which user interface is runing in various threads. Also memory usage shal be monitored while testing because of pending bugs.

[aleksey_s]

i decided to post here my changelog, to any who want to see how development goes on.
1 alex 2008-03-21

basic commit

2 alex 2008-03-21

*injecting with using windows hooks, not CreateRemoteThread?
*AttachToProces? -> AttachToWindow?
+DetachFromWindow?
+synhronization

3 alex 2008-03-22

*improved synchronization
*Now log file will create in user temporary dir

4 alex 2008-03-22

*code was rewritten to store and use text rectangles, not only coords. so now when new text appears with coords on one pixel different from other (it is really possible, lol) it will be rewritten instead of saving it. also it means GetTextAtCoords? will return no text at that really coords but text with which rectangle given coords intersects.
-removed hooking of functions which are really wrappers of other (such as DrawText?, DrawTextEx?)
*implemented bit handling of multiline text, it is trivial but it works and now we have not (i hope) repeated text
*possibly bugs fixed, added a lot of new

[aleksey_s]

5 alex 2008-03-23

*Started removing abuses of using shared memory, now Attaching and unattaching read info from remote process directly.

6 alex 2008-03-24

+implemented class, which incarnate dynamic arrays in shared memory. so now memory usage is more better.
+Added PTInRect function, becouse that, which windows api provides works not such i thought :-)
*fixes to ClearStorageInRect?
*other improvements

Now there precompiled debug versions of both libs, named GDIHook_debug.dll and GDIHookHandler_debug.dll. you can put their in nvda folder and delete suffix "_debug", then libraries will provide a log with all happened stuf and you can see what is incorrect. log can be found at your_temp_dir/gdihook.log

[aleksey_s]

7 alex 2008-03-25

*fixed crash when detaching

8 alex 2008-03-25

*a lot of try..except added, so now error handling will be more smart. when not debug compilation, may be error messages can be not so informative, but with debug compilation i hope it will be easier to find where an error occurs.
*again fix to synchronization code, i hope last :-)
*more new debug messages to SharedDynamicArrays?.pas and Storage.pas

9 alex 2008-03-25

*a litle fix to prevent crashes when twice attaching to one application

[aleksey_s]

10 alex 2008-03-26

+Now library will know exactly where hooked multiline text drawn on the screen. I believe, it is big step forward.
Problem was becouse when program want draw multiline text, it cann pass it whole to one of drawing functions with start coordinates, and rectangle where text will be clipped. so program can call drawing function many times with passing each time one line less, and drawing function will calculate what text will be really drawn by given rectangle.
+OpenStorage? function which try to open existing Shared Array instead of creating new
*GDIHookHandler exported functions will try to OpenStorage? instead of doing it in DLL_PROCESS_ATTACH, so when GDIHookHandler will hook to process in which need to inject GDIHook it will no greater try to init storage (possibly crash fixed)

currently there are following problems in the library:
*i do not know how to handle (determine real coordinates from logical etc) text from windows with handle 0 (especially dialogs)
*i do not know how to handle menu closing, dialog closing, window closing events to clear their rectangles, but GDIHookHandler exports function ClearStorageInRect? which might be called from NVDA to do these things

[pvagner]

wrapper for GDIHook.dll. In this version all the code which tryes to avoid duplicates is removed as it's perfectly done within the library it-self. anyway my attempt was not correct.

[aleksey_s]

11 Ђ¤¬Ё­Ёбва в®а <Ђ¤¬Ё­Ёбва в®а@NOTEBOOK> 2008-04-02

*Completely rewritten hooking mechanizm, now it must be stable in multithreaded applications. Thanks to my friend Sergey Starovoy for routines to calculate opcode size and other tips.

[aleksey_s]

r12

[aleksey_s]

12 Ђ¤¬Ё­Ёбва в®а <Ђ¤¬Ё­Ёбва в®а@NOTEBOOK> 2008-04-03

*fixed some small bugs in InterceptAPI.pas, HookedFunctions?.pas

[jteh]

I've updated Peter's patch for current trunk and committed it to a bzr branch for further development. The branch is at:
 http://bzr.nvaccess.org/nvda/gdi/

[jteh]

Aleksey, any further progress on this?

• Have there been any updates to the dlls since they were last posted here?
• What are the current limitations/problems of which you are aware?
• You noted that you needed some assistance at some point in order to make further progress. Can you elaborate about the information you need or at least the problems you are unable to solve?

Are you using bzr for version control? If so, we should get you an account on our hosting server at some point so you can publicly post the bzr branch.

I am moving this out of 0.6. This does not mean that it will not go into 0.6, but it means that it will not block 0.6 if it's not ready when 0.6 is to be released.

[jteh]

• Ideally, we want to attach in gainFocus of Window NVDAObjects and detach in loseFocus. This allows any higher level object to use the GDI support as needed.
• While testing, I experienced crashes in Notepad++ if attaching for all Window NVDAObjects. Any ideas on why this might be or can anyone else reproduce this?
• It seems WM_PAINT is not really the correct way to redraw the window. This seems to work quite nicely and can be done in one call:

  user32.RedrawWindow(windowHandle, None, None, RDW_INVALIDATE)
  

  where RDW_INVALIDATE = 1.

[aleksey_s]

Replying to jteh:

Aleksey, any further progress on this?
* Have there been any updates to the dlls since they were last posted here?

yes, i have worked around some major mechanizms of intercepting and attaching/unattaching, as well as storing data. i decided to store each hooked data in the in-process structures, and write it in shared memory when needed. so now each gdihook library when injecting creates some hided window to receive commands, and uses shared memory as some type of stack to receiving and sending data to handler. however, all problems i noted earlier still exist (e.g. with clearing unnecessary data, controls in dialogs etc).

* What are the current limitations/problems of which you are aware?
* You noted that you needed some assistance at some point in order to make further progress. Can you elaborate about the information you need or at least the problems you are unable to solve?

look at my earlier comments.

Are you using bzr for version control? If so, we should get you an account on our hosting server at some point so you can publicly post the bzr branch.

yes, i am.

I am moving this out of 0.6. This does not mean that it will not go into 0.6, but it means that it will not block 0.6 if it's not ready when 0.6 is to be released.

i am completely agree.

[jteh]

(In #151) This issue occurs because some menu items which display an icon don't seem to expose their name to MSAA. As far as we are aware, the only way to work around this is to obtain the name using display information. (We believe this is how other commercial screen readers solve this problem.) This requires display hooks. Although we have a very early implementation of display hooks, it is not yet ready for inclusion in NVDA. See #40 for details.

Note that the "Open With" menu in Vista does not seem to suffer from this issue.

[Bernd]

Aleksey, any further progress on this?
Have you done more work on this ticket

since your last post here? I'm asking because I'm interrested on your work.

[aleksey_s]

Replying to Bernd:
no. i am waiting for implementation of unified way for nvda to receive text info from another process, as i know Mick is working on this.

[aleksey_s]

i am thinking about continuing my work. for this i have to consider few necessary moments:

• is it possible to continue development using delphi (or other pascal compiler)? i know object pascal best of all, but i want to progress and practice in C++. for now i haven't nothing completed in this language, just learning programs.
• form in which gdi hooks will be distributed and api must be considered. in form of distribution i mean will it be a separate files or may be included to nvdaHelper? it depends on how do you see the last. if it will be library, which contains all low level (read - c++) code for NVDA, then including gdi hooks in it is quite reasonable. as i know it allready loads in all processes when nvda starts. then it must be rewritten in c++, with some restructurization / more abstraction. Also, we must to decide carefully which api gdi hooks will provide. i really feel that those current exist are not way to go. i think that gdi hooks and winevent processing depends on each other. you can see it with menu stuff. i consider that responding on events twice (inside nvda code and inside nvdaHelper) is not good idea, however for now it is so. it must be decided at some point how to proceed with it.
• what injection mechanizm to use? win hooks is nice for system wide hooking. but it has its disadvantages. i found some of that on  http://www.codeproject.com/KB/system/hooksys.aspx
  • Windows Hooks can degrade significantly the entire performance of the system, because they increase the amount of processing the system must perform

for each message.

• It requires lot of efforts to debug system-wide Windows Hooks. However if you use more than one instance of VC++ running in the same time, it would simplify the debugging process for more complex scenarios.
• Last but not least, this kind of hooks affect the processing of the whole system and under certain circumstances (say a bug) you must reboot your machine in order to recover it.

injection also possible through CreateRemoteThread, but in this case system-wide injection is difficult to implement: we cannot receive notifications about new processes creation without nt kernel driver. so we have to decide whether or not we want gdi hooks injected in all processes or just specific ones. i myself prefer first way.

• what interception mechanizm to use? I do not like hacking import table of each module for changing pointer to real function to the own fake. advantage of this method is that it is multithread-safe, but bigger disadvantage is that when you receive pointer to function with dynamic linkage (by calling GetProcAddress), you get pointer to the real function. however, i am not sure in it. Richter recommends this method in his "windows internals".

second method is to replace a few first bytes of function with jmp instruction to our code, which will execute needed manipulations and after put real code back, call true function and after again put jmp instruction to the beginning of true function. big disadvantage of this method is that it isn't multithread-safe: consider situation when one thread calls api function, jmp points processor to the our code, which make special processing of parameters and after replaces beginning of api function with it real code (to call true function). and in this moment, other thread calls api function too. what we will have? thread two is calling not hooked function. worst, it might call function when instructions are not completely replaced - with unpredictable result...
third method is most complicated. However, i will try to describe it here. note also, that in current implementation it is allready partially used. for installing hooks, injected code freezes al running threads (for garantee that no one will cal uncompletely hooked function). then for each function, it copies a few of it first instructions (how many needed to match 5+ bytes size). most difficult is to calculate length of instruction. it works like disassembler. then it must replace all relative jumps/calls to nonrelative (absolute). so, it creates buffer in memory, where it writes these (possible changed) instructions, and on the end of the buffer, it puts jmp instruction to address after these cutted instructions in hooked function. instead of cutted instructions in real function, it writes jmp to the fake function which will process parameters etc. after doing it work, fake function will jmp to the known buffer, in which real instructions are copied and jmp on they continuing. this method is multithread-safe and has a good performance, you cannot somehow jump through fake function (as you can with first method i described), but it has its difficulties in implementation: for now i do not change relative addresses in cutted instructions to nonrelative, so if there are allready one api hook installed, it will be broken.

for me, even major is your opinions on the first question (about architecture), if you are unfamiliar with all this lowlevel technical details i will try to investigate they myself.

[Bernd]

Hi guys,

is there any further progress on this? I ask because some german people asked me.

[mdcurran]

New implementation of gdi hooking in branch:
 http://bzr.nvaccess.org/nvda/displayModel

This implementation is in c++, and is built right in to nvdaHelper.
The previous gdi hooking library was very useful reading while creating the new implementation.

I'd probably suggest that for future work on the new implementation we create new tickets.

Some notes:

• Due to the current way of API hooking, this code only works on 32 bit systems, and only works on XP SP2 or higher.
• This code is far from complete, we will definitely let the curious peoples know when this is ready for testing etc. For now, this is very much heavily under development.
• For testing purposes, there is a default appModule script (test_navigatorDisplayModelText) (nvda+control+f2) which simply speaks and logs the text chunks that intersect the navigator's location rectangle.
• There is one display model per device context -- DCs we create display models for are either actual window DCs (the client area of a window), or a memory DC that was made compatible for a particular window DC.
• Only unicode API functions are hooked at the moment. Ansi ones certainly could be added, though it seems that on XP and above, ansi API calls end up eventually getting routed through a unicode one at some stage.
• Currently ExtTextOut?, and a few functions from uniscribe are hooked. All high-level text writing functions since win 2k now go through lpk.dll and on to uniscribe (USP10.dll) for language processing and output. Note that they do end up calling ExtTextOut? anyway, but only giving it glyphs, not actual characters, so its necessary to hook at least as high as uniscribe.
• Basic bit blitting is supported, by hooking BitBlt?. Watching for bit blitting is necessary as many applications create a temporary memory DC, write text to it, and then bit blit the content back to the window DC in one go (AKA double buffering).
• From my testing, I can see text for things such as:
  • Windows 7 Windows Explorer list view and tree view.
  • Standard comctrl buttons, combo boxes, check boxes etc.
  • Standard edit controls
  • Importantly for me, some custom delphy window classes used by the Australian E-Tax program.

As I have been develping this on Windows 7, I know it sort of runs ok here. Havn't really tried on XP/Vista yet, though if there are issues, its most likely in my bad coding of critical sections etc.
Multi-thread access to various maps etc needs to be improved a bit.

[mdcurran]

Extra note:
If I try running this branch on an old XP SP2 laptop I have, the NVDA startup sound plays, and then nothing at all. If I press escape there's a Windows critical stop sound. Yet if I then hold down the power button, the NVDA shutdown sound plays fine. So, no doubt there's something I'm assuming in the display model code that isn't true for XP... either that or that laptop is just really old, and or needs a reinstall.

As for Vista, I just tried this branch on my wife's computer, and all works fine there.

I'm interested if any other developer is able to run this branch on XP sp2/3, with out it freezing on startup?

I'll look more in to this in the coming days.

[pvagner]

I have tested it on XP SP3.
It builds all fine with Windows 7 SDK.
Also first run is okay. I can get the text for some controls e.g. edit field in notepad, clock in the system tray, however buttons, treeviews listviews don't seem to be speaking at all.
Also I have found a way to reproduce a hard crash on demand.

str

* Open notepad,
* write something in the edit field,
* press ctrl+o to open a new file,
* Answer no to the question asking whether you would like to save the current file,

Actual results

As long as the open dialog comes up NVDA crashes. Nothing usefull is written to the log, I can't kill it using the task manager nor using other memorized methods such as dismissing the dialog to close it. I can't start another copy of NVDA at this point and the only way I can use to recover from this crash is to reboot.

expected result

Heh I think I shouldn't expect anything at this stage. Good job mick anyway. Perhaps I am at least a bit helpfull.

[jteh]

I tried this on an XP SP2 VM. Frequently when it tries to load into task manager or explorer (and possibly many other applications), I get this dialog:

Microsoft Visual C++ Runtime Library
Runtime Error!
Program: C:\WINDOWS\explorer.exe
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

I'll see about getting a debugger attached so we can try to debug this.

[mdcurran]

With changeset:displayModel,3410 NVDA seems to run ok on my old XP SP2 laptop.
Peter/Jamie: does this help you also?

[pvagner]

Yeah now with this update and with subsequent additions it is running fine here.
I am on XP can retrieve text from list views, treeviews, some buttons, edit fields and also owner-drawn menu items. No crashes so far.

[jteh]

(In #200) Please don't change these fields unless you're certain about the change. For example, this is definitely not a virtual buffer related issue.

This issue requires display hooks to fix.

[pvagner]

(In #605) Textpad uses custom control for displaying text. I am not aware of any programatic method we can use to retrieve the caret position. Perhaps this will only be possible after display hooks are implemented.

[jteh]

Basic functionality implemented. New tickets should be opened for further fixes and enhancements.

[jteh]

Oh... changeset:main,3508.