GDIHooking library for NVDA

[aleksey_s]

i am implementing a library, with which nvda can to have access to the text, which is written by GDI functions directly. this is the delphi sources, as well as precompiled binaries. GDIHook.dll is an in-process dll which making hooks to api, GDIHookHandler.dll is manager for it. GDIHookTest.exe is an test application which works with GDIHookHandler exported functions.
note, that all stuff can have bugs and might crash your system!

[pvagner]

To test the library the following code is implemented in the patch attached to this ticked
1) ability to read owner-drawn menuItems (such as those found in the open with submenu in the windows explorer context menu)
2) ability to read owner-drawn ListBoxItems? (tested with miranda-im chatrooms)
3) test script bound to NVDA+f4 which can be used to speak all the text drawn to the client rect of the current navigator object.
4) some more class mappings in the iaccessible which I am using; so I did not want to split my testing copy to two branches.

Some notes
This is really a smal preview for testing purposes or for those wishing to help in the development. We do need to come up with better logic when and where to hook into. Current implementation might crash any application. Most likely it will crash applications which user interface is runing in various threads. Also memory usage shal be monitored while testing because of pending bugs.

[aleksey_s]

i decided to post here my changelog, to any who want to see how development goes on.
1 alex 2008-03-21

basic commit

2 alex 2008-03-21

*injecting with using windows hooks, not CreateRemoteThread?
*AttachToProces? -> AttachToWindow?
+DetachFromWindow?
+synhronization

3 alex 2008-03-22

*improved synchronization
*Now log file will create in user temporary dir

4 alex 2008-03-22

*code was rewritten to store and use text rectangles, not only coords. so now when new text appears with coords on one pixel different from other (it is really possible, lol) it will be rewritten instead of saving it. also it means GetTextAtCoords? will return no text at that really coords but text with which rectangle given coords intersects.
-removed hooking of functions which are really wrappers of other (such as DrawText?, DrawTextEx?)
*implemented bit handling of multiline text, it is trivial but it works and now we have not (i hope) repeated text
*possibly bugs fixed, added a lot of new

[aleksey_s]

5 alex 2008-03-23

*Started removing abuses of using shared memory, now Attaching and unattaching read info from remote process directly.

6 alex 2008-03-24

+implemented class, which incarnate dynamic arrays in shared memory. so now memory usage is more better.
+Added PTInRect function, becouse that, which windows api provides works not such i thought :-)
*fixes to ClearStorageInRect?
*other improvements

Now there precompiled debug versions of both libs, named GDIHook_debug.dll and GDIHookHandler_debug.dll. you can put their in nvda folder and delete suffix "_debug", then libraries will provide a log with all happened stuf and you can see what is incorrect. log can be found at your_temp_dir/gdihook.log

[aleksey_s]

7 alex 2008-03-25

*fixed crash when detaching

8 alex 2008-03-25

*a lot of try..except added, so now error handling will be more smart. when not debug compilation, may be error messages can be not so informative, but with debug compilation i hope it will be easier to find where an error occurs.
*again fix to synchronization code, i hope last :-)
*more new debug messages to SharedDynamicArrays?.pas and Storage.pas

9 alex 2008-03-25

*a litle fix to prevent crashes when twice attaching to one application

[aleksey_s]

10 alex 2008-03-26

+Now library will know exactly where hooked multiline text drawn on the screen. I believe, it is big step forward.
Problem was becouse when program want draw multiline text, it cann pass it whole to one of drawing functions with start coordinates, and rectangle where text will be clipped. so program can call drawing function many times with passing each time one line less, and drawing function will calculate what text will be really drawn by given rectangle.
+OpenStorage? function which try to open existing Shared Array instead of creating new
*GDIHookHandler exported functions will try to OpenStorage? instead of doing it in DLL_PROCESS_ATTACH, so when GDIHookHandler will hook to process in which need to inject GDIHook it will no greater try to init storage (possibly crash fixed)

currently there are following problems in the library:
*i do not know how to handle (determine real coordinates from logical etc) text from windows with handle 0 (especially dialogs)
*i do not know how to handle menu closing, dialog closing, window closing events to clear their rectangles, but GDIHookHandler exports function ClearStorageInRect? which might be called from NVDA to do these things

[pvagner]

wrapper for GDIHook.dll. In this version all the code which tryes to avoid duplicates is removed as it's perfectly done within the library it-self. anyway my attempt was not correct.

[aleksey_s]

11 Ђ¤¬Ё­Ёбва в®а <Ђ¤¬Ё­Ёбва в®а@NOTEBOOK> 2008-04-02

*Completely rewritten hooking mechanizm, now it must be stable in multithreaded applications. Thanks to my friend Sergey Starovoy for routines to calculate opcode size and other tips.

[aleksey_s]

r12

[aleksey_s]

12 Ђ¤¬Ё­Ёбва в®а <Ђ¤¬Ё­Ёбва в®а@NOTEBOOK> 2008-04-03

*fixed some small bugs in InterceptAPI.pas, HookedFunctions?.pas

[jteh]

I've updated Peter's patch for current trunk and committed it to a bzr branch for further development. The branch is at:
http://bzr.nvaccess.org/nvda/gdi/

[jteh]

Aleksey, any further progress on this?

• Have there been any updates to the dlls since they were last posted here?
  
• What are the current limitations/problems of which you are aware?
  
• You noted that you needed some assistance at some point in order to make further progress. Can you elaborate about the information you need or at least the problems you are unable to solve?
  

Are you using bzr for version control? If so, we should get you an account on our hosting server at some point so you can publicly post the bzr branch.

I am moving this out of 0.6. This does not mean that it will not go into 0.6, but it means that it will not block 0.6 if it's not ready when 0.6 is to be released.

[jteh]

• Ideally, we want to attach in gainFocus of Window NVDAObjects and detach in loseFocus. This allows any higher level object to use the GDI support as needed.
  
• While testing, I experienced crashes in Notepad++ if attaching for all Window NVDAObjects. Any ideas on why this might be or can anyone else reproduce this?
  
• It seems WM_PAINT is not really the correct way to redraw the window. This seems to work quite nicely and can be done in one call:
  

  user32.RedrawWindow(windowHandle, None, None, RDW_INVALIDATE)
  

  where RDW_INVALIDATE = 1.
  

[aleksey_s]

Replying to jteh:

Aleksey, any further progress on this?
* Have there been any updates to the dlls since they were last posted here?

yes, i have worked around some major mechanizms of intercepting and attaching/unattaching, as well as storing data. i decided to store each hooked data in the in-process structures, and write it in shared memory when needed. so now each gdihook library when injecting creates some hided window to receive commands, and uses shared memory as some type of stack to receiving and sending data to handler. however, all problems i noted earlier still exist (e.g. with clearing unnecessary data, controls in dialogs etc).

* What are the current limitations/problems of which you are aware?
* You noted that you needed some assistance at some point in order to make further progress. Can you elaborate about the information you need or at least the problems you are unable to solve?

look at my earlier comments.

Are you using bzr for version control? If so, we should get you an account on our hosting server at some point so you can publicly post the bzr branch.

yes, i am.

I am moving this out of 0.6. This does not mean that it will not go into 0.6, but it means that it will not block 0.6 if it's not ready when 0.6 is to be released.

i am completely agree.

[jteh]

(In #151) This issue occurs because some menu items which display an icon don't seem to expose their name to MSAA. As far as we are aware, the only way to work around this is to obtain the name using display information. (We believe this is how other commercial screen readers solve this problem.) This requires display hooks. Although we have a very early implementation of display hooks, it is not yet ready for inclusion in NVDA. See #40 for details.

Note that the "Open With" menu in Vista does not seem to suffer from this issue.

[Bernd]

Aleksey, any further progress on this?
Have you done more work on this ticket

since your last post here? I'm asking because I'm interrested on your work.

[aleksey_s]

Replying to Bernd:
no. i am waiting for implementation of unified way for nvda to receive text info from another process, as i know Mick is working on this.