Tuesday, July 25, 2006

And then, some calls just don't return

A small number of library procedures are "called" but never actually return. Eventually I'd like to have a way to specify these procedures with anotations in the signature files, but for the moment they are hard coded in the frontend. That's acceptable for the moment as there is only five: _exit, exit, ExitProcess, abort and _assert. Thing is, what happens when you have a branch over one of these calls, as you often do. Typically you end up with phi statements that refer to the call or the statements before it because there's no way to show that these definitions are killed by the call but not defined by it. We could add this to the SSA form but a simpler solution is available. Whenever the decompiler determines that the destination of a call is one of these no return procedures then we simply drop the out edge of the containing basic block. Without an out edge the definitions cannot flow beyond the call.

Using dataflow based type analysis and some of my new changes the decompilation of extract_kyra.exe is currently looking a bit better. In particular, proc9, proc10 and proc2 are showing significant improvement.

6 comments:

  1. I believe that the type analysis engine has also changed from the ad-hoc version to the data flow based one between the first output and the second.

    This type analysis engine has not seen much action with structures, and from proc2 it is obvious that it missed out on typing param1 as a pointer to some sort of struct. Hence the
    *(int*)(param1+8)=...
    This should be fixed soon, which will make proc2 even closer to what it should be.

    I'll stay tuned for the next installment.

    - emmerik

    ReplyDelete
  2. I remember some strange code, in MS programs, where arguments and one or more return addresses follow the call instruction. It might not matter nowadays, but there may exist more cases than only the simple "never return" pseudo-subroutines.

    ReplyDelete
  3. I won't comment on a few missing Win32 functions that are noreturn, but I will comment on the mistake in thinking assert (or any of its namesakes) is terminal. It is not.

    MS C RTL allows you to continue execution after an assert, and that's something I frequently do by attaching a debugger (MS runtime libraries allows you to start+attach a debugger to a running program that has triggered an assert) to crawl back up the call stack to see context of what has gone wrong on a "live" program. Sometimes it's then even possible to do runtime patching of data, to continue execution without the assert triggering. This is quite useful if you otherwise have minutes, or even hours, of runtime to reproduce the problem.

    So tagging assert as noreturn is an error.

    ReplyDelete
  4. I won't comment on a few missing Win32 functions that are noreturn, but I will comment on the mistake in thinking assert (or any of its namesakes) is terminal. It is not.

    MS C RTL allows you to continue execution after an assert, and that's something I frequently do by attaching a debugger (MS runtime libraries allows you to start+attach a debugger to a running program that has triggered an assert) to crawl back up the call stack to see context of what has gone wrong on a "live" program. Sometimes it's then even possible to do runtime patching of data, to continue execution without the assert triggering. This is quite useful if you otherwise have minutes, or even hours, of runtime to reproduce the problem.

    So tagging assert as noreturn is an error.

    ReplyDelete
  5. Hmm. That's interesting Tamlin. The exit block for _assert in MSVCRT.DLL is at 0x77C33D8C. There are only two in-edges to this block, 0x77C33D8B which is an int 3 instruction and 0x77C33D95 which is a jump. Although you can step over an int 3 I think it's fair to say that in normal execution we're not going to return from _assert if we followed that path. 0x77C33D95 however is a block with only one in-edge, 0x77C33D90 which also has only one in-edge, 0x77C33D86 which also has only one in-edge, which is the result of a call to the code that displays the "Press Retry to debug the application" dialog. For a console application it's a totally different library too.

    ReplyDelete
  6. The version of msvcrt.dll I tried to check with is 6.1.9844.0, the one in Windows 2000 sp4, and its first address is 0x78001000 why we can't compare that. Nevermind, the source code for it is available (as is almost all of the C RTL) so one can always check with that.

    From disassembly though, I'd say it's got at least four exit blocks - two separate jmp's to abort() (jmp's instead of call's, as abort() is noreturn and takes no arguments), one int3 (which breaks into the debugger - same as DbgBreak() - if the app is being debugged, if not debugged it terminates the process) followed by pop esi, leave, retn, and finally there is a call to _exit around 0x2a bytes before the int3.

    So there nuances here to look out for. There is different runtime behaviour not only between GUI and console apps, but even in what the int3 (possibly even int 3, the non-single-byte version, but that I never checked) instruction performs. For that reason, one can't even say that int3 (0xcc) is terminal/noreturn as it all depends on the current runtime environment.

    For a more authorative reference of what MS CRT functions are noreturn, you could grab a copy of MSVC (the free VC 2005, aka MSVC8, Express version f.ex.) and grep the include directory for "__noreturn". I found 5 headers and 8 functions, but I believe ony 4 of those functions are of interest:
    terminate, exit, _exit and longjmp.

    ReplyDelete