Cxbx + 64bits

[tabai]

Many of us here know that for Cxbx to run on a 64bit OS we need to get rid of the FS segment and use an alternative. Well I've been trying to kinda search and replace the instructions where FS is used and I've had a bit of success. Just a bit. Why? Because the FS points to the data structure where the thread's information is stored. Turok uses at least 3 threads so if I just replace the insturctions where FS is used with a static value it won't work. So I was thinking of replacing those instructions with others that get the correct values from other places in memory, without using the FS register. I know there's a guy that managed to get this working, but I really need to discuss this with anyone that has some knowledge of the inner workings of Cxbx...

[patrickvl]

Hi there, good of you to mention this.

The Cxbx hack that scans and patches opcodes that accesses the FS segment register, was originally made by Robert Yates. (He once managed the reverse-engineering.info website and made an Xbox Flirt file for IDA Pro).

On 15 july 2008 he mailed me a screenshot of Turok running with his proof-of-concept, along with a part of the code he used. Since then, I never heard from him again (even after several attempts), so this is all we have I'm afraid, and we'll just have to take it from there.

Even though I was missing the remainder of his code, I typed it over, and just made up the rest.
Indeed, this code does nothing to fix multi-threading issues.
Several people have looked at it already, but no-one has reproduced a running Turok with this, let along made it into something production-ready.
Last thing I heard, a Dxbx dev had trouble using this code because of getting incorrect FS-patches.

What we probably need, is a clear understanding of the layout of the Xbox Thread Information Block (TIB), which (if I remember correctly) is the structure that the FS-register points to.
If we could create a mapping from the Xbox layout to the native WinNT layout, we could update the opcodes indexing the FS register to using NT-based offsets. This way, thread-safety is returned, and Cxbx/Dxbx won't need to use LDT anymore, nor are there any FS-swaps needed anymore. (The LDT-part is what's preventing Cxbx/Dxbx running under WOW64).

Is anyone feeling up to this?

The code follows here, for anyone interested to take this to the next level :

NewPcr->Prcb = &NewPct->PrcbData;

prcbdataoffset = (DWORD)&NewPct->PrcbData;
ethreadoffset = (DWORD)&EThread;
newtls = (DWORD)pNewTLS;

__asm
{
      pushad

      mov eax, 00011000h
      mov edi, 1FA2ACh
      mov ecx, 1E92B2h

search_me:
      // 64 a1 28 00
      cmp dword ptr [eax], 0028a164h ; mov eax, large fs:28h
      je fix_eax_fs28

      // 64 a1 20 00
      cmp dword ptr [eax], 0020a164h ; mov eax, large fs:20h
      je fix_eax_fs20

      // 64 8b 0d 04
      cmp dword ptr [eax], 040d8b64h ; mov ecx, large fs:4
      je fix_ecx_fs04

      // 64 8b 3d 04
      cmp dword ptr [eax], 043d8b64h ; mov edi, large fs:4
      je fix_edi_fs04

      // 64 0f b6 05
      cmp dword ptr [eax], 05b60f64h ; movzx eax, large byte ptr fs:24h
      je fix_eax_fs24

      jmp cont111

fix_eax_fs28:
      mov byte ptr [eax], 0B8h ; mov eax, {00000000}
              mov ebx, EThread
      mov dword ptr [eax+1], ebx
      move byte ptr [eax+5], 090h ; NOP
      jmp cont111

fix_eax_fs20:
      mov byte ptr [eax], 0B8h ; mov eax, {00000000}
              mov ebx, prcbdataoffset
      mov dword ptr [eax+1], ebx
      move byte ptr [eax+5], 090h ; NOP
      jmp cont111

fix_ecx_fs04:
      mov byte ptr [eax], 0B9h ; mov ecx, {00000000}
              mov ebx, newtls
      mov dword ptr [eax+1], ebx
      move byte ptr [eax+5], 090h ; NOP
      move byte ptr [eax+6], 090h ; NOP
      jmp cont111

fix_edi_fs04:
      move byte ptr [eax], 066h   ; Put 66 bf, which means :
      move byte ptr [eax+1], 0BFh ; mov di, {0000}
              mov ebx, newtls

// GUESSWORK FROM HERE ON :
      mov dword ptr [eax+2], ebx
      jmp cont111

fix_eax_fs24:

      move byte ptr [eax+0], 090h ; NOP
      move byte ptr [eax+1], 090h ; NOP
      move byte ptr [eax+2], 090h ; NOP
      move byte ptr [eax+3], 090h ; NOP
      move byte ptr [eax+4], 090h ; NOP
      move byte ptr [eax+5], 090h ; NOP
      move byte ptr [eax+6], 090h ; NOP
      move byte ptr [eax+7], 090h ; NOP
      jmp cont111

cont111:
      inc eax
      cmp eax, edi ; Use ecx instead?
      jne search_me
}

Attached Images

[POC]_cxbx_modified_to_avoid_LDT_issue.jpg (184.4 KB, 90 views)

[tabai]

I have tried the patching first by replacing the instructions using fs:28 to calls to a function on the dll that just does the same as the instruction (mov eax, fs:28). It should work? It doesn't but I can't find out why. If I debug the code (using ollydbg) it always crashes at some point, but if I run it without debugging it won't crash. I have looked at the generated code and debugged through it, and it appears to work properly.

If I try to patch the instructions using fs:20 the game will always crash at some point regardless of being debugged or not.

Edit: I'm trying this out with Turok using a Virtual WinXP 32bits on a Win7 64bits host.

[tabai]

Ok, so now I know what's wrong. The problem is that the call modifies the EFL because it does a check when returning...

CMP EBP,ESP
CALL CxbxKrnl._RTC_CheckEsp
POP EBP
RETN

[tabai]

IT was just a matter of disabling runtime checks! In other words, running it as a release version!

Or for it to work in debug: Configure project properties: C\C++ - Code Generation - Basic Runtime Checks = Uninitialized Variables (/RTCu)

[tabai]

patching works except for "mov eax, large fs:20h"

[tabai]

IT was just a matter of disabling runtime checks! In other words, running it as a release version!

Or for it to work in debug: Configure project properties: C\C++ - Code Generation - Basic Runtime Checks = Uninitialized Variables (/RTCu)

[blueshogun96]

Okay, so let me get this straight, you actually got this working? Good for you! I tried numerous times and couldn't do it myself. If you can show me the code you modified (preferrably your EmuFS.cpp or any other modified source files), then I can add it to the current SVN and the next release will have x86-64 support!

__________________

---

Last edited by blueshogun96; May 3rd, 2010 at 02:29..

[tabai]

Not really I'm still working on mov eax, fs:20 ... the other stuff works! Shogun, how much do you know of the xbox threads and the FS?

[blueshogun96]

Little to nothing. That's the only thing I haven't been able to understand.

__________________

[tabai]

I'm attaching the files I modified... EmuFS.cpp and EmuFs.h. It's kinda messy, but I'm just testing it out....

TODO:
- Implement structures to hold the relevant data per thread...

I'm just saving the last created thread's data, and only one replacement function (EmuMEaxFs28) checks if its in the correct thread (using the fs register).

- Remove all the FS dependent code (EmuSwapFS, etc.)

Not all has to be removed, but much has to be replaced with the proper code.

- Fix the damn mov eax, dword ptr fs:[20h] @ 0x14227

For some reason if I replace this one the program will crash... It has proven to be really hard to debug because pausing the emulation causes it to crash too! (I think it is so because the other threads keep running and eventually die...) So for the moment I skip patching it at 0x14227 offset, but all the others are patched...

I won't be able to work on this for a week probably... anyone is welcome to pick up the code and build on it.

Edit: I use shogun's branch... I don't know if it matters, but just so you know... I haven't tested it on native x86, nor x64, just emulated WinXP x86 (VMWare). Hehe working on an emulator in an emulator.

Attached Files

EmuFS.zip (5.1 KB, 69 views)

---

Last edited by tabai; May 3rd, 2010 at 05:19.. Reason: More info...

[patrickvl]

Using XBEExplorer, I saw the address you're having problems with (at 0x14227) is part of _RtlFreeHeap@12, which starts at 0x141F2 in Turok.
Cxbx and Dxbx both patch this function (routing it to EmuRtlFreeHeap, which forwards the call to CxbxRtlFree), so it's strange you're experiencing a crash, as the original code shouldn't ever be reached anyway!

On the removal of EmuSwapFS : I think it's better to leave all calls to that function in, but make the implementation conditional. This way you can have two versions : One that runs as usual, and one that tries to become WOW64-friendly.

One other thing : As far as I understand the subject, in 'WOW64' mode (when patching FS-using opcodes), all code that initializes LDT and swaps FS should be completely disabled. So, don't call EmuInitLDT() nor EmuAllocateLDT(), nor EmuDeallocateLDT(), nor EmuCleanupFS(), and disable everything in EmuGenerateFS() except the FS-patching itself. (It's probably better to make the distinction in the calling code instead, but that's not the most important thing right now.)

[patrickvl]

Oh, and maybe this didn't come across clearly, but for this FS-patching to become thread-safe, it's best to keep accessing the FS-register - but just alter the offset used, so the value being read becomes the actual NT-based version. As I said, we need to create a mapping-table for this, so we can map Xbox TIB offsets to WinNT TIB offsets.
Has anyone more info on this?

[Nokiaman]

Oh man oh god oh man oh god! Good luck to all of you guys. You got me excited that I will finally can test this on my native OS and get rid of VmWare.

__________________
AMD Phenom II X4 965 3.4Ghz
nVidia GTX 580 1536MB
4096MB DDR2
Windows 7 Ultimate SP1 64bit
Click Here YO!

[tabai]

Originally Posted by patrickvl

Cxbx and Dxbx both patch this function (routing it to EmuRtlFreeHeap, which forwards the call to CxbxRtlFree), so it's strange you're experiencing a crash, as the original code shouldn't ever be reached anyway!

Wow, that's odd... I set a debug point at that address and it was hit, so its definitely being executed. (patched or not) Does this happen on DxBx too?

Originally Posted by patrickvl

Oh, and maybe this didn't come across clearly, but for this FS-patching to become thread-safe, it's best to keep accessing the FS-register - but just alter the offset used, so the value being read becomes the actual NT-based version. As I said, we need to create a mapping-table for this, so we can map Xbox TIB offsets to WinNT TIB offsets.
Has anyone more info on this?

From my understanding, caustik combined the Xbox TIB and the WinNT TIB in Cxbx, and even used the available data slots to change among different FS (EmuSwapFS, FS:[0x16], FS:[0x14]).

I think it would be better if we identify the functions in which the FS register is used, maybe when can replace or rehook them like we do with all the DirectX calls?

It is quite a problem that we don't know most of the contents of the Xbox registers and memory locations. Do you think we can get a better understanding of the Xbox internals if we get a Debug Xbox and poke around to find out what's going on?

[patrickvl]

In Dxbx, if I manually set a breakpoint in Turok at 0x14227, it never gets hit. (And our log shows that EmuRtlFreeHeap is called quite frequently.) So that's probably a symbol-detection and/or patching problem in your Cxbx build.

If you have an Xbox SDK, you could try to look into XDK\xbox\symbols\Latest\xboxkrnl.exe + xboxkrnl.pdb using IDA Pro, search for FS-using code, and try to guess the meaning of the TIB-offset being used, by comparing that function to the same function in the ReactOS codebase. Please document what you find, so someone else can write a clean-room piece of code from that.

Oh, and I think it's not really feasible to try and patch all library functions that access FS, as that doesn't cover game-code, which can access FS too. In my humble opinion, it's really best if we could map Xbox TIB offsets to WinNT TIB offsets, as that would fix all code, not just those functions we know about. Perhaps that we could even remove a few patches, once it's been proven that they can run normally once the TIB-offset mapping is applied to that code.

---

Last edited by patrickvl; May 3rd, 2010 at 18:02..

[tabai]

I don't have an Xbox XDK . And I'm still trying to understand how the TIB works (not in concept, but in implementation)... I have read that the main difference in x64 from x86 is that the GS register is the one that's used. But I don't understand the relationship between the LDT and the TIB (or TEB).

If we were to map the Xbox TIB to the NT TIB, how would we implement it?

Patrick, I think I read somewhere that someone was porting your symbol detection to Cxbx?

---

Last edited by tabai; May 3rd, 2010 at 19:18.. Reason: Automerged Doublepost

[patrickvl]

As long as the offsets in the WinNT TIB are of the same size as the Xbox TIB, it's simply a replacement of one opcode arugment with the other.

Imagine the Xbox TIB offset 28 is at offset 24 in WinNT; Then the opcode bytes need to be updated, like this :

64 a1 28 00 (mov eax, large fs:28h)

should be changed to :

64 a1 24 00 (mov eax, large fs:24h <- updated the TIB offset to 24)

Obviously, we must find & update all FS-using opcode, which means a lot of scanning, but it's doable.


About the back-port of my symbol-detection code, that's on hold. The reason is two-fold:
- the person working on it (Defiance) is quite busy with IRL issues
- I'm still busy improving the Delphi version, so it's better to wait a bit until that's finished

Rest assurred, it has my full attention. I'll start codeing on it again, after I finish a bit of research on offsets into cross-referenced data symbols, which are currently missing in our pattern files. I'll probably build a scanner for dumpbin's output, as that's the only tool that outputs the needed information. Alas, that will mean we'll have to get dumps from all XDK's again. Hopefully, my social network is strong enough to make this a breeze

[tabai]

I've read a bit more and it seems that the problem is that FS is a segment selector, and in x64 it only works as a base segment, not a selector... Please correct me if I'm wrong... ?

[patrickvl]

Don't know about 64 bit itself, but WOW64 should use the same TIB layout as 32 bit WinNT kernels. Where did you gather your information?

The layout of the WinNT TIB is documented already, so it's really the Xbox1 TIB that we're after... If you want I could try to document anything I can find on the Xbox TIB layout, so you can write the mapping code for it. But first I want to rewrite Dxbx's pattern-generation code, as correct symbol detection is currently the main reason that Dxbx crashes where Cxbx works.