Well I managed to get the 64-bit build working on Vista 64-bit, don't know what the problem is with XP-64.
It seems the 64-bit version is no faster than the x86 version and is quite a bit slower in comparison to the last Dolphin 64 Beta 3 release on my machine.
I already enabled Dynarec and Dual core but it doesn't seem to make much of a difference.
I think something is fubar'd with the 64-bit compile like it wasn't compiled properly as speed is no different than the x86 version, and it is quite a bit slower than the last Dolphin 64 Beta 3 release as mentioned above.