Recently I’ve been diving deep back into reverse engineering. What I wanted achieve had me needing to use a concept known as ‘OpenGL Hooking’
through the process called ‘Inline Hooking’. I decided to write about it here because explanations online related to these topics are scarce at best.
Please bare with me if theres any mistakes I’m still learning and mastering this concept! Also I’m not the best with my words so please forgive me for any grammatical errors
Well, what the hell is ‘Inline Hooking’?
Inline hooking is what I consider the absolute black magic of computer programming. It is the process of intercepting calls to a specific function by modifying the instruction flow directly in memory.
A simpler way to put it: Inline hooking is the process of redirecting a function within a running process’s memory. By overwriting the function’s entry point (push most times I believe?), you can divert the execution to your own code (usually inside a DLL using C++). This means when the program tries to run that specific function, it hits your detour first, giving you full control over the data before or instead of running the original code!
Using the process of inline hooking we can hook directly in a graphics engine of a program that uses OpenGL through hooking into opengl32.dll’s wglSwapbuffers function!
By hooking into opengl32.dll, we are essentially stepping into the middle of the graphics chain of processing (pipeline). This gives us the power to intercept render calls before they ever reach the GPU. From here, we can filter things like assets. We can tell the engine to skip drawing certain textures (creating a xray effect), replace textures with our own, or even change how light and shadows interact with the environment, the possibilites are endless.
What you need (Not including a dll with hooking code inside):
Debugger
I personally use x64dbg, You can’t use a dissambler I believe because those show addresses that are dead (Process is not running therefore the address are not alive), we’ll use this to breakpoint the function we want to take over.
Cheat Engine
While you might not always need to verify addresses for standard functions like wglSwapBuffers, it’s essential when targeting custom functions elsewhere in the code. We use Cheat Engine to find a static offset.
It’s important to note that in modern games, functions aren’t at a fixed ‘static address’ because of ASLR (Address Space Layout Randomization). This means the function’s location changes every time the game launches. To bypass this, we don’t look for the raw memory address (e.g 0x7FFE1234), instead we find the Offset (the distance) from the start of opengl32.dll. By calculating DLL Base Address + Offset, our hook will work every single time, regardless of where ASLR moves the file.
Your process of choice
If you want to hook into an OpenGL program, it must be using OpenGL linked to opengl32.dll. But if you just want to hook into a program and intercept its calls, you don’t need to worry about it.
DLL injector
I use Process Hacker (zuzus)
Step 1: Find the function.
Open your program in x64dbg. First, make sure you are passing exceptions to the program (or ignoring them in settings) so the debugger doesn’t pause every time the game pauses. If it does pause, hit F9 until the status bar in the bottom right turns from a yellow ‘Paused’ block back to a running state.
Next, go to the Symbols tab and look for opengl32.dll (or your main executable). Double click it, then search the function list for the one you want to hook (like wglSwapBuffers if your trying to hook into OpenGL). Double clicking the function name will take you to the CPU view, landing you right at the Entry Point. Once there, press F2 to set a breakpoint. Now, perform an action in the game that triggers that function, which the debugger will kinda snap to an instruction (You dont need this if your hooking into OpenGL).
Step 2: Verify the address. (Can be optional)
You want to make sure your ACTUALLY looking at the enterance to the function, therefore we need to verify it. In the debugger it should’ve taken you right to the function hit. You know if your at the right spot if you see something like: mov edi, edi, push ebp, or sub rsp. Now take the address you found and shove it into the Cheat Engine’s manually added address list, if you see the address listed in green, you know you have found it!
If you hook into the wrong spot in your dll, the program will crash the second the function is reached.
Booya!
Step 3: Creating a DLL that hooks into the address you found.
I will not be explaining step by step on programming your dll or executable but I will give a basic blue print.
When hooked (In the case with OpenGL we are intercepting wglSwapBuffers):
Change the first 5 bytes of the original function we got (push or maybe mov) to make room for your jmp instruction so it detours the original code to yours.
Helper function to restore those stolen bytes:
Create a helper function in the dll that puts the bytes we stole back to the program (push or maybe mov) so it will execute the bytes we originally stole.
Reconstruct the function gateway:
So our reconstrcuted wglSwapBuffers function would be like this:
typedef BOOL(__stdcall* twglSwapBuffers) (HDC hDc);
twglSwapBuffers owglSwapBuffers;
twglSwapBuffers wglSwapBuffersGateway;
BOOL __stdcall wglSwapBuffers(HDC hDc)
{
printf("OpenGL Hooked!\n");
return wglSwapBuffersGateway(hDc);
}