So this is a bit of a fun one. I figured seeing as I hardly use C++ these days and I am probably rusty I decided to test my knowledge by creating an API to intercept DX11 calls. Warning going to be a long one. Those that want to see the result scroll to the bottom.
Getting the Application to Load to Hook
This, funnily enough, is the easiest part. There are 2 main ways of doing this to my knowledge. The first is to use an external application to start the process in question and inject the DLL into it some of you may be familiar with this through applications such as RenderDoc. The second is to take advantage of the order Windows searches for DLLs, Windows will first check the local directory of the application before anywhere else, this means you can drop a DLL into the application directory and Windows will load that DLL for the application. The second method is the one I ended up using.
Defining an Entry Point
This is where it gets a little weird normally windows has a series of entry points you can use when a DLL is loaded but for whatever reason, they were not working. To get around this I had to change how I would go about things. Luckily C++ has .def files that you can use to define global functions so I ended up defining the functions used to create the DX device and doing my initialisation there. Also before anyone asks the real DirectX DLL is not loaded yet it is not loaded until after I finish my initialisation. In order to load the real DLL, it is as simple as using the absolute directory.
Starting the DX11 Hooks
This is where it gets interesting. A common way to go about hooking into DirectX is to abuse Namespace, Defines and includes this, however, this is too simple(simple is no fun) and requires me to reimplement a large part of DirectX 11. To get around doing extra work I had to think back to my theory days of C++ this may get a bit random so stay with me(Also I am going to massively oversimplify so please don’t roast me). As many of you may know a lot of languages these days are Object Orientated which makes life so much easier. One of the features that make it great is polymorphism this allows whoever is inheriting to be similar to the parent but perform a different function if they so chose. The problem, however, is to keep track of what type object performs which task. A solution that some compilers use to get around this is to use a virtual function table. This is where each object has a list of addresses to the functions they use. Luckily DirectX has all the functions they use as virtual and Microsoft based compilers and APIs use this method. Essentially what I do is access the virtual function table of each object and replace the address of any functions I want to use with the address of my own functions.
So this was the annoying part cleaning up after other developers. There where multiple issues I had here that was kinda annoying.
So this may sound strange but I had no idea each GPU manufacturer had their own implementation of DirectX. This one took me a while to figure out. There is nothing actually wrong with this seeing as they probably run better on their hardware, however, this meant I had to redirect them to the standard DirectX
This is either the biggest coincidence or Steam also plays with the virtual function table. Why do I say this? I had an issue where some functions would stop giving me callbacks. This was annoying to find seeing as I had to debug assembly to find that the virtual function table had been changed after I changed it. My first attempt to fix this was to set the function back to my function afterwards then set it to read-only, this, however, caused a crash within the Steam overlay. So I had to go the more annoying route of just setting it back every frame.
This is not so much of a problem as it is an observation. I noticed that about 30% of the games I tried actually hung onto resources and where not disposing of them properly. Luckily in most, it was just an Asset here and there. Makes me wonder but how many people use the debug layers. Are we sure we are ready for DX12 and Vulkan?
Hooking the Windows Message Loop
So a big problem I had when I was trying to implement UI is that I could not get input. To get around this I ended up looking into the standard method of using the Windows message loop. Luckily Windows lets you change the callback with the pointer to the Window which I got when the application made the Swap Chain. The major problem, however, was the application still needed to receive messages. At first, I tried just invoking the old callback if I did not need the message. Windows, however, showed its face with access violations. To get around this I tried changing the permissions of the function but this for whatever reason caused a crash in DXGI(I suspect Windows spaghetti code strike again). Then I had an idea and realised it is possible to receive a message while inside the callback for whatever reason. I then used this to set the old callback, resend the message and set myself as the callback again.
After all this I was able to achieve intercepting calls to DirectX 11 in addition to sharing the message callback loop with the application. This will allow me to make a wide variety of things. I also learned a few quirks around Windows and DirectX thanks to this.
A short video to demonstrate using it to view render texture(It can read all but I don’t think the devs want their assets out in the open :D)