So after a while, I have decided that I am not happy with the current performance of the engine. Side note for those that say to use C++ yes generally faster than C# because of the extra control. It, however, does not solve the problem of being slow it just masks it(As I like to say a bad driver in a good car is still going to be bad).
I ended up switching over to a deferred render. A lot of my “lighting” calculations were taking up the processing time of the GPU seeing as if the pixel was disposed of it would have to recalculate the colour and lighting (C++ cannot magically fix this). This allowed me to calculate the lighting at the end of the frame when all the pixels where final causing much less wasted processing time. It also means that all lighting calculations are O(G + L) rather than O(G * L) (Yes I can get forward to be less than O(G * L) as well but it gets very dodgy and messy). As a side effect, I am going to have a much harder time with transparency but I do not plan on using it much.
To have it not look as bad I ended up writing and modified version of the Phong lighting model to have a toon like look to it. Not really much to talk about here seeing as there is not much to it. I also added a post-processing pipeline to allow effects to be applied to the frame before it is sent off to the back buffer.
I have finally settled on a version that I feel comfortable showing off so sometime in the near future there may be a demo of it.