I have been banging my head against the wall for the past week migrating to matrices and have made some forward progress. However the migration may have set me back more then it has helped me. By migrating to matrices I have left my camera in a broken state and have to redo all my calculations. It has also thrown of the positions for my crude physics setup.
Matrices have always been a bit of a love/hate for me. They are extremely painful to setup. However once they are setup keeping track of transformations is so much simpler.
All in all I will probably have to spend another week cleaning up the mess.