Turns out that I was only getting a slowdown when the mouse was over the window. I had also turned on mouse trails today!?!?
I just wanted to say beware! If you think you are doing something wrong when you are getting less than half the fps you'd thought you'd be getting, you should check for that stupid mouse trails option.