Sometimes the simplest things are the best.
By doing a few simple changes to the SD card library, I've managed to squeeze out a whopping 3x performance out of the SD card reading! I learned that once the card has been initialized into SPI mode it's possible to raise the clock speed A LOT.
Currently I'm setting the clock speed after init to ten times the speed before init. This means a full frame loading in just 2.5 ms and by doing this, I've increased the overall refresh rate by 15 Hz! The card is now reading approximately 614KB/sec!
The card functions with higher speeds, but data gets corrupted along the way. This is probably because the rise/fall times of the resistors in the SD card circuit needs time to switch. With a faster circuit I'm pretty sure I could squeeze out more!