For some reason the SD card library seem to ignore all my requests for unbuffered reads (or maybe the SD card I'm using doesn't normally allow this?) so yesterday I took the liberty of further customizing it to read into my own buffer directly, instead of first reading to an internal buffer and then copying the information byte-by-byte into the destination buffer.
I believe this bypassed a lot of the delay in the SD card reading, but I have not yet confirmed this with any actual numbers - but at the very least it certainly runs faster now. By the looks of the new animation speed, I'm figuring it's close to 40-45 FPS streaming video now instead of the previous 30-35 FPS.
I hope this can be further improved even though it's more than acceptable at the moment!