Now what?

Ok, so I went ahead and soldered separate data, clock, latch and enable lines for rows and columns and updated the code to reflect the changes.

STILL ghosting.

Albeit less, it's still apparent when the surrounding is dark. 

There's no ghosting when not switching rows so at least that narrows it down slightly.
I'm thinking that either the LED modules suck, or the MOSFET's doesn't close fast enough. The MOSFET's are rated for logic voltages and should fully saturate. However - Considering the data sheet the opening and closing times are rather large thou:

Turn on rise time: 210-430 ns 
Turn off fall time: 110-230 ns

This leads me to believe that (I haven't calculated this yet) perhaps the switching is a bit on the slow side for a LED matrix purpose.

But it is what it is, and quite possibly this is not an issue when connecting the actual lights.
So I'm moving on for now.


Edit:

So I lied.
I haven't moved on.

Apparently I need to apply some kind of load that removes excess current still in the circuit/LED's after the MOSFET's themselves have closed. It makes sense, as explained by Chris (dcel):

"With no load after the voltage falls below the LEDs turn off voltage, there is nothing to bleed off that stored energy, hence that "afterglow". You may want to put a pull up resistor on your n-chan drain as well so its not floating.

     In my troubleshooting, I put an incandescent lamp in parallel with the LED lamp and that made the channel turn off hard. Uhm... interesting, threw a 100R in with same hard off result. I experimented with increasing resistances and found for my app, that 100K was sufficient to turn the FET off hard at 1ms or less. Good enough for me. "

Something like this.

1) Fancy schematics. The resistors around the LED should lead current away from the circuit when either side shuts off.

Sounds fair enough and relatively easy to test without destroying something. #FamousLastWords

There's also ways to push the fall time below normal by forcing it shut with negative voltages. But that sounds at lot more complicated and risky to me. On a plus side; I've also found evidence that Williams had problems with ghosting in hardware as well and worked around it in software.