The board is now running different voltages without any interference!
I swapped the P-Channel for a N-Channel on the 12V "rail" and pull-downs to pull-up's on the remaining P-Channel MOSFET's. No more weird double triggering.
As for ghosting;
There's unfortunately still plenty of it.
I'm thinking this is my fault when designing the board since rows and columns are operated on the same chain of shift registers. This means I cannot turn off rows and columns independently from each other but instead have to rearrange everything at once. I've also hardwired the OE (output enable) of the shift registers to permanently being on, which of course is a part of the problem.I don't expect this to be a big issue, but I'll have to see.
In case it is a major nuisance I can always add a few control lines and possibly OE as well.
I'm thinking OE would be most significant, as this would allow me to properly turn off the matrix while transitioning rows (and columns).