...Chop-shop!

I've gone ahead and started butchering the light cables.
And I only killed seven switches in the process...

Man, it kinda hurts...but it will be all worth it when finished! :)
I will be able to say "light #5, that's red/blue" and see if the problem is the bulb, the board or MOSFET's etc. Troubleshooting will be a lot easier!

1) Old cables removed and the new ones await installing!

2) They don't look much, but will mean the world to Mr Bubbles!
I might need to order another batch of cables thou - apparently 60 m isn't enough ...

Hmmm-stable? Unstable?

So, long story short -
I've struggled with random crashes and potential memory corruption scenarios for a while. But now I believe I've nailed down all bugs, starting from the String library and ended with the sound card serial commands... 

1) This is what most of my evenings look like nowadays. Can't say I'm not enjoying programming the machine, but a proper debug mode with breakpoints would be nice...

What I have done lately:

+ Rewritten most of the core
+ Written tons of rules
+ Changed switches, solenoid and rendering to interrupt driven updates on Timer3 (tried Timer5 as well, no difference)
+ Added a large const array of animation information. (tried to cut it in half, made no difference for the crashes)
+ Added communication over Serial1@57600 to wavTrigger and Serial3@250000 to Arduino Mega2560
+ Added softPWMServo instead of hardware servos
+ Added EEPROM reading/writing of highscores and settings
+ Switched to dynamically allocated data (this was AFTER the crashes were observed, and it's never deleted/recreated - only once during startup)

I've switched all String's to const char* and...ta-ta - const Strings (sorry Majenko!) as well as some generic safety precautions. I've measured RAM during idle and play and it does indeed stay fixed now. If I do a "new" without delete the available RAM ticks down and once low enough it crashes, so the RAM function seem to work. There were just too many functions and special occasion that I needed to replace, and by doing so I'd most certainly would have ended up with an even buggier version of the machine. So once I saw that RAM usage was static, I moved on.

I've also implemented the hardware watchdog so in case it does crash it will reboot and it won't start burning (tried that, wasn't pretty). Works pretty great too! A peculiar thing thou - if the watchdog was held alive by an interrupt, the main loop could crash and the interrupt continue. Don't ask me how, but it did. I did a divide-by-zero on purpose and all game loop and switches etc froze, but the rendering kept going. So the watchdog seems like it must reside in the main loop to have a proper effect.

Lastly, what still caused crashes after everything else was fixed was actually the sound card, a Wav Trigger.

If I sent "too many", i.e more than a few request per 10-15 millisec it would eventually crash. I'm not sure which end causes the problems, but I do know that I can crash the machine by sending a faulty command to the sound card. (RobertSonics, if you're reading this - there could be something wrong with the latest firmware. It shouldn't crash if the command is formatted correctly but unknown...But then again - Chipkit shouldn't crash if the sound card crashes either...)

What I've done is to simply add a cooldown on each command sent to the sound card so that it will sit and wait for it's turn and then send the command. This works and no sound is every skipped/missed, but the implementation is not nice as everything except for solenoids and rendering is stalled in case there's lots of audio. Realistically there's probably only ever gonna be 3-6 samples being started every 300 ms during play, so perhaps I can add the cooldown only after a certain amount of active tracks instead of every time, for instance.

On top of this I've added animation, sound and rules to most standard switches now. I'll try to get this down in a video soon as it's looking pretty good, if I may say so myself. :)

...

Well, hello there, Mr Big-pile-of-cables...
Should we get you all soldered up?

Time flies!

Apparently I'm using a really really old version of MPIDE to compile the code with....
The servo-problem was solved a while back:

  12/21/2013 <BrianSchmalz>:
    * Fixed bug that caused glitches every 107 seconds.

The question is - do I dare to update the core software, considering the numerous changes I've made to the stock libraries? Can't say I recall exactly what I've changed and where.

But I suppose I need to do this someday anywho since the laptop I'm using is a very old one and can't even compile the code with optimizations active. For the release version I'm pretty sure I want to use compiler optimized code. :)

Also, a programmer called Majenko suggested that the Arduino cannot reliably communicate over serial at higher speed than 9600 baud. I'm not sure what to believe here, since I've used much higher speeds - but still, Majenko is a great programmer and has supplied a lot of libraries and corrections, so I figure he/she knows better. If the serial interface used to program the device is different than the actual interface the MCU is using for the pins inside sketches this might explain why the sketch is running great connected to my computer while failing connected to the MCU.

"The Arduino is unable to communicate reliably at higher baud rates due to the clock being unable to accurately divide down to the right kind of values, but the chipKIT boards, because of the higher speed master clock, can do it more accurately. As a result the Arduino sometimes can't communicate through serial with the chipKIT boards. Try running at a lower baud rate (9600)." [Chipkit forum]

9600 baud could possibly be enough for my needs, but I'm not sure what the "damage" to SD-reading and frame building would be. Still, it's something that can be investigated in the future.

Code-bumping

I've installed the sound board and light board now (light cables are still not done thou).

There's a slight ghosting issue present, hopefully it won't be disturbing with the proper lights as I've seen commercial machines do this as well, if you look closely enough. The problem is caused by the weirdest error ever - the lights are running perfectly when connected to a computer and receiving serial input. But when connected to the Chipkit it just goes bonanza. Naturally I suspected the Chipkit first due to recent interrupt changes, but the sound board worked fine on different serial ports and speeds so there's no problem with the serial ports, of that I'm sure.

I realized I was using a rather long (with delayMicroseconds()) interrupt routine on the Arduino and this caused problems with the serial input - but only when connected to another microcontroller... I shorted it down and now it works with the Chipkit, but is instead plagued with ghosting. Which it really shouldn't be doing. It's pretty weird, but for now this "workaround" will do great, but I'll  revisit this in the future if it turns out to be a proper problem. 

I've also programmed highscore saving/loading as well as proper settings saving/loading and a hardware alert routine during boot. The servos started acting up as a result of the interrupt driven display so I replaced the interrupt servo routine with a software based one instead. Seems to work well, except for an occasional servo "tick" now and then on the first out of two servos. This can probably be avoided by doing some magic in the library-code, as I suspect it's dropping due to a timer reset or similar. Quite possibly a dummy servo can be configured to take the hit instead...

Overall the machine is starting to feel really slick now and the software is bumped up to version 0.85!