I've struggled with random crashes and potential memory corruption scenarios for a while. But now I believe I've nailed down all bugs, starting from the String library and ended with the sound card serial commands...
 |
| 1) This is what most of my evenings look like nowadays. Can't say I'm not enjoying programming the machine, but a proper debug mode with breakpoints would be nice... |
What I have done lately:
+ Rewritten most of the core
+ Written tons of rules
+ Changed switches, solenoid and rendering to interrupt driven updates on Timer3 (tried Timer5 as well, no difference)
+ Added a large const array of animation information. (tried to cut it in half, made no difference for the crashes)
+ Added communication over Serial1@57600 to wavTrigger and Serial3@250000 to Arduino Mega2560
+ Added softPWMServo instead of hardware servos
+ Added EEPROM reading/writing of highscores and settings
+ Switched to dynamically allocated data (this was AFTER the crashes were observed, and it's never deleted/recreated - only once during startup)
I've switched all String's to const char* and...ta-ta - const Strings (sorry Majenko!) as well as some generic safety precautions. I've measured RAM during idle and play and it does indeed stay fixed now. If I do a "new" without delete the available RAM ticks down and once low enough it crashes, so the RAM function seem to work. There were just too many functions and special occasion that I needed to replace, and by doing so I'd most certainly would have ended up with an even buggier version of the machine. So once I saw that RAM usage was static, I moved on.
I've also implemented the hardware watchdog so in case it does crash it will reboot and it won't start burning (tried that, wasn't pretty). Works pretty great too! A peculiar thing thou - if the watchdog was held alive by an interrupt, the main loop could crash and the interrupt continue. Don't ask me how, but it did. I did a divide-by-zero on purpose and all game loop and switches etc froze, but the rendering kept going. So the watchdog seems like it must reside in the main loop to have a proper effect.
Lastly, what still caused crashes after everything else was fixed was actually the sound card, a Wav Trigger.
If I sent "too many", i.e more than a few request per 10-15 millisec it would eventually crash. I'm not sure which end causes the problems, but I do know that I can crash the machine by sending a faulty command to the sound card. (RobertSonics, if you're reading this - there could be something wrong with the latest firmware. It shouldn't crash if the command is formatted correctly but unknown...But then again - Chipkit shouldn't crash if the sound card crashes either...)
What I've done is to simply add a cooldown on each command sent to the sound card so that it will sit and wait for it's turn and then send the command. This works and no sound is every skipped/missed, but the implementation is not nice as everything except for solenoids and rendering is stalled in case there's lots of audio. Realistically there's probably only ever gonna be 3-6 samples being started every 300 ms during play, so perhaps I can add the cooldown only after a certain amount of active tracks instead of every time, for instance.
On top of this I've added animation, sound and rules to most standard switches now. I'll try to get this down in a video soon as it's looking pretty good, if I may say so myself. :)
...
Well, hello there, Mr Big-pile-of-cables...
Should we get you all soldered up?
No comments:
Post a Comment