...Chop-shop!

I've gone ahead and started butchering the light cables.
And I only killed seven switches in the process...

Man, it kinda hurts...but it will be all worth it when finished! :)
I will be able to say "light #5, that's red/blue" and see if the problem is the bulb, the board or MOSFET's etc. Troubleshooting will be a lot easier!

1) Old cables removed and the new ones await installing!

2) They don't look much, but will mean the world to Mr Bubbles!
I might need to order another batch of cables thou - apparently 60 m isn't enough ...

Hmmm-stable? Unstable?

So, long story short -
I've struggled with random crashes and potential memory corruption scenarios for a while. But now I believe I've nailed down all bugs, starting from the String library and ended with the sound card serial commands... 

1) This is what most of my evenings look like nowadays. Can't say I'm not enjoying programming the machine, but a proper debug mode with breakpoints would be nice...

What I have done lately:

+ Rewritten most of the core
+ Written tons of rules
+ Changed switches, solenoid and rendering to interrupt driven updates on Timer3 (tried Timer5 as well, no difference)
+ Added a large const array of animation information. (tried to cut it in half, made no difference for the crashes)
+ Added communication over Serial1@57600 to wavTrigger and Serial3@250000 to Arduino Mega2560
+ Added softPWMServo instead of hardware servos
+ Added EEPROM reading/writing of highscores and settings
+ Switched to dynamically allocated data (this was AFTER the crashes were observed, and it's never deleted/recreated - only once during startup)

I've switched all String's to const char* and...ta-ta - const Strings (sorry Majenko!) as well as some generic safety precautions. I've measured RAM during idle and play and it does indeed stay fixed now. If I do a "new" without delete the available RAM ticks down and once low enough it crashes, so the RAM function seem to work. There were just too many functions and special occasion that I needed to replace, and by doing so I'd most certainly would have ended up with an even buggier version of the machine. So once I saw that RAM usage was static, I moved on.

I've also implemented the hardware watchdog so in case it does crash it will reboot and it won't start burning (tried that, wasn't pretty). Works pretty great too! A peculiar thing thou - if the watchdog was held alive by an interrupt, the main loop could crash and the interrupt continue. Don't ask me how, but it did. I did a divide-by-zero on purpose and all game loop and switches etc froze, but the rendering kept going. So the watchdog seems like it must reside in the main loop to have a proper effect.

Lastly, what still caused crashes after everything else was fixed was actually the sound card, a Wav Trigger.

If I sent "too many", i.e more than a few request per 10-15 millisec it would eventually crash. I'm not sure which end causes the problems, but I do know that I can crash the machine by sending a faulty command to the sound card. (RobertSonics, if you're reading this - there could be something wrong with the latest firmware. It shouldn't crash if the command is formatted correctly but unknown...But then again - Chipkit shouldn't crash if the sound card crashes either...)

What I've done is to simply add a cooldown on each command sent to the sound card so that it will sit and wait for it's turn and then send the command. This works and no sound is every skipped/missed, but the implementation is not nice as everything except for solenoids and rendering is stalled in case there's lots of audio. Realistically there's probably only ever gonna be 3-6 samples being started every 300 ms during play, so perhaps I can add the cooldown only after a certain amount of active tracks instead of every time, for instance.

On top of this I've added animation, sound and rules to most standard switches now. I'll try to get this down in a video soon as it's looking pretty good, if I may say so myself. :)

...

Well, hello there, Mr Big-pile-of-cables...
Should we get you all soldered up?

Time flies!

Apparently I'm using a really really old version of MPIDE to compile the code with....
The servo-problem was solved a while back:

  12/21/2013 <BrianSchmalz>:
    * Fixed bug that caused glitches every 107 seconds.

The question is - do I dare to update the core software, considering the numerous changes I've made to the stock libraries? Can't say I recall exactly what I've changed and where.

But I suppose I need to do this someday anywho since the laptop I'm using is a very old one and can't even compile the code with optimizations active. For the release version I'm pretty sure I want to use compiler optimized code. :)

Also, a programmer called Majenko suggested that the Arduino cannot reliably communicate over serial at higher speed than 9600 baud. I'm not sure what to believe here, since I've used much higher speeds - but still, Majenko is a great programmer and has supplied a lot of libraries and corrections, so I figure he/she knows better. If the serial interface used to program the device is different than the actual interface the MCU is using for the pins inside sketches this might explain why the sketch is running great connected to my computer while failing connected to the MCU.

"The Arduino is unable to communicate reliably at higher baud rates due to the clock being unable to accurately divide down to the right kind of values, but the chipKIT boards, because of the higher speed master clock, can do it more accurately. As a result the Arduino sometimes can't communicate through serial with the chipKIT boards. Try running at a lower baud rate (9600)." [Chipkit forum]

9600 baud could possibly be enough for my needs, but I'm not sure what the "damage" to SD-reading and frame building would be. Still, it's something that can be investigated in the future.

Code-bumping

I've installed the sound board and light board now (light cables are still not done thou).

There's a slight ghosting issue present, hopefully it won't be disturbing with the proper lights as I've seen commercial machines do this as well, if you look closely enough. The problem is caused by the weirdest error ever - the lights are running perfectly when connected to a computer and receiving serial input. But when connected to the Chipkit it just goes bonanza. Naturally I suspected the Chipkit first due to recent interrupt changes, but the sound board worked fine on different serial ports and speeds so there's no problem with the serial ports, of that I'm sure.

I realized I was using a rather long (with delayMicroseconds()) interrupt routine on the Arduino and this caused problems with the serial input - but only when connected to another microcontroller... I shorted it down and now it works with the Chipkit, but is instead plagued with ghosting. Which it really shouldn't be doing. It's pretty weird, but for now this "workaround" will do great, but I'll  revisit this in the future if it turns out to be a proper problem. 

I've also programmed highscore saving/loading as well as proper settings saving/loading and a hardware alert routine during boot. The servos started acting up as a result of the interrupt driven display so I replaced the interrupt servo routine with a software based one instead. Seems to work well, except for an occasional servo "tick" now and then on the first out of two servos. This can probably be avoided by doing some magic in the library-code, as I suspect it's dropping due to a timer reset or similar. Quite possibly a dummy servo can be configured to take the hit instead...

Overall the machine is starting to feel really slick now and the software is bumped up to version 0.85!

Major Code and Sgt Refactor

So...
I haven't gotten the motivation yet (damn weather!) for soldering the new light cables, but I have spent some time with the code the last days. Figured I could do something better than what was in place.

Said and done -
Previously the code was run sequentially and always in the same order. This has now been changed to be done via interrupts instead. I am drawing line by line, layer by layer, and it works great!

I've also sacrificed a little RAM for speed/comfort by adding a backbuffer. While the display is blazing away as fast as it can from the drawbuffer (i.e frontbuffer), all new drawings occur on the backbuffer. When the buffer is finished it simply redirects the current line-pointer to the other buffer and the display never knew what happened. I've tried this in the past, but I never got the flickering to go away. I realized I was going about it the wrong way - there's no need to restart the drawing from the top whenever the content changes. I'm already drawing the screen line by line times eight so nobody would notice really. I was also previously stalling the rendering for buffer creation due to the sequential handling and that caused flickering too. But by avoiding the above and rendering with double buffers, interrupts has given me a rock solid 60 fps screen that is flicker free and brighter than it's ever been. Best of all - it's fixed fps, so it will never dip below this. Ever.  

I've also moved solenoid and switch handling into the very same line-drawing-routine, except that I only process information from these once 32 lines has been drawn (i.e at vertical blank, like the good old days). This has almost given me a 100-200x frequency on I/O without causing any flickering!

This is extremely good news as this means less chance of a missed switch hit or a solenoid firing too long. Previously, if the rendering and SD-card loading took 30ms, that was 30ms that couldn't be spent on anything - i.e a switch hit would not be registered during that time. That is remedied now thou and just to be extra sure I rewrote the switch code to be 4-5x faster as well. Turned out that piece of code still used the old and slow digitalWrite-code, but I took care of that. 

Furthermore - with the changes above the logic loop didn't have to be running as fast as possible. So I've locked the game logic to 40Hz, including SD-card reading, buffer creation, transitions etc. I did a little counting and found that it never dipped below 42 Hz (this is all done on the machine's "spare time" now, mind you), so I lowered it a little extra for safety. A fixed rate means no surprises to game logic and the frame is guaranteed to be finished in time. Should it not be, however, the double buffering will prevent any flickering - which is always nice!

For the record.
I've tried using interruptTransfers in the DSPI-library, but they refuse to work for me due to trouble with the Max32 board definition and conflicting hardware vectors. Ironically, I'm the one who reported the bug in 2013 and it hasn't (to my understanding) been fixed yet...

Anyhow - had that been fixed I could have load animation data and sending data to the display "in the background" and go about my business the usual way. My way works just the same, but different.

Finally, I wrote a script to remove the initial two bytes of junk from wav-files. However, as I didn't read up on the wav-format, the files ended up being the correct size but not valid wav-files. Instead of working around it I simply powered through and converted the 636 sound files (will they all be used anyway?!) manually. That was joyful, for sure, but now the audio seems to be working fine. Still need to convert the old audio routines into new ones and decide on a sound priority/scheduling scheme.

Baby steps...

Comparison

I found this pretty interesting -
This is a (blurry) shot of the inside of a Briarwood Aspen pinball machine. That's a commercial game from late 70's.

Behold!

1) Briarwood Aspen inside. Not much in there to be honest...

Compared to my own pinball that machine looks like the homebrewed one... 

2) My own BioShock pinball. The cabling is arguably neater in the BA machine thou.
But in my defense, so was mine once... :)

On the other hand - had I been making an late 70's/early 80's machine I'd be done by now....

Two Bytes

Two bytes were enough to break the WAV-Triggers loading...

When I did the export of the WAV-files I pressed the "Clear metadata"-button in Audacity. It seems this buttonpress added a "no metadata"-flag, or something like that and thus causing the file to not function in the WAV Trigger.

When exporting the file again and simply pressing "Export" the file is 2 bytes smaller and works.

I'm currently toying with the idea of either exporting all files manually or doing a crude batch script to remove the first two bytes of all sound files...

Pause!

I've been busy with my kids and family lately so not much has happened on the machine, I'm afraid. 

I tried the WAV Trigger, which for some reason doesn't play audio. But I'm pretty sure the problem is in the SD card - OSX has a nasty habit of putting a hidden folder on the card whenever a deletion occurs and that probably interfered with the indexing of the WAV-files. I'll have to look into this - once summer ends... :)

Until next time!

Quick update!

Things are moving along!

Had some trouble getting it to run and I looked all over the board for errors... then I realized I'd simply entered the wrong pin order in the sketch... That's what you get when creating a program before the hardware is made. :)

1) The lightboard works and the light update routine has been written! The last row is blank simply because that power line has not yet been connected. Also - a special "bonus" is getting its programming done as well...

The old Mega-board has made its comeback. For some reason the Nano-board has thrown in the towel and the 5V-pin measurs only 2+V. On top of that the serial port does not get recognized by the computer, so I figure it's broken. But I'll drop in the Mega as a replacement rather than purchasing (yet) another board. :)

Lights. Aaaw yis!

And it's completed! 

1) The finished lightboard, all populated and ready to go! This is easily my most aesthetically pleasing and most well thought through circuit board I've made. I'm also pretty satisfied with the fact that I've more or less never touched electronics on this level before starting the build. There are two power chains, 5+ and 12+ (or whatever I decide to hook up), and I've put 2A fuses on each power chain. My plan is to run the device with all lights active and in case the fuses burn out I'll increase the fuse rating. I'm not putting it back untested again...

I've spent the evening finishing up the lightboard (yesterday was spent on upgrading the WAV Trigger firmware) and it turned out great! Unless I've forgotten a cable or two in the logic channels it's ready to be put back into the machine. I'll probably develop the new required light code out of the machine first (on the Arduino board) so that I don't have to hunch more than necessary. :)

 The only thing I'm slightly worried about is burning out the last row of the LED-matrices, since the current limiting resistor is the same as the others. I haven't done the math for those yet, but I kind of hope that the green ones will simply be dimmer and the red ones within safe margins...

2) Green lights on the top seven rows...

3) ... and red lights on the bottom row, visualizing the higher voltage power chain.

Once it's tested and ready to go, there's only the small matter of soldering new cables to each light. It's around 60 meters of cable to cut, rig and solder.

Matrices be matricing!

1) Getting closer...

Most components are in place now, just need to connect the dots basically. Missing are the fuse boxes and the actual shift registers. The underside looks horrendeous thou, but there's basically no other way to do it with experiment cards... It works perfectly, and that's the important part. :)

2) The flip side. Ginormous resistors are ginormous... Behind them there's a ground "bridge" and one is visible in the top right as well. Soldering islands together like this is obviously not the best solution, but it works and works pretty well. "Next time" I'm building something I'll have the PCB's prefab'ed. It's cheap enough and saves a ton of time!

Lights are coming!

Quick update today.

This is the current state of the second revision of the lightboard. Much neater design, although there's been two mishaps. The first is that I accidentally ordered much too beefy resistors, so they're too big for the initial design and I've been forced to put some of them under the board. The second thing is that the LED-modules were almost twice as big as expected and will be sticking out from the board.

But it's all good, I believe.

1) Semi-populated lightboard. This time I went for a much more streamlined design, along with "seats" for the IC circuits
which will avoid a lot of pain when/if needed to be replaced. The cable connectors has since the picture was taken been adapted to be unique so that the cables "can't" be plugged in incorrectly. Unless I've made a fatal design error, this board will be a lot more stable than the old and far easier to debug, troubleshoot and correct!

74 000

74 000 video frames has been compiled into a single video master file today.
Who knew a pinball machine could have that many frames of animation... but then again, this machine is not your average run of the mill machine. It's truly one of a kind. ;)

I'll try to get started on the new lightboard this weekend. Work's been going real slow on that one so far with the flu, tour and time off needed to regain some energy. But soon I hope I'll be able to present some nice news for you all!

Just checking in!

Just wanted to say I'm still alive and well!


Currently very busy with the band and touring so there will be no updates until (most likely) June.
In case you want to listen to death metal music, or discuss my build, feel free to swing by any of the dates below! :) 

Kill me now...

I've been sick with influenza the last two weeks.
That means nothing has been done on the machine and I'm still waiting for life to return to me.

To be continued.

America's Most Haunted

Benjamin Heckendorn, the person who got me started with my pinball machine with his Bill Paxton pinball, returns with a making of blog about his latest game America's Most Haunted. It's co-designed by legend John Popadiuk and looks pretty cool.

Check it out here!

http://www.benheck.com/americas-most-haunted-making-of/


Also -
I've received the parts for my light board as well as my WAV Trigger, and I've rendered most (if not all) of the videos I need for the game. So this weekend will be all about BioShock for me! :)

Lightboard V2, revision B

Slight update.

Turns out the LoL Shield was charlieplexed, which is rather unsuitable for my needs and I really don't wanna spend the extra processing on the debugging tools. I have therefor decided to insert two 8x8 dual-color LED-modules instead, which is even better actually as they work exactly the same as the light-matrix I'm building - simply extend the wires a bit.

I've also figured out it would be nice with a simple and quick push-to-test button. The plan is to step through all lights, column by column when pressed and held down. This allows quick and easy troubleshooting of broken bulbs or chains without having to interface with the menus, and I could do this with the playfield up as well.

1) Revision B. Tactile button and multicolor LED's for troubleshooting added. The LED's for each lamp will be static
but the flashers/higher voltage things will be using red and the lower voltage green. There's also empty pins to ensure the
cables are inserted correctly, however I believe nothing happens if plugged the wrong way - except that it won't work.

P or N...

Lucky for me, I noticed that I'm all out of P-channel MOSFET's... need to order some!

Someone told not to quote him or her on this, but it seems true enough:

"When triggered, a p channel mosfet connects the input on the load to the positive source whereas an n channel mosfet connects the output from the load to ground or the negative source."

This means my N-channel MOSFET's doesn't work as a current supplying unit and would have rendered the whole device "useless"... The pin layout is also switched (source becomes drain and vice versa) so I need to redesign the board slightly.

Things you notice when trying stuff on a bread board! ;)

Lightboard V2...in the making.

First draft of the lightboard:

1) Lightboard v2. Will be matrix-driven and hopefully a lot more stable/cleaner/better than v1!

I'm using a LoL-board for the light matrix (this is only debug info) as it's almost perfect in size, and saves me a ton of soldering. I've also decided to put in a few fuses on the board to hopefully prevent things from burning up in the case of a short.

No error checking has been done yet. And I'll have to confirm the diode placements, but it's looking a lot cleaner than the old version. I'll use ten different wire colors as well, so I should be able to debug light problems quite easily in the future.

Enter the Matrix....

Semi-success! And fail! 

I've moved all light processing into a separate Arduino Nano board that I had lying around and it worked a treat. Rock solid lighting and freeing up resources on the Chipkit for gameplay!

But I accidentally loaded the lights in the wrong order, enabled some lines that probably should have stayed off (most likely shorted ones... Maybe they were the problem with the lightboard in the first place?) so now the lightboard is once again acting up, and it's going in the dumpster instead.


Two options exists:

1) Recreate the board, twice the scale but still with 96 "inputs". Expensive and very hard to solder without making mistakes, even with the larger spacing. Also, did I mention it's expensive? If I need to add a resistor, make that 96 resistors. MOSFET's? Make that 96 of 'em...

2) Create a matrix board instead. A LOT cheaper and quite easy to solder. I'll be able to visually debug the lines by using led's on the row and column MOSFETs. The downside is that I'll have to re-do ALL the lights wiring, at least most of it anyway. Each light needs to get a new ground and power line. The old board sunk current, while this will be providing instead.

I'm thinking about doing it the proper way now with a matrix, since that's how real machines does it. It also allows for easier troubleshooting and replacing of bad components. But I'll probably won't go down the PCB route this time either, the risk of me making a digital error in the drawing as a first-time user is too big. But we'll see...


Oh, and I'll be replacing the three MP3 Triggers with a single WAV Trigger.
The follow-up product is the same size, but allows for 14 simultaneously mixed/individually running stereo tracks and a bank of 1000 files, including individual settings for crossfades, looping, volume etc per track.  This will allow me to properly play sound and music instead of continuously compromising which effect should have priority over the others etc.

It will also free up two hardware serial lines (one's already in use by the Nano) for additional things, such as a hardware clock countdown for timed modes etc. Great stuff! 

Take That! No, not the boyband.

1) Newly soldered and pristine power board.

Take that, stupid machine.
The new power board is wired up and good to go!
Now all I've got to do is make sure nothing breaks, again, and figure out the source of the misfiring MP3's and I'm good to continue with the rules.

As for the misfires, I got two leads - interrupts (won't make the same mistake twice!) and possibly lack of pull-down resistors on the rx/tx pins. I noticed that sometimes the boards are restarting when many lights are triggered simultaneously, so it could possibly also be a power routing failure, i.e power taking a wrong turn somewhere, or not enough power from the power supply. The latter would however be unlikely as it should be able to provide plenty of amps for everything, so it's probably the actual toggling that somehow interferes with the audio playing.

Misery loves company!

While waiting for the parts to arrive I figured I could continue working on the ruleset. 
But as it turns out, I've been spending my time trying to figure out why half the lights suddenly doesn't work anymore. After tinkering with the lightboard (Satans lightboard, remember...) I managed to get almost all of the lights running. After re-setting one loose lamp, there were seven still not functioning, but I managed to salvage and free new connections for them by grouping together GI light chains. A minor sacrifice, since most GI is on or off at the same time.

I found two faults, probably caused by me pulling the playfield out during the "Fiery fumes of Hell"-episode of yesterdays': One being a resistors legs touching a leg of the 10'th shift register. It was simply too long and got bent by a cable. The other was a loose ground connector on the 2'nd shifter register. It seems that I must have missed soldering it completely or it somehow has heated up enough to melt the soldering at that point. I'm still leaning towards manufacturing a PCB for the lightboard...

I'm also almost positive the sound boards run better when the whole light update-function is disabled. It's really disturbing me since the sound effects are erratic and quite unreliable at the moment. Maybe not for a casual player perhaps, but as the designer I notice right away when effects are missing. Gotta investigate this more closely.

1) Mr Bubbles, Mr Bubbles, does whatever a Bubble-man does!

Mr Bubbles had a little reconstructive surgery as well. I decided to go for function over form and drilled and screwed his arm in place. Hot glue just wasn't kicking it, but this solution - however ugly - works perfect.

The plan was to get a nicer screw that fits better, but that's a future thing to worry about.






I'll end this post with a nice reminder of the "dark side" of the playfield. 

2) This is starting to become impossible to troubleshoot. Not enough documentation, wires that has been redirected
all over the place and no real logical grouping of cables.