Should have learned by now...

 ...to not say that things are going well. 

So I've been having some weird issues lately, problems that I've sometimes been unable to reproduce and they are seemingly random in behavior. I've mistakenly assumed these have been part of the conversion to using "several threads" described earlier.

But this time I noticed that some weird graphical bugs creeped in and since that code "cannot" do anything wrong (it basically sets a number on or off in a fixed array), I figured it must be a disk/flash issue since the letters are hardcoded . 

1) The M is consistently erroneous despite working fine the upload before etc.  

Then I realized, I hadn't enabled "Verify flash after upload" .. and, yeah. This. 

2) Verification always fails at the same adress on the installed Chipkit. 

Note that the flash is completed, so the machine boots and is playable. But it appears that whatever ends up in that space (this time it was the letter 'M') gets corrupted and could cause issues.

I've been uploading to the Chipkit a lot over the years, so it's not unlikely that the flash has finally failed. I also tested to do an upload and verify against a spare Chipkit MAX32 that I have, and that verified just fine.

I might try to keep uploading and trouble shoot on the broken (?) Chipkit until it completely breaks rather than swapping it now, since I'm working on isolated code segments at the moment.  

Update: 

The verification sometimes do succeed.  Obviously I won't leave things to chance later on, but for development of game scenes and thing that are "fixed" in scope - I can manage. 

Achilles, Agony and Ecstasy In Eight Parts

So, I decided to fix the power rails. 

First I removed the old fuses.... 

 

...built a board to host all three PSU's. 

Yes, three. The scratches on the 12V line is me severing that line to allow me to use a 24V PSU instead of 12V. This line also provided power to the Chipkit, so Chipkit got its own jack now too. Real pinball machines strobe their light matrix with 18V, so 24V should provide more vibrant lights and flashers. 

Black is the new ...black, baby! 

Prepared the little power circuit.... 

And removed / accidentally messed the traces when removing a capacitor (more about that later)...

....but finally got everything in place! 

Nice! Looking almost as if was part of the plan. 

The board lights up! 

But, did it work? 

Of course not. 

It's infuriating and fascinating how something can be tested and tested again, and work perfectly in a controlled environment - and yet fail miserably in the real world. Even when loosing my cool and removing a few of the caps (and doing a hack-job at it too....) it wouldn't work. 

So here's Mr Cable once again. At least it's before the fuse this time, although I'm not sure that'll help. 

The big downside is that I cannot start the PSU after I've inserted the USB-cable, so I have to power up the machine/board - then - plug in the USB-cable, or only use the USB-cable. 

To be fair, it does work without the extra cable - eventually. After a few attempts and the power has stabilized it's possible to re-init the IC's that fail and they'll work. But it's highly unreliable and even when removing the 5V fuse, the LED still lights up, so power is sourced from somewhere. I don't really know where the power creeping is coming from - it feels unsettling to leave it without the extra cable.

 Anyhow, win some, lose some. Story of this build. :) 

Jinxed-Schminxed ...

So, yeah... 

"unless - epic ninjas forbid - connecting the solenoids mess something up"

Guess what? 

I had messed up.

Almost burnt down the board since +48V pretty much forced all the gates open.

1) That mini-heartattack when you apply power, "all is well?" and suddenly smoke rises from the board
while the solenoids are going tick-tick-tick... 

I had pull-down resistors on the gates, but since the +48V and +5V didn't have a common ground the gates were essentially floating anyway. (Thanks for the help again, Majenko!) It didn't show up in earlier tests since I used +5V and ground from the board itself to test the circuit, but when the other supply was used it didn't work correctly. For some reason I assumed the high voltage power could be completely separated - and I realize how naive that was in retrospect. 

The fix was easy enough after actually finding the error, just a cable between GND & GND and Bob's your uncle...

...until I realized that I've somehow completely forgot to add pull-down resistors to pretty much all other MOSFET's on the board. *sigh*. So I'll have to fix that by soldering on resistors between gate and source even though that probably means putting components on the backside, but...

... my soldering iron broke down.

 
And now I play the waiting game with the Swedish postal service once again.
This build is cursed, I shit you not... 

But besides doing two lame n00bzor 2000-mistakes (which I didn't do when designing the circuits on breadboard, mind you) the machine can now flip balls.

And that's pretty dope. 

I also got two potentiometers in place to (in theory) help me dial in the opto's to only react to the IR-LED and not (as much from) the sun, for those rare Swedish summer days. 

2) Dialed In! ;) 
JJP are pushing out some high quality boards these days. STERN needs to up their game, imho... 

Footnote: 

I've been hunting a few bugs that are seemingly random - perhaps they too are caused by the floating pins. Would also be interesting to see if the +5V and +3.3V cables are required once all floating pins are properly grounded.  

 

I Dream In Infrared...

 F**king finally - opto's are online! 

Pardon the french, but I've been troubleshooting the optos for 6+ months without success, and yesterday the solution came to me lying half-awake in bed... It's stupidly simple, has a few downsides - but it works. 

After a few months of this...

1) Breadboard horror...

I realized (but not sure why) there was no power reaching the +3.3V line that powers the opto's. Come to think of it, pretty much all +3.3V devices are powered by the Chipkit already... hmm hmm. 

Turns out it was easier to create a new circuit on a breadboard and use a regular remote to trigger the IR receiver. This allowed me to see exactly what worked, and I could then apply the same method of troubleshooting on the board itself. 

2) Dusty horror....

Anyhow - I suspect a grounding issue (remember way back when I had problems with the +5V line not functioning correctly, or rather some of the MCP23S17 IC's would not work) so I simply took the same approach here. Basically, allowing the Chipkit to force the current the right way, since I suspect they're better at designing PCB's than I... 

And sure enough - we got opto's. 

3-4) Opto off and on. Tried and tested in the VUK, so it works
properly with a ball too and not just some giant hand covering it up.

Bloody brilliant! 

Pretty happy that I didn't had to make any (more) changes to the PCB and/or create external circuitry. This was also the last remaining hardware that needed to be solved, so onwards to creating the actual game!

Still here? 

What do you mean 'downsides'?

Well, yes. 

This change has pretty much rendered the fuses on the +3.3V and +5V lines completely useless as power will be drawn from the Chipkit (also) regardless. I have moved the screen to it's own power line now though, so worst comes to worst - only a few MCP's will burn to a crisp in case of failure. 

I hope. 

It's all the mind

The progress as of lately has been mostly theoretical and the new PCB has been untouched.

Been really busy with life in general while the machine has been thoroughly playtested during parties, birthdays and integrity checked by my kids. It holds up, mostly, but there are weird bugs that I believe cannot be fixed in the current state. Weird stuff happens and cross-circuit spikes cause things to trigger wrongfully and what not, and for some reason you can sometimes no longer drain the ball when you're on the second ball. This occurs at random, and when it happens totally unrelated buttons stop working, such as the Start-button, and the debug mode for the lights starts etc. Wiring is alright, but as I wrote, I believe cross-talk or voltage spikes are to blame.

Anyhow;
I'm looking forward to four sweet weeks of uninterrupted work on the machine, which is when the new motherboard will be transplanted. Hopefully everything checks out and I can finish up the new wiring as well. Really looking forward to tidying up everything, getting the new code and video in place and get on with actual game programming!

Thoughts for another day;
Thinking about making the playfield artwork brighter, possibly remaking the artwork. But that'll be done once this playfield has gone fubar.

Let's introduce our little friend; Little Mr Duty Cycle

Forgot all about the LED's being duty cycled while being multiplexed.

8 rows active (~12.5% duty cycle):

1) 12.5% duty cycle, quite dim, but still alright with LED's. Incandescent bulbs barely light up.

1 row active (100% duty cycle):

2) 100% duty cycle. Everything works peachy, including incandescent bulbs.

When all rows are active the voltage is roughly 1.2V, thanks to the ghosting - otherwise it would be lower. This is with 5V input. To get it as bright as it should be I think I'd need to go up to 30-48V for 5V actually, which requires separating the logic chain voltage from the drive voltage. 

And even weirder - the LED doesn't light up much at all when using the 12V line, so it looks like something is wrong there (probably programmatically, as that row requires special handling etc).

Welcome to nightmareland. :/

Weird Science

Ok, so I've found the problem with the lightboard after extensive search - When using two different power voltages (sources?) it doesn't work.

Why?
I honestly don't know.

I've double checked the hardware and rewritten the software, nothing seems to make a difference. I've done no change in the circuit, but simply removing the +12V and attaching a cable from the +5V input to the +12V makes it work perfectly. The both grounds are tied together as well and has always been. So the circuit itself should be alright, I guess?

My best guess now is that somehow power leaks into the "on-switch" of the 12V MOSFET since it's not connected to the same power chain as the rest normally, and therefor possibly allows the current to flow differently there. Putting diodes could possibly remedy it. But there's really no room on the board for extra parts, meaning some ugly Frankensteining of parts on top of parts etc.

I'll keep looking. Nobody wants weak 5V flashers... :C

Edit - 

Ok, I think I'm on to something here. And if so, it's a hardware error by design (by me). 
I'm connecting the gate to ground to keep it from floating, but that is how it's done for N-Channel MOSFET's, not P-Channel as those are active when grounded/low. So that'll have to change and will probably help a little with the ever so slight ghosting issue. 

Secondly; 
I thought the VGS value of a MOSFET was some kind of fixed requirement for turning it on. Well, it is - but it also depends on the load. So in this case when using logic level voltages (i.e up to 5V) the gate can be properly held shut by the shift register as the voltage is less or exactly the same as the gate voltage. In comes 12V and I'm suddenly short 7V to properly close the gate using my 5V, and the circuit is held open. Imagine closing a door standing in a river compared to closing a door in a shower, the pressure of the river is just to big to properly close the door and water will leak through etc. 

So I need to first fix the resistors from pull-down to pull-up, and then I guess add a N-Channel MOSFET on top of the P-Channel to act as a valve for opening/closing the P-Channel and thus the heavier load. I'm so glad I've moved the board to an easier to reach location!

1) Colors are off and the light numbers doesn't correspond with the correct channels yet, but you can clearly see there's no wrongful "double" triggering here.

Shame! Shame! Shame!

My plan failed - I have not been able to work on the pin as much as I wanted to.
Music, life, family and a lot of other cool things has simply been taking all my time.
But fear not; Soon the new workflow shall rise from the ashes of the old! ;)

Also; Game of Thrones.
Last two episodes were killer! \m/

They call him jinxed.

I was just about to start soldering on all the diodes and....
My soldering iron decided to die on me.

At this point of "two steps forward, one step back" I just got to ask myself - How badly jinxed am I?
Seriously, does that even happen to people?

A replacement soldering iron has been ordered so now I'm waiting for that to arrive. :(

1) Fresh diodes! Get your diodes here. 

We meet again, Captain Setback!

I was having a great time soldering new light cables and everything went smoothly... I thought.

1) I was happy. Until I tested the circuits...

Introducing, in the red corner - The Problem.

2) A typical layout with two power lines and two ground lines.

In the picture above, connecting Power 1 and Ground 1 should light up Lamp 1. The problem is that since I don't use LED's, which are single direction only, but lamps/bulbs in most locations, powering that line will also light up Lamp 2, 4 and 3 (in that order). Basically it's a grid of interconnected lines so "everything" lights up simultaneously. Doh.

...and in the blue corner - The Solution(s).

I planned the matrix layout based on LED's, so the easiest solution would be to simply replace all bulbs with LED's instead. That is rather expensive however and I really want to keep it "lamp-agnostic" in order to use whatever lamp suits the most.

Another possible solution would be to use current restricting diodes on each ground and power line. Slightly less expensive and should be quite easy.

The third - and here's where my hope lies - possibly solution: Using the light board's built in current restricting diodes on each ground and power line. However I'm not fluent enough in electrics to know if a single diode at the end / start of each line would be enough to keep current going in the right direction all over the playfield. I only know this once everything is hooked up, and I have roughly half of the ground cables left to solder, and in case I need to adjust something I'd rather do it now than retrace my steps later.


The million dollar question: 
Is it enough to "correct" the current direction at the start and end of each line, even if the cables are not drawn in series, but rather branch out every now and then? Would the, say, blue chain above benefit from a diode between Lamp 2 and Ground 2? Would the diode stop that from happening since that line is still connected to the diode?

Or would the current pass on from Lamp 2 to Lamp 4 regardless?

Edit:
I've found a great tool online to troubleshoot circuits. Using this (which I assume is rather correct) I can see that it won't work with a single diode. The current would still pass in the other "legs".
I will see what the best solution would be, but right now I must say changing to all LED's would be preferable... :/

Check for yourselves here:
Lamp/Bulb based:  http://tinyurl.com/jk38dwm
LED based: http://tinyurl.com/gvbssy2

Kill me now...

I've been sick with influenza the last two weeks.
That means nothing has been done on the machine and I'm still waiting for life to return to me.

To be continued.

Enter the Matrix....

Semi-success! And fail! 

I've moved all light processing into a separate Arduino Nano board that I had lying around and it worked a treat. Rock solid lighting and freeing up resources on the Chipkit for gameplay!

But I accidentally loaded the lights in the wrong order, enabled some lines that probably should have stayed off (most likely shorted ones... Maybe they were the problem with the lightboard in the first place?) so now the lightboard is once again acting up, and it's going in the dumpster instead.


Two options exists:

1) Recreate the board, twice the scale but still with 96 "inputs". Expensive and very hard to solder without making mistakes, even with the larger spacing. Also, did I mention it's expensive? If I need to add a resistor, make that 96 resistors. MOSFET's? Make that 96 of 'em...

2) Create a matrix board instead. A LOT cheaper and quite easy to solder. I'll be able to visually debug the lines by using led's on the row and column MOSFETs. The downside is that I'll have to re-do ALL the lights wiring, at least most of it anyway. Each light needs to get a new ground and power line. The old board sunk current, while this will be providing instead.

I'm thinking about doing it the proper way now with a matrix, since that's how real machines does it. It also allows for easier troubleshooting and replacing of bad components. But I'll probably won't go down the PCB route this time either, the risk of me making a digital error in the drawing as a first-time user is too big. But we'll see...


Oh, and I'll be replacing the three MP3 Triggers with a single WAV Trigger.
The follow-up product is the same size, but allows for 14 simultaneously mixed/individually running stereo tracks and a bank of 1000 files, including individual settings for crossfades, looping, volume etc per track.  This will allow me to properly play sound and music instead of continuously compromising which effect should have priority over the others etc.

It will also free up two hardware serial lines (one's already in use by the Nano) for additional things, such as a hardware clock countdown for timed modes etc. Great stuff! 

Misery loves company!

While waiting for the parts to arrive I figured I could continue working on the ruleset. 
But as it turns out, I've been spending my time trying to figure out why half the lights suddenly doesn't work anymore. After tinkering with the lightboard (Satans lightboard, remember...) I managed to get almost all of the lights running. After re-setting one loose lamp, there were seven still not functioning, but I managed to salvage and free new connections for them by grouping together GI light chains. A minor sacrifice, since most GI is on or off at the same time.

I found two faults, probably caused by me pulling the playfield out during the "Fiery fumes of Hell"-episode of yesterdays': One being a resistors legs touching a leg of the 10'th shift register. It was simply too long and got bent by a cable. The other was a loose ground connector on the 2'nd shifter register. It seems that I must have missed soldering it completely or it somehow has heated up enough to melt the soldering at that point. I'm still leaning towards manufacturing a PCB for the lightboard...

I'm also almost positive the sound boards run better when the whole light update-function is disabled. It's really disturbing me since the sound effects are erratic and quite unreliable at the moment. Maybe not for a casual player perhaps, but as the designer I notice right away when effects are missing. Gotta investigate this more closely.

1) Mr Bubbles, Mr Bubbles, does whatever a Bubble-man does!

Mr Bubbles had a little reconstructive surgery as well. I decided to go for function over form and drilled and screwed his arm in place. Hot glue just wasn't kicking it, but this solution - however ugly - works perfect.

The plan was to get a nicer screw that fits better, but that's a future thing to worry about.






I'll end this post with a nice reminder of the "dark side" of the playfield. 

2) This is starting to become impossible to troubleshoot. Not enough documentation, wires that has been redirected
all over the place and no real logical grouping of cables. 

Of(f) course...

As always, just when things are moving forward they turn to a screeching halt again.

For some time the MP3-triggers has behaved badly and "misfired" when playing MP3's. I thought this could be due to the solenoid-interrupt stepping in and messing with the communication. Should be easy to check, just remove the interrupts temporarily, right?

Wrong. 

What happened was that I've coded the solenoids to fire right away (for speed) and be turned off in the... that's right, the interrupt function. This lead to the solenoid being stuck in active and took me a while to notice since I was debugging code on the computer at the same time. Things that happen when you leave a project for some time, it's almost impossible to remember all decisions that were taken and for what reason. After that I heard what almost sounded like gravel dropping on the floor and thought "what the hell is that?".

Then came the smoke.

I moved faster than Mick Jagger on speed and shut off the machine, threw the glass front off and pulled the board up. I expected to see fire, but there were just one sizzling solenoid. I had to keep it cool by blowing air on it for a good long time. Phew.

Damage control.

1) Sizzling solenoids, anyone? I've heard they're all the rage...

The first thing was obvious. The drain kick solenoid was more fried than a bucket of KFC. Clocking in at almost 0Ω it was basically a short circuit.

Next thing - not a single solenoid worked after the incident. They seemed fine resistance-wise and didn't feel warm, so I suspected the power supply and check the voltage. It read 6V... not good. Should be 48V. After unplugging the short circuit solenoid the voltage went up to 48V again. Thank you, China manufacturer, for voltage limiting when shorting instead of blowing up!

2) New. Crispy.

I replaced the solenoid with a spare one I just happened to have in my "garbage" pile. (An exact match too. Seriously!?). Still no tango....

3) Spot the burnt MOSFET....

...because of course, the solenoid driver board MOSFET was blown as well.
I looked high and low for a replacement, but could only find IRF640's. I used these in the past and they worked, but kind of slow for the flippers so I scrapped them. They also became quite hot due to not opening fully when activated (not really a low-level voltage MOSFET).

The board seemed fine, so I soldered a IRF640 on there and included a heat shield. How about now?

4) Surely this has got to be it, right?

Nope.

Nothing.

Crap.

My theory is that the last time I used them was with the Ardunio Mega, which runs at 5V 40mA per output. The Chipkit has 3V and 20mA, if I'm not mistaken. This is simply to weak to open the gate at all. I raised the on-time with 10x and it still didn't work, so I once again scrapped the idea of using them.

I've now ordered a new power driver board (just in case) and a bunch of replacement MOSFET's (RFP30N06LE). It GOT to work.

On the plus side...
I've created the Atlas, Big Daddy and Upgrade game modes now, along with additional tweaks and things (like user selectable background music before launching the ball). Just got to get the MP3's to trigger reliably, as said, but Big Daddy is even now a treat to shoot at! :) 

5) Junkyard/graveyard. I've grown used to stuff not working, but this is getting ridiculous. 

Days of future past...

In the essence of 'two steps forward, one step back' I've spent most of the day figuring out why all of a sudden some switches stopped working while playing a game but working flawlessly in the diagnostics menu.

Turns out I did a little sloppy coding that overrode the later functions...Whoops.
All fixed now thou! 

I've also replaced the switch in the center VUK to one that should be more robust. The downside is that it can only detect if a ball is entering the VUK, not if there's a ball in the VUK. This shouldn't be a problem if the VUK functions normally and if it doesn't - the ball search will eject the ball anyway.

While working continuously on the machine it's somehow second nature, but being away from the hardware a while really gives you respect for the amount of wiring and work that has gone into the machine. When time comes to do hardware maintenance it's really a case of inventing the wheel once again. And this is despite all my attempts to keep everything neat and organized / documented.

Just a reminder of what it looks like under the hood...

1) Oh, the joy of troubleshooting this... I still don't know why the magnet works now and why it didn't a while ago. I just close my eyes and pray that it works every time I've tinkered with the playfield.

...which is somewhat forgotten when the hood is closed.

2) Players view. Somewhat. At this point, with the glass and all - it really does look commercially made!
The translite is, obviously, not a very tight fit but I intend to put some small edges around to prevent light from leaking out. It's also missing the metal side protectors at this point.

It's a beaut', ain't it! ;) 

Status update!

All right.

So far I've got everything but the plastic transport from ramp-to-VUK in place, including servos, motors, lights, switches and solenoids. Meaning, I've more or less finished the machine.

What I didn't expect (well, should have...) was that my crappy soldering of the Chipkit USB port would come loose, so now I can't program the board. I'm rather fed up with it so I'll just order a new Chipkit Max32, move wire by wire to the new board and hopefully don't break anything else in the process.

Before moving the USB cable, and ultimately breaking the port, I managed to get the servos and motor up and running in code, as well as the accompanying diagnostics menu for them.

Here's some videos for a change!

Fantastic!

In the videos everything just goes on/off in an instant. The final version of the code will of course feature speed control and less jerky, smoother movements. The idle light sequences are just placeholders as well (i.e random lamp lights up) and will obviously be replaced when the new board is up and running.


I've also noticed that the magnet doesn't work anymore, and by measuring the port it would seem that the port is indeed broken (regardless what I've previously said...)

So, up next is:

1) Making the transport section. Don't know why I've not done this part already...
2) Checking the magnet driver port. The port is broken, will order a new board for this unless repairable.
3) Order and replace the Chipkit.
4) Beautify the playfield - attach top plastic overlays. Not critical and might be skipped completely, actually.
5) Tilt blobbin'. Also optional. We'll see about this.

Then - I can finally start coding the ruleset!

Holy smokes, Batman!

Got use for my fuse-box in the pinball the other day...

Yes, I know I should not be doing stuff on the machine while it's on and active.

Yes, I know I should turn it off before disconnecting wires or reconnecting them.

Despite this I always manage to break something by doing just that...
This time the fuses took the hit and saved me from burning most of my electronics. It's rated at 8A so imagine what damage could have been caused if that would have passed through the Chipkit instead.

Yikes.

Son of a...

I'm pretty sure either the machine is jinxed, or I am.
Was doing a quick adjustment and fried the entire row of switches on the right side of the playfield.
Joy to the world.

The worst part is - I've done this two times before!
"I better turn of the power this time!", and so I did. Did a quick fix and turned the power back on, realized I needed to do another quick fix and BOOM. Short circuit.

I guess I should be thankful the switchboard took the hit and not the CPU, myself or anything else that's hard or expensive to repair! Knowing what I know now I should have made the IC's buffer/overload protected or easily replaced by using sockets. Again, next time.

When doing a quick damage control I noticed that one of the solenoid cables had come loose, so I figure I'll do a full go-through on the machine in a couple of days.

Cables...cables everywhere.

Sometimes I really annoy myself, or rather become annoyed by my "I can do this myself"-attitude.
Like the light board for instance - why, oh why, didn't I just order this pre-made on the internet?

1) Satan's lightboard. This is a painful reminder why the invention of circuit boards came to be.

What you see on the picture is a gazillion solder points and A QUARTER of the cables needed to complete it. After that, some serious testing and hopefully at least some of them function... Right now I don't remember if I have any light slots to spare, so hopefully they'll all work. I hope.

I thought about both remaking it using a matrix instead or simply design it in CAD and have it manufactured instead. Both things cost money thou, but at the moment I don't really care...
No matter how hard I try, all I see when I look at the board is this...

Source: http://www.wired.com/beyond_the_beyond/2008/03/ 

Major Setback, or Sergeant Crap.

I left the USB-cable dangling outside the pinball machine, still connected to the Chipkit MCU.
Then my daughter found it.
And pulled it.

Now I've got to get myself a new Chipkit.