Achilles, Agony and Ecstasy In Eight Parts

So, I decided to fix the power rails. 

First I removed the old fuses.... 

 

...built a board to host all three PSU's. 

Yes, three. The scratches on the 12V line is me severing that line to allow me to use a 24V PSU instead of 12V. This line also provided power to the Chipkit, so Chipkit got its own jack now too. Real pinball machines strobe their light matrix with 18V, so 24V should provide more vibrant lights and flashers. 

Black is the new ...black, baby! 

Prepared the little power circuit.... 

And removed / accidentally messed the traces when removing a capacitor (more about that later)...

....but finally got everything in place! 

Nice! Looking almost as if was part of the plan. 

The board lights up! 

But, did it work? 

Of course not. 

It's infuriating and fascinating how something can be tested and tested again, and work perfectly in a controlled environment - and yet fail miserably in the real world. Even when loosing my cool and removing a few of the caps (and doing a hack-job at it too....) it wouldn't work. 

So here's Mr Cable once again. At least it's before the fuse this time, although I'm not sure that'll help. 

The big downside is that I cannot start the PSU after I've inserted the USB-cable, so I have to power up the machine/board - then - plug in the USB-cable, or only use the USB-cable. 

To be fair, it does work without the extra cable - eventually. After a few attempts and the power has stabilized it's possible to re-init the IC's that fail and they'll work. But it's highly unreliable and even when removing the 5V fuse, the LED still lights up, so power is sourced from somewhere. I don't really know where the power creeping is coming from - it feels unsettling to leave it without the extra cable.

 Anyhow, win some, lose some. Story of this build. :) 

MOSFET's... You will never find a more wretched hive of scum and villainy.

So - apparently there's a thing called an ideal diode that can be created using MOSFET's instead. 

For instance, comparing two circuits built in Falstad: 

1) Diodes in series with the power lines, compared to P-Channel MOSFET's (+ protective diodes and resistors)

Even using default values (there are better - and worse - components out there) the P-Channel MOSFET version provides a voltage closer to the source and provide the same features. This MOSFET version should be possible to use in my "vertical fuse holder"-design as well, but require a bit more components and "not as clean setup".

I will have to do some breadboard tests to see which one to use as both will work, but it's nice to keep as much power as possible so I'm leaning towards using the MOSFET one at the moment... 

Beam me up, Schottky!

So, I think I finally figured out - or at least the remedy for - why the extra cables were needed. 

It was pretty much as I expected and that voltage somehow got reversed in the circuit, and me either knowing too little about electronics or expecting too much from the power supplies. Rookie mistake in assuming the power supply would overpower the other components, but it seems that it doesn't. 

Long story short, it would seem that the capacitors I have on the PCB is enough to offset the "power balance" in the board, and by connecting the power lines to the ChipKit's lines I basically hijacked the voltage rectifiers to "heal" the PCB. 

By replacing the fuse with a diode (temporarily) all of my problems goes away: 

1) It ain't safe and it's not pretty, but proof is proof. 
Notice the disconnected +3V3 and +5V cables. 

Oh, fun fact:
Turns out that the +3V3 cable wasn't really needed.
I had damaged the trace on the board while removing the capacitor
in my first feeble attempt to solve the power problem a year ago or so. 

I used diodes to force current this or that way on the board, but forgot/ignored the main lines since I assumed they would be alright coming directly from the PSU. 

So now I'm basically trying to find a way to incorporate diodes (Schottky's) in series with each power line. The voltage drop isn't that bad (-0.48V) but still a lot more than I had hoped to achieve. I want to avoid having several boards and something like the below should probably work, but I could also use a P-channel MOSFET as a very low resistance diode. 

But I'm going to try to use a setup similar to the one at the top in the above image, where the diode would be standing up and connect to a vertical mounted fuse cylinder instead. It should work, and it seems there's enough space. Would be nice to get rid of this pesky power problem once and for all. 

But we'll see. 

Jinxed-Schminxed ...

So, yeah... 

"unless - epic ninjas forbid - connecting the solenoids mess something up"

Guess what? 

I had messed up.

Almost burnt down the board since +48V pretty much forced all the gates open.

1) That mini-heartattack when you apply power, "all is well?" and suddenly smoke rises from the board
while the solenoids are going tick-tick-tick... 

I had pull-down resistors on the gates, but since the +48V and +5V didn't have a common ground the gates were essentially floating anyway. (Thanks for the help again, Majenko!) It didn't show up in earlier tests since I used +5V and ground from the board itself to test the circuit, but when the other supply was used it didn't work correctly. For some reason I assumed the high voltage power could be completely separated - and I realize how naive that was in retrospect. 

The fix was easy enough after actually finding the error, just a cable between GND & GND and Bob's your uncle...

...until I realized that I've somehow completely forgot to add pull-down resistors to pretty much all other MOSFET's on the board. *sigh*. So I'll have to fix that by soldering on resistors between gate and source even though that probably means putting components on the backside, but...

... my soldering iron broke down.

 
And now I play the waiting game with the Swedish postal service once again.
This build is cursed, I shit you not... 

But besides doing two lame n00bzor 2000-mistakes (which I didn't do when designing the circuits on breadboard, mind you) the machine can now flip balls.

And that's pretty dope. 

I also got two potentiometers in place to (in theory) help me dial in the opto's to only react to the IR-LED and not (as much from) the sun, for those rare Swedish summer days. 

2) Dialed In! ;) 
JJP are pushing out some high quality boards these days. STERN needs to up their game, imho... 

Footnote: 

I've been hunting a few bugs that are seemingly random - perhaps they too are caused by the floating pins. Would also be interesting to see if the +5V and +3.3V cables are required once all floating pins are properly grounded.  

 

I Dream In Infrared...

 F**king finally - opto's are online! 

Pardon the french, but I've been troubleshooting the optos for 6+ months without success, and yesterday the solution came to me lying half-awake in bed... It's stupidly simple, has a few downsides - but it works. 

After a few months of this...

1) Breadboard horror...

I realized (but not sure why) there was no power reaching the +3.3V line that powers the opto's. Come to think of it, pretty much all +3.3V devices are powered by the Chipkit already... hmm hmm. 

Turns out it was easier to create a new circuit on a breadboard and use a regular remote to trigger the IR receiver. This allowed me to see exactly what worked, and I could then apply the same method of troubleshooting on the board itself. 

2) Dusty horror....

Anyhow - I suspect a grounding issue (remember way back when I had problems with the +5V line not functioning correctly, or rather some of the MCP23S17 IC's would not work) so I simply took the same approach here. Basically, allowing the Chipkit to force the current the right way, since I suspect they're better at designing PCB's than I... 

And sure enough - we got opto's. 

3-4) Opto off and on. Tried and tested in the VUK, so it works
properly with a ball too and not just some giant hand covering it up.

Bloody brilliant! 

Pretty happy that I didn't had to make any (more) changes to the PCB and/or create external circuitry. This was also the last remaining hardware that needed to be solved, so onwards to creating the actual game!

Still here? 

What do you mean 'downsides'?

Well, yes. 

This change has pretty much rendered the fuses on the +3.3V and +5V lines completely useless as power will be drawn from the Chipkit (also) regardless. I have moved the screen to it's own power line now though, so worst comes to worst - only a few MCP's will burn to a crisp in case of failure. 

I hope. 

Operation Mother

So - finally got my greasy fingers around properly installing the motherboard inside the machine.

Still haven't got my opto's to work but at this moment I'll have to live without them. Will need to debug the circuit and possibly rebuild it separately in the future. 

Like most of the work on this build it's a bit of "one step forward, two steps back" since I was only going to screw the board to the backing plate. But... I forgot that the backing plate was made out of cheap fiber board so screws won't work - and - of course the cables didn't reach the intended position for the board.

So I did what any hobby engineer with(out?) self respect would have done; I cut a big hole inside the frame of the head.

This worked alright, but due to the placement of the intended cut everything turned out a bit crooked. Not that it matters much, since now it's done. But I know my future self will grin at this during any maintenance session, haha...

"Fun fact" - while checking out the playfield, it turns out most of the rubbers have cracked and need replacements. Again. But I guess I kind of was prepared for that and it'll have to wait until the game logic / game rules are completed and the machine is closed up properly.

Other than a few mishaps; the board is now installed and I didn't break anything in the process!

Installed and ready to be developed!

P.S, ignore the double vision display, there's
a small gap between the display and the panel at the moment. 

A Pin Best Servo'd Cold...

Woho, a proper update!

I've rewritten (or rather written) a servo library that piggybacks on the existing I/O loop. The good thing is that there's zero jitter and no "overflow" or "rollover" issues that is present in some libraries, and it seems to work pretty well. Although I had to guesstimate certain values, as the servos require a rather exact timing to function, it should be close enough for my purposes. I expect to trim and tune these parameters once everything is connected properly (like everything else) but it's nice to see the "last bastion" of remaining functions taken care of.

Furthermore, I've finalized solenoid, switch and light functions so they work properly and reliably with callbacks. I think I covered the switches in an older post, but the hits are buffered and executed in the order they were triggered and is put in the outside loop at ~40 Hz. 

For time critical events I've add high-prio callbacks that gets called right away in the inner loop (even before buffering the hit) that will be used by slingshots, flippers and bumpers, to ensure these solenoids gets triggered within the next microsecond. Sound, points and game related callbacks will run as usual in the outside I/O-loop. Hopefully this will, and should, make the game feel super snappy and responsive.

I've started the time consuming work of making the cables needed for connecting the new PCB-board to the pinball machine, and it's a few hundred meter cables that needs to be redone. Obviously I only do the ends of the cables, but still... ;)

So things are rolling once again, which is rather nice!

1) Doesn't exactly show the servos, but at least the colors are nice. ;) 

Remembering The Past

Needed the soundboard from the "old system", so I opened up the pinball machine for the first time in a while... And I was actually surprised to see what a mess it was/is! It's a wonder it even worked at all, haha. :D

Anyhow -
It was quite satisfactory to see that where the old setup used tons of different boards, two MCU's and a cabling from hell, the new setup is ultra neat in comparison. And despite having "only" one MCU, the performance is 4x in most cases, but 200x when it comes to I/O. Hardware stability is improved a lot as well! I'm also pleased with the new code library compared to the old, which will make game programming a breeze.

The maintenance mode is always enabled, which gives me detailed input on sounds running, scenes started, sound and input / output etc - so I can see right away if there's a problem. Love it!

1) The old....monstrosity. This is actually HW3.0, if we're counting revisions.
HW4 (current) is seen in the post below, which despite being in development "mode", still looks tons better. 

Weirdness and wetness!

Spent some more time with the code and got dual animations on a single SD-card up and running!
It's an unlimited number really, but limited to two for performance. The theory is simple, since I'm not using a file system I simply load raw sectors and can grab any frame at will.

For simplicity and synchronicity's sake, I've decided to load SD-card animations with a blocking approach (as opposed to interrupt driven). This greatly simplifies things as I can keep the main loop/scene rendering to a single function instead of spreading out across several frames and wait for various inputs to complete etc. Since switch inputs and all I/O in general is moved outside the main loop, it doesn't matter really. I've locked the main loop to 40Hz which should be plenty for updating animated text, verifying animations and other non critical reasons. 

1) Basic render "geometry" examples with background animation, textlayer and front animation (layered above the text-layer).
The lower left are status flags for SD card, MCP, ULN etc. All 8 should be lit for a 100% functional machine.

The extra FX-layer really spice up the DMD, since I can overlay smoke, sparks and what not at will. I could also use two background layers, or two front etc.

I've also found a way to successfully detect if ULN and MCPs are functioning correctly, and can alert and disable features accordingly. I've completely lacked this troubleshooting feature previously, so this is a major leap forward.

I've also had the "joy" of realizing that the flash-burning sometimes fails, and that verification fails. It has been like that for a while, and I never knew why and it has "always" worked. But this weekend some simple code started to fail seemingly randomly and it basically boiled down to 'if verification fails, the upload isn't correct' after all. Usually a reboot or reselecting the toolkit in UECIDE allow me to upload successfully. As long as I know about the issue I can work around it. It wasn't exactly a straight path trouble shooting it though, since I both suspected that the DMD was broken. I took a rewrite of the routine in a new sketch to see that it was indeed working correctly. A new screen is _not_ something I want to buy at this stage...

But on the plus side the 13h+ troubleshooting made me rewrite SD card and DMD rendering routines, further optimizing them and avoiding unnecessary variable writes and operations. The downside is tough-to-read code, but once in place they won't change often (at all?).

So - three steps forward, one temporary step back.

ULN2803A!

1) Fully populated board with everything up and running. Since this is my first circuit board ever made
and my first venture into advanced electronics, I'm very proud to see it functioning properly!

Yes!
It was a broken ULN2803A that was the problem with the light matrix.
Very satisfied with that little bit of knowledge, as this now means that all parts of the hardware has been checked and verified. Now it's simply a matter of fine-tuning everything and create, solder and connect the final cabling.

Exiting times to say the least! :D 

DJ SD is in the house!

...SD card loading is in! 

The graphics library is still in its infancy, so there's no shadows etc. But all-in-all I'm pleased with the results!

1) DMD with SD-card loading (and displaying) demonstrated. The 16 shades shows the possibilities pretty good here,
with a dark big text in the background, several layers of graphics etc. Once shadows are in place it will be killer!

We shall rebuild!

Getting closer...
I've been struggling getting the light matrix to run properly, since all rows are for some reason active at the same time. I'm thinking the ULN2803a is broken, so I've ordered a replacement. The ULN activates by pulling the row TIP107's to ground, which seems likely in case of a short circuit or similar. We shall see. 

On the other hand - input and output are all up and running and running "very often". I haven't crunched the actual numbers, but it should be in the 500-600Hz range at least. The graphics are a rock solid 60hz (or so, there's no noticeable blinking which would suggesting a minimum of 50hz ) and it doesn't matter if the drawbuffer takes 1 second to create itself, it won't affect solenoids, switches or graphics etc. This means I can most likely use several layers of DMD animations, for instance, one background and one foreground. Should be pretty wicked!

1) Here's a snapshot of my Enterprise Commercial Grade Ultra Deluxe Setup 2000. Still, the various shades
does make a huge impact on the graphics. Looking forward to getting SD card animations up and running!

I'm quite satisfied that there's absolutely no delay anywhere in the code. When the DMD is crunching bits into its registers, the MCU is processing switches/lights etc. The result is a nice double buffered screen in 16 shades and super responsive inputs/outputs. The setup even allows for "autonomous" triggering of slingshot, bumpers and flippers with microsecond delays instead of 15-20 ms previously.

Overall I'm very pleased with the progress thus far!

Turtles....

...are faster than me.

But at least they're not doing a million things at once. Or building pinball machines. :)

Since last I've had some issues with the MCP23S17's, and they didn't want to function properly. I think I got the hang of them now though. There were a couple of issues, for instance;

1) One of the MCP's relied on the Max32 to send an enable signal to it, but it seems the pin can't supply the necessary voltage and/or amperage needed to keep it enabled. I temporarily jumpered it to 3.3V and that solved that. I will probably solder this to the watchdog output instead, since that and the other two MCP's relying on it seem to function properly.

2) The switch matrix MCP displayed the status incorrectly on the DMD. I'm still using a temporary DMD "library" that lets me dot pixels for basic trouble shooting. I forgot to add the clear buffer function, so every draw could only add to the screen and not remove. This fooled me to think that the built in pullups malfunctioned and/or the software for it. I've verified functionality with a LED instead and everything works as intended, and has most likely always done so.

3) I've moved away from all fancy MCP-libraries and communicate directly via SPI, and that probably helped a lot in knowing exactly what's going on. Will encapsulate this into functions optimized for the "thread" handling I've planned for them.

Lastly, I've, err, confirmed that the fuses function properly.
While testing the inputs I did something very wrong (i.e voodoo connecting things) and blew the 5V fuse. It's rated at 3.25A so that's a pretty decent amount of current that was blocked instead of frying other components. 

But at least things are moving forward! :)

Get the jig' with it!

Not a "proper" update, really, but I've attached the PCB, screen and PSU to a spare piece of plywood which allows for easier handling and development. All necessary connectors are reachable as well, so I'll be able to hook up I/O and even solenoids to test things as well.

Since I am developing outside the box (no pun intended) this will help a lot to keep things tidy.

1) Piece of scrap plywood with components attached. Not much to say really. ;)

Pixels be rollin'...

It is with immense satisfaction the first few pixels has been drawn with the new motherboard!

Everything is very basic and the frame is currently hardcoded in 8 shades, but runs well in 16 as well (it's slightly difficult to see without having a proper image though). But it is running the new render routine with interrupt transfers and as a CoreTimer service.

1) Motherboard and DMD, sitting in a tree... The "pulsing" effect on top and bottom rows are simply
8 different shades being drawn. The timing of each color has not yet been adjusted and the screen overlay
is not in place either, which dims pixels and thus make each shade more pronounced.

I'll have to redo the main project file as well, as the one I did the PCB-tests on no longer works. I don't know why and haven't really investigated - now I can implement each section properly and without any guess work instead.

Next I'll get some basic DMD functionality in place before going on to other components.

It's all the mind

The progress as of lately has been mostly theoretical and the new PCB has been untouched.

Been really busy with life in general while the machine has been thoroughly playtested during parties, birthdays and integrity checked by my kids. It holds up, mostly, but there are weird bugs that I believe cannot be fixed in the current state. Weird stuff happens and cross-circuit spikes cause things to trigger wrongfully and what not, and for some reason you can sometimes no longer drain the ball when you're on the second ball. This occurs at random, and when it happens totally unrelated buttons stop working, such as the Start-button, and the debug mode for the lights starts etc. Wiring is alright, but as I wrote, I believe cross-talk or voltage spikes are to blame.

Anyhow;
I'm looking forward to four sweet weeks of uninterrupted work on the machine, which is when the new motherboard will be transplanted. Hopefully everything checks out and I can finish up the new wiring as well. Really looking forward to tidying up everything, getting the new code and video in place and get on with actual game programming!

Thoughts for another day;
Thinking about making the playfield artwork brighter, possibly remaking the artwork. But that'll be done once this playfield has gone fubar.

"If normal at first, measure again"

So...
The board didn't work.

Or at least it didn't before.

My previous post was a false positive and everything started to behave really weird, with negative voltages and what not, as soon as something was connected and plugged in. The power switch was controlling a MOSFET, which in turn disconnected or connected all power lines to GND. For some reason the current flow was really wack because of this. For instance, inserting a LED in the 5V line really shouldn't reroute power from the 3.3V line and vice versa. This was a real pain to troubleshoot, but after butchering the board quite badly (#sadface) I ultimately found the problem (#happyface).

The solution simple enough - just bypass the MOSFET by shorting the drain and source. I really wish that I'd found this simple solution before massacring all capacitors. Now things are finally working correctly with the exception that I cannot turn power on and off with the switch, as I've hoped to do. But it's a minor issue and was more of a nice touch anyway, as it wouldn't be used in production.

1) Everything's looking dandy! For real, this time...

A major give-away, which I can't believe I overlooked, was that the status light previously didn't light up. It's directly connected to pin 13, so it should have blinked during boot. The fact that it never once did (before) would have told me that current was going the wrong way - or was insufficient. But I was simply assuming that I was shorting stuff when in reality it's perfectly fine to measure where and how I did. The wonders of stress...

Another issue leading to "premature optimization" was that I'm using an old Chipkit, which of course had some pins enabled that are now used for MOSFET control. This lead to having the light rows being always on for instance, but since they turn off during reset of the Chipkit they work perfectly fine.

Lesson learned? Don't put in untested parts in the schematics on the basis that "it should work, right?". The power switch was a last minute addition and a huge pain the Arizona.

Capacitors... Crap-acitors?

After investigating I think that the board actually might be working correctly after all (why do I sound surprised?).

A couple of issues -

* I have only tried to power the board via another ChipKit, since I haven't got a spare PSU. The only one I have is in the machine itself.

* The capacitors are "huge" and store a ton of energy compared to it's current consumption, which is more or less nil in its current state. This led me to think that the circuit was powered when not, and also the other way around - turning them on takes a while and is not instant.

* The SD-card reader draws quite a lot of power, and when it was connected (or possibly due to its connection with the ChipKit). Whenever the SD card was inserted, the 3.3V lane "died", i.e all power went to the SD card. As I said, it could also be that the ChipKit is trying to power itself from the SD card data lines etc.

* When attempting to read voltages I must have created a connection with the multimeter and supplying current instead of simply reading it.

So - I removed the SD card (temporarily) and powered up 3.3V and 5V simultaneously with a 3V battery pack. (not ideal, I know). And viola;

1) The connected power lines are active! #celebration
The LED's are protected with resistors for each voltage, so the 5V LED naturally looks dimmer. The green LED's are also diffused instead of clear, and of less brightness naturally than the power and status LED's.

The board look correct and both lines get power. Removing the SD card only instead of the whole connector also works, so there's no short circuit either.

Flipping the power switch to off yields this:

2) Everything is off!
Side note: The yellow capacitor on the SD is not connected to 3.3V and CLK, but 3.3V and GND. It just looks that way. :)

Which is exactly the way it should be - all lines are off!
When putting a LED in the 5V + GND sockets ("unconnected" test-areas) it properly lights up and turn off when supposed to. Only putting power on either line when the switch is off does not power up the LED over +5V + GND, just like it should.

So I'm thinking the capacitors are the main culprit, along with insufficient power to boot the board properly?

I guess the next step would be to arrange a PSU and proper power and take it from there. I'm still taking baby steps since I don't want to fry any components, worst comes to worst. :)

Trouble in paradise!

Still waiting for the final parts... but...

It seems there might be an issue with the power section.
It would seem that current leak from 12/5V into the 3.3V and more or less turn on the circuit even if there's just a single voltage lane (that should be off) active. It could also be that the debug ChipKit I am using, an old broken one, could be outputting current on the power status connection (which will be an input later) enough to open the gate.


As always with electric and digital troubleshooting, the hardest part is usually finding the error. In this case, most likely a well placed diode will remedy the issue. I will look into this tomorrow, but I'm pretty sure it's a minor detail.

On a side note - 
I'm certain the end result will be worth it, and there's really no other way to do it properly than to remove all the old "gunk" and re-do it piece by piece. Properly.




But.
So close, almost there - only to be torn apart again.
It's a bit scary to dismantle the machine...

Quick update from the land of PCB's!

Almost done with the circuit board!

Just the ATX connectors that needs salvaging from other boards and the power sockets that I've forgot (?) to order. Other than that, it should be ready for a check and conversion from old to new.

I'm especially proud of the big capacitor "UPS" for the ChipKit and the accompanying power sense circuit, meaning that the board will run for a few milliseconds after the power disappear, just long enough to save data to the EEPROM and shutdown things neatly. The watchdog will still kill any lingering solenoids, but it won't hurt to properly turn them off.

1) Almost fulled assembled. Sorry for potato quality picture!