Profile cover photo
Profile photo
Mark Mullin
153 followers -
Small chance of success, virtual certainty of death..... what's not to like
Small chance of success, virtual certainty of death..... what's not to like

153 followers
About
Posts

Post has attachment
errmegerd - it works - this is high density lidar scanning with overlapping 2 degree visual field capture

Post has attachment
First a shout out to old Tango friends – yup, I stayed with it, I caught it, and I beat it. That said, it was f**g scary and most of the time I thought I was going to die an ignominious death.

This video is the result of a very long strange trip, starting with when I was invited into the Google Tango project, and then starting in earnest when Google steadfastly refused to do a decent job and I wandered off like a 3-year-old going “Imma do it myself!” I ended up focusing on inexpensive edge AI stereo machine vision lower down the stack than the more commonly encountered segmentation and classification activities, though I now plan on romping about extensively with them using this data.

This shows how my system processes the parallax error it locates in left and right views in a stereo camera. There’s a vicious twist right out of the gate in that the two physical cameras are fused into a single synthetic camera based on the overlapping FoV. It was an interesting day when I discovered you still had to calibrate a synthetic camera, even if it was consuming fully rectified data from the physical cameras.

For every image it uses a bunch of math to calculate the relationships with adjacent off-axis images, about 5 degrees to the right and left, or, each images parallax data is a definite integral from -5 degrees to +5 degrees from the center axis where the camera is currently aimed. Of course, there’s also a homography that has to work inside that integral in order to correctly remap all of the off-axis coordinates. It’s a prickly beast.

Where it measures parallax error with the left camera it paints it blue, and where it measures parallax error with the right camera it paints it red. At the color changes to magenta and becomes more intense, this shows the very high accuracy data, i.e. its seeing it with both cameras from a number of angles and everything keeps agreeing. It processes this data first, and then uses the weaker magenta points and blue and red points like small children in the kitchen – they may be useful, but you can’t quite trust them.

I only use about a 5 degree off-axis range with these cameras, as they’re cheap, thermally noisy and have poor lenses. Penalties of self-funded research, I guess. As you watch the colored pixels shift between red and blue and magenta, you can see more easily how error is cancelled and concordance is merged. One thing to keep in mind is that there’s a true parallax measure to look at where each of those colored pixels is, but to see that in bulk takes a movie of vector fields, and trust me, those can be profoundly disorienting. I don’t need hate mail from people who barfed on their keyboard.

I’m looking forward to making another version where the computers and cameras get jumped up, for example switching the Raspberry Pi’s out for bigger SoCs and giving it a couple of high end Nikon backs and lenses for imagers. Should be very interesting, and the visual resolution and tracking should approach bone-chilling.

One thing I’ve deduced on this journey is that the deep levels of this visual cortex (all this is is the deep levels, Googz is playing in the higher levels and I’ll just stand on their shoulders for that) are probably similar to deep levels of biological stacks at that level. They’re target acquisition and tracking machines. Nothing more, nothing less. They rapidly identify deltas in the stereo image, and they track and combine those. I’ve worked on letting the head run active tracking on me, and it’s an odd experience. Simplifying the math behind it to a ridiculous degree, it’s just running the servos to make me as magenta as it possibly can, and it does a damn good job at it. The only time I managed to get it to loose me I crashed into the tool chest and it broke a servo axle trying to keep up and then stop.

In closing, it’s kind of bittersweet to be delivering this at last and knowing Tango died early this year. It never should have happened, there was talent and skill, but it was unfortunately overwhelmed by ineptness and arrogance. That is one reason I do love working in ML and MV, it is absolutely unforgiving to BS. I’ve got my popcorn stored up for the inevitable implosions to come, as ML ‘players’ ignore this not just to their peril, but to their absolute destruction. ML is incredibly powerful, and it can be fiendishly complex. As to what it will do in any new situation, all you’ve got is a wing and a prayer. If you pay attention, if you are rigorous in your model building, if you don’t contribute to additional error to a system already trying to suppress the error pouring in, victory may be yours. Fail to do that and you will almost certainly die, it will be unpleasant, and it will hurt the entire time it’s happening.

For those going cool, I wanna go do this too, I can offer one item of note. What I didn’t know when I started was exactly how important calibration was. Calibration in this world is the attempt to solve for n^2 unknowns with n equations, only slightly tongue in cheek. Calibration will rule your every waking moment, and much of your dreams as well. If you do not calibrate, then all you do is add more noise, and the whole house of cards keeps falling over and bursting into flames. Success in this space is driven by the quality of your calibration, it is the absolute limit on what you can achieve.

https://youtu.be/byfJ-lt6Eus

Post has attachment
Add a comment...

Post has attachment
Add a comment...

Post has attachment
So my little project morphed over time into 'Stereo Machine Vision Using Mass Produced Imagers", or "How to find meaning in noise" :-) That said, all the code is checked in and available, and we've got an extra special for +Chuck Knowledge - there IS a confidence measure available at the sampled depth points!
Photo

Post has attachment
Add a comment...

Post has attachment
OK, +Chuck Knowledge we're finally making some progress - this is calibration of a single 55mm (about) lens in the new stereo rig. Getting the whole opencv math pipeline to run in a numerically stable fashion was a right treat, the pain yielding the delight of a reprojection error less than image resolution. It works for larger images than this just fine, although it causes the cpu fan to have big long sad for a 1600x1200 image intrinsics calc - mean ol Taylor polynomial. Anyway, now to create the stereo intrinsics, and then yes, I promise - you get a confidence measure with every depth estimate :-) Anyway, here's the raw and rectified for a subtle problem, not just 'oh yeah it doesn't look like a soap bubble anymore'

For those who might not know why I'm poking Chuck, I used to play with Tango. Then I got unhappy and decided I'd rather try on my own with a purely visual solution. The 2nd generation incarnation is based on multiple stereo cameras. All the code thus far for individual stereo camera modules is OSS licensed (I owe a checkin right now :-( ) and you can find the code here and the project logs here.
Photo
Photo
1/20/18
2 Photos - View album

With respect to the “turning down” of Tango support and the announcement that this would continue as an unofficial platform for Tango development going forward under Chuck's leadership, one key question remains. What is Google doing with respect to copyright on the top-level Tango API? I have seen no mention of ceding the API rights to the public domain in any fashion.
I ask this question as an external technology developer, as I could consider adding a Tango compatible API to my camera system. This is of course potentially beneficial to me, and to others that are currently committed to the Tango platform. Trust me, I’m not championing the API for the long term, rather I’m wondering about its short-term viability. In order for it to have any viability at all, then developers have to be able to code against the API, and Google has just made it quite clear that API is dead to them. So they should let interested parties code against it, if there are in fact any interested parties.
The alternative is to make the announcement somewhat duplicitous at best. If Google is turning down support for Tango, and there are active developers extant, then all of the work they have done up to this point is invalidated, they have no road forward at all, and all of their expenses are now both non-recoverable and pointless. I’d really hope this isn’t the case. If Google’s going to give up on Tango, then they should give up on a commercial lock on the high-level API as well, otherwise, as the real owners of this site, they should just do the right thing. Nuke it and admit all the players got screwed.

Post has attachment
OK, lets discuss Google's performance on Boston Dynamics - when they bought the company, BD was pioneering highly mobile hydraulic robots - when der Googz had finished, they left a pitiful wreck behind - pretty much exactly the same story as with Tango

So here's BD's latest that the internet is slavering over - it's now out of google, the reduction in size and apparent increase in efficiency is good, but it's pretty much a smaller version of the same damn thing - maybe they can move forward again as they've escaped from the fool factory, but it's pretty much just another incremental data point at this time --
https://youtu.be/kgaO45SyaO4

Now, let's compare and contrast..... Here's a Brit company, OC Robotics that specializes in elephant trunk robots (or use other scarier names)

https://youtu.be/MaG0p2PWuRI

Now lets look at a German company, Festo

First, they're prototyping and building things that DARPA (who funded BD before Googz told them to f off) are still trying to figure out how to write the funding contracts for
https://www.youtube.com/watch?v=7-JvyzOddTM

and a fish

https://www.youtube.com/watch?v=TKeVVKGEDLc

oh, and they're in the elephant trunk space as well

https://youtu.be/ZPUvA98uSj8

These aren't the only players, but they're my current faves - none of the companies I follow are American - we had a leader when Googz bought it, and now its stuck in a desperate race to catch up again, thanks to Googz.

When I left the Tango program, I knew it had failed, I knew that the team was dysfunctional, the leadership was pitiful, and the strategy was non-existent - I said goodbye nicely, wished everyone the best, and except for a few close compadres, kept my mouth shut - given a recent experience with Google that horrified AMEX card security people and brought Lily Tomlin's "we're the phone company - we don't care, we don't have to" to mind, I've decided I'm angry - they play on what they used to be and what they are is not what they used to be (exceptions yes, I LOVE TENSORFLOW!) and I intend on bitching long and loud - Googz, either get back to what you were or go the f**k into the grave and let us get on with actually trying to not be evil and see what the hell is going to happen next.

So Googz, how does your business model work if I run tensorflow on AWS because I don't trust you enough to run it on your infrastructure ? How do you explain the Tango debacle ? How did you manage to screw the pooch with BD ? You really need to ask yourselves these questions! You really have two choices - fix it, or continue the race to the bottom. Truth be told I want you to fix it. Or I want to know the Google I admired is dead.
Add a comment...

Hmmm - So google has decided to suspect my wallet, provide NO supporting data, and ask for information I will not provide, such as scans of my drivers license - I kept my mouth shut when Tango so utterly screwed the pooch they had to roll out with Lowes as their flagship - no longer - as an organization, google is failing - they failed at tango, they failed at boston dynamics, gmail is turning into a fail, i can only hope they can keep the damn search engine running long enough for someone to pick it up. EVERYTHING is coming off google, to the degree I'm going to be dumping thousands of dollars of media - for those of you that know me, watch for the domain falkenlabs.com - you'll find me there, and AMZN will be powering it - by the end of the year google will power nothing, receive nothing, and I am absolutely done with them - they were great at one point, and I miss that greatness dearly. What is left isn't worth shit. AWS has proven they can do it, they get all the infrastructure - here's hoping they don't do the same sad transition
MMM
Add a comment...
Wait while more posts are being loaded