Topic awaiting preservation: ShadowPlay video - better late than never (Page 1 of 1)

Some of you may remember me talking about ShadowPlay a while back (so long ago I was probably still posting as dracusis back then). I searched for the thread I made back then but I can't remember what it was called so this'll have to do. Anyways, I finally got around to capturing some of the video footage I have of this.

The video is kinda crappy though as I still don't have access to the second mini DV tape and as most of the university computers are in transit (my faculty is moving to a new campus), I had to capture this on a rather underpowered Mac and edit it with iMovie. As a result, the sound is well, shite -- but that's beside the point.
www.thecoffeeboy.com/ShadowPlay.mov ( just shy of 30 meg -- my apologies to those on dialup )

Unfortionatly the footage of that is from the actual input camera used to track and monitor the shadows by the computer running the program so it's all shot from the same angle. I'll have more when I get my hands on the second tape and a decent computer to capture and edit the footage on.

I slapped this together the other day as I have a meeting with some folk from the city council who want to discuss some stuff about this with me. I'm not entirely sure what they?d want to do with it, nor have I had any conclusive thoughts as to what I'd like to do with this project since the prototype showing at the exhibition. Still, I'm a little surprised and curious about the meeting.

Wow....just.....uh....wow!

I'm speechless. That is one nice piece of video tracking and interpretation you coded there Cameron. Good work bud!

Nice thingy. I've seen similar things and even one done with Flash MX ( the input was not taken live but with the webcam feature I suppose it's feasible )

Could you give some infos about the computer runing this, the resolution at which you extract the shadows, and at which framerate, the language used ... Thanks in advance.

oof, awesome. Cut that sucker in hald or 1/3rd and it'll sell better I like the chick/martini glass bit, don't cut her!

That's really neat man. Congrats.

poi, that's running on a P4 1.6ghz with 256 MB of memory. The input resolution for the video was 160x120 although I've run higher resolutions on faster computers. It was done in director (which would make the language 'lingo') using the "Track then colours" xtra for the video input, but the xtra's built in tracking didn't handle that too well so I simply ran a threshold filter over the live input image and did the tracking based on simple black/white pixel picking in an off screen buffer.

Most of the image drawing was done at 320x240, but there were some post effects slapped on after I scaled all that up to 640x480 (the final display resolution) like some simple feedback blurs. It ran at around 30 frames per second on that PC, but I've had it much faster on my home PC running off a simple USB web camera. I'd be interested to know what you've seen or done with similar things in flash. I researched flash a little before building this but as far as I know there's no way you can actually sample video/image data, unless you vectorize (sp ?) it beforehand.

I wish I'd had the foresight and the knowledge to do this in Java. 50% of the time spent on this was figuring out how I could get director to run fast enough to handle it. As with most scripted animation, anything much below 20fps looks bad. Given the amount of dogey hacks I had to do to get this running at a decent speed I'd defiantly consider this a prototype at best. I updated the video input buffer around once ever 3 to 4 display frames. This didn't seem any different to a 1:1 (much slower) setup seeing as live video capture will always incur a delay between 0.2 to 1.5 seconds thus the animation was all interpolated to compensate for the out of sync input.

The main speed issues I encountered with director were mostly from the video input and not from the sampling and collision detection (although I knew directors limitations in this area from previous projects so I simply avoided anything too taxing). The simple mouse based tests could run at 40+fps in 1024x768. As soon as I plugged in the live video capture in it felt liked I'd just knee-capped the computer. Getting live video data into an image buffer in director is a very slow process. I've run a few simple tests in java which suggest it'd run around 2 to 10 times faster in that language.

Yet doing this in directory did have it's benefits. I got to see how people interacted, or more importantly how they thought they should be able to interact with and it was easy to make small updates each night to try and accommodate peoples expectations. I probably learnt more in the four days it was exhibited than the whole 3 months (not full time) I spent making it.

Jason - yeah I realised the video dragged on a bit. But I figured it'd be better to have more to show than not enough. Maybe... *shrug*... I'll be redoing the video when everything is setup at the new campus and I have the tape form the second camera, I?ll be sure to be more heavy handed with my virtual scissors next time.

I'm impressed.

quite a bit actually.

When you do get around to editing it... I would love to see some of how it was made in a short video, and also a short demo of the coolest things it can do. I would try to keep the videos to about 2 minutes max for the web, and have more than one.

cant wait to see more though.


[antique sigs are us]

I'd be interested to know how this works/was made as well.

"Nothin' like a pro-stabbin' from a pro." -Weadah

Hmm, interesting.

You know, we've had something similar to this at our local mall for quite a while now. There's a projector on the ceiling that projects an image of a pond on the floor, and when you step on the image, ripples form and expand from around your footprints. Not sure it that's based on the same technology as this, but it's nifty nonetheless.

quote:

---

Wow....just.....uh....wow!

---

Ditto that man, that looks really cool, and fun to boot. Nice work Cameron.

Out of curiosity, is this simply of entertainment value, or are there other applications you have in mind?

Very cool Cameron.

--tiki

Well, I just got back from my meeting with the Brisbane City Council and they liked it quite a lot. It appears that the showing at the SOOB festival had a bigger impact than I first thought. I might be building a permanent version of this to be installed in a new library/museum being built in the centre of the city by the end of the year!

I'm still trying to come to terms with this prospect, given that this whole thing was a bit of a pipe dream that I never really expected to pull off as a prototype in the first place. It might not happen, but the fact that it's being seriously considered is an honour in itself considering that I'm still an undergraduate student.

-------

Izz: several shorter videos would definatly be the way to go.

Ramasax: Entertainment and or art is probably the main focus with this, but parhaps education as well. Given it's ability to react to whole body movement makes it suitable for many different things. Yet the cost (data projectors and digital cameras aren't cheap, I flogged.. err borrowed some stuff from university for the exhibition) lands it firmly in the area of experimental and or artistic installations for public spaces. Although, technically the eye-toy for the PS2 does similar things but it's limited to displaying an image of you on screen so you can see what you're interacting with.

ozphactor: Sounds similar indeed. Unless they've installed pressure pads (unlikely) I'd be betting it uses some kind of video tracking. It might even be infared based. Maybe there's some special IR reflective paint on the floor, which would likely give a nice indication as to the foot position and size if the area was flooded with IR light.

---------

How does it work? Well, the overall technology model is mind numbingly simple. But the inner workings and the design process it a bit of a mind twist.

In short, there's a data projector, which projects the image of the program onto the screen. Think of it like a full screen flash animation if you will (but it?s not flash). Then, you step in front of the projector and cast your shadow onto the screen. There's a camera sitting off to the side (or above, or below etc...) that can see the screen and the user?s shadows cast upon it without the user?s actual body being in-between the camera and the screen. This image is fed back into the computer running the program. It takes the live camera feed and skews it back to the proportions of the screen (you'd do this in a buffer, i.e. not visible on screen), which is a bit tricky, but I wrote a calibration program to help with this. Then you run a black & white threshold filter over the image and hey presto, you now have a black and white image map of where the user's shadow is on the screen.

After all of that, you then have to program the interaction. Which I must say was a bit of a mind twist from the usual x/y mouse coordinates I usually design interactive elements for. All of a sudden you're dealing with entire shapes as an input instead of a single x/y point and a few key presses. In some instances this makes it harder to program interactive elements, but what you can do instantly feels more natural (to the user) than anything you'd use a mouse for. There are lots of things I've yet to experiment with and some which I've already tried have failed horribly. Like the silly idea I had of trying to merge this with music and a win amp like visual, which just created visuals too complex to interact with.

There's also a lot of performance issues when it comes to reading the black & white bitmaps, as I was unable to get director running fast enough to trace the image into vectors (vectors of any decent resolution anyhow), so most of the collision detection was done by sampling a limited amount of points in the image each frame, which limits what can be done with it. So I'm hoping to build this again in Java at some point, which should give me the power to try out a lot of new things and maybe add some real time 3D visuals into the mix.


And I'm going to shut up now before I crap on for another 3000+ words.

Cameron: Thank you for the infos. That basically what I thougt, except the use of Director. In fact it's pretty easy for the coders doing computer graphics.

I'll have to dive deep in the mails I sent me with some URLs to find the similar thing done in Flash

Anyway, it's certainly worth having a look at Processing which is a language based on Java and is ideal to make interactive toys. Thanks to the QuickTime for Java it can handle live video capture as you can see in this example.

[edit]

it could be fun to make some flat objects using a 2D spring systems to interact with the users. The visitor could hold the objects and push them in the air or play "virtual" voley ball with the bloby wobbly object.

[/edit]



[This message has been edited by poi (edited 02-16-2004).]

[This message has been edited by poi (edited 02-16-2004).]

Poi, I know about proce55ing, I started a thread about it in the multimedia forum a month or so back :P

quote:

---

In fact it's pretty easy for the coders doing computer graphics.

---

No doubt, but most of the issues I encounter were with the speed everything ran at. I actually had a working mouse based demo within the first week. The camera input slowed things down to a zippy 4 frames a second, so I had to completely re-work larges sections to cope with that overhead. That and I wouldn't consider myself a coder of any sort. Being self taught in this area I usually hit the wall then figure out how to get over it, so things probably seem more difficult from my perspective.

I'm a little daft when it comes to math not having a proper IT background (my university doesn't offer much in the way of graphics programming, it's either design/authoring or IT focused on applications or information systems) so if you know of any decent physics / graphics programming texts or resources I'd be grateful for the heads up. =)

I'd love to have a look at that flash stuff if you can find the links for em.

Actually, it'd be interesting to see if I could use java just to get the input and process it then somehow send that to another application like flash. I wonder if flash supports XML-RPC... probably not.

I had a little pong like game happening in one of my demos, but it didn't fit into the theme of that installation so I left it out. Although if a budget, suitable wall space and a little luck permits for this library project then I'll likely be re-designing it to fit in better with the overall history of Brisbane, which would be fun.

I've also been toying around with negative shape recognition, like someone making a circle with your arms or just the thumb and index finger as well as other general shape recognition which can allow more precise control and interaction. Although, I've had things on hold for a while pending the port to java, hoping that will allow me to step up the video resolution to 320x240.

The other option is to use a reverse of this concept to track light spots with a camera, possibly by giving the user a flash light to point at the screen (which would display mostly dark visuals obviously). This could make for an interesting interface for a game if you had different light levels/colours to control a first person avatar and shoot with (would be perfect for a horror/thriller game, especially if done in a faux cave... drool...). It'd also give an interesting perspective to war/strategy games that employ table-top gaming metaphors. A virtual map on a tabletop you can manipulate with various light tools or perhaps some rigged/coloured gloves. Mounting the projector on the roof and pointing it down would also lend more weight to the top-down perspective employed is most of those kind of games.

And I could rant on about this all day, but you're all probably getting sick of me by now so I'll shut up.

The poster has demanded we remove all his contributions, less he takes legal action.
We have done so.
Now Tyberius Prime expects him to start complaining that we removed his 'free speech' since this message will replace all of his posts, past and future.
Don't follow his example - seek real life help first.