Puppets as storytelling tools

[elbows]

Ive long been interested in virtual puppetry, realtime manipulation and rendering of 3D character models controlled by midi.

Since I started thinking about it, 4 years seem to have gone by with very little progress on my part. Finally in the last few months things have started to slowly fall into place. A combination of motorized midi faders, Pilgrim, Touch Designer, several PCs, and Ableton live for midi sequencing and triggering appears to be yielding results.

Anyway I have hundreds more hours work before things are good enough to be useable, but I wondered if anybody is interested in this sort of stuff. Im interested in 3 different areas: creating characters that react to the music in an AV context (music either played live or prerecorded, all midi, no audio analysis), creating characters to tell a story - manipulated and remixed live, and playing back pre-recorded story/narrative. Also I suppose as a tool for creating lots of non-realtime video footage. eg a virtual newsroom where the characters etc are the same every day, but the story content changes. Or a low budget soap ;)

Ive pursued VJing software for the 3D stuff as opposed to traditional character animation/3d software because I want to try a technique where the face and specifically eyes and mouth of the characters are created by mapping video of the eyes/mouth to the right surfaces. This might be useful for massive shortcut and funny looking effect when wanting characters to speak.

Some kind of slightly cheesy slightly scary bush-bot is my test-run project with touch designer trial version. Its a very crude and simple model, and I havent got the inverse kinematics of the characters limb joints configured properly yet, but its starting to come to life.

[many2]

I believe Spark (who was working at Jim Henson's studios the last time I heard) could give you good insights about puppets and their integration with real-time graphics and video. I remember seeing a very good puppet manipulation tool at the Game Developer Conference like 3 or 4 years ago and it was from Henson's studio.

[elbows]

I bet they can do some amazing stuff. Id love to hear about that sort of thing. Id guess theres a rather large gap between this technology and my wallet lol, so this is why Im spending time on DIY type solutions. Full body motion capture suits have long featured in my plans too, but the problem is my plans feature no way of affording one lol.

[iKande]

i'm not quite sure what you hoping to end up with but here are a few thoughts i had reading your post.
i think the way i'd do something like this is to control it as a proper puppet, i.e. using strings. maybe a couple of mice could be cannabalised and strings wrapped around the wheels. so you control it as if there is a puppet connected but the computer takes the input of the strings and yanks up the arms/legs/etc and renders it. if you can i'd say the best method for getting alot of control and scene complexity would be an sgi box with one of these dials boxes http://www.xsnet.com/retail/XQ/catalog_name.SGI/category_name.O2+Series+Other/parent_category_name.O2+Series/product_id.SGI%2DALL%2DPRT%2DP%2DDIALS/QX/images/catalog/sgi/parts/dial%20box.gif. they have 8 dials that are already mapped into the X Input extension ready to use. if you pull one apart and used one dial per axis you could get two two-handed puppets going. O2 and others allow the use of real-time video as a texture so you (or the puppeteer or an actor) could have their face glued to the puppet. the box is standard serial so an sgi is not required (non-irix support unknown) and more than one can be used at once.

[many2]

If I was to create interactive 3d puppets I would go for something based on a 3d game engine. Think about it : a video game is exactly what you want to do. You want to control a 3d character in real-time, you want it to react to external stimuli (what's the difference between a gun shot and bass beat for a 3d character ?), and you also want high-level control (you don't want to rotate an ankle directly, you want to be able to blend between different character movements loops with an intuitive interface). Adding video-textures for the face, eyes, mouth, or anything else is just an artistic choice - no matter what the technology you choose it should be easy to implement.

If you play any fighting game you'll see real-time character animation is very very impressive now. Replace the fight moves with dance moves and you're ready for an interactive waltz ;)

[Exavior]

What you are referring to is called ?machinima.? It is an art form that has existed for quite some time now, and is currently being used as a both a live and pre-recorded performance tool.

A good resource for this is http://www.machinima.com/

Many 3D game engines are built with the expectation that their user-base will extend the life of the engine through modding. (Creating their own projects with the games engine) For this reason, most will release SDKs (software development kits) to help users create maps, models etc. Machinima animators develop custom mods (levels, textures, animations, scripts, etc) to enable them to animate the game characters as if they were puppets.

With some of the more advanced game engines you can setup virtual camera rigs, including mounting an invisible camera frame on each game character with various cameras (POV, over shoulder, chase, etc) that you can switch between.

I have a pretty complex setup I plan to build for use in live performance, but I am waiting for the halflife2 engine and SDK to be released.

I?m also researching possible interface methods, so any advice there would be great. A simple solution is to mod a keyboard. Then there is the possibility of using a keyboard encoder (like those used for MAME systems) and micro-switches/joysticks/etc.

MIDI control would be great, and worth looking into.

[many2]

Machinima is indeed close to what I was talking about. It's a shame most editing tools are so primitive or aren't giving as much access to the engine as the developpers themselves have. The game mod scene in general is a great ressource too. Elbow's project fits between Machinima and Game Mod because he wants to create a story with virtual actors (machinima) but he wants real-time control over them (Mod).

Kaydara's solution is great but it's very expensive. I don't know much about Touch Designer but I suppose that could be a good solution if it supports character animation.

As for a low-budget soap inspiration I really recommend to watch the Red Vs Blue series - it's very simple and a lot of fun. It's the closest thing to a machinima blockbuster ;)

[Exavior]

Yeah RvB is great. But as far as cinematic blockbuster stuff, a few of the movies found on machinima.com are really well done. Many you would never guess were done in real time 3D. But there is also a lot of post-production work done on most of them.

You can also find guys out there who do the whole thing real time in theaters. Basically 1 guy takes the role of camera man, and the others play the part of actors.

The end result is pretty cool, and the level design tools make building custom sets pretty easy.

[elbows]

A joyful thanks for all the quality info so far, I've not had a chance to check it all out yet but this all sounds very helpful and interesting. I got a trial of MotionBuilder to experiment with a while ago, but then got distracted by other stuff and it timed out, I will have another look.

I originally avoided game engines because at the time the idea of video mapped to textures was outside the gaming world due to slower machines etc, wheras now it is even a part of directX stuff so is a far more mainstream concept I guess. However Im not sure of any engines that support it in an easy way. My main problem really is that my coding ability is effectively nil when it comes to modern non-web programming languages, and I dont have a decent budget to enable me to pay for additional developments. Using high-end VJ software I dont have to worry about the VJ/realtime freindly aspects of the program, I have to wait for the 3D and animation and rendering quality stuff with VJ apps to improve. visualJockey got me started when 3ds object support was added, but then I wanted more than it could do. Pilgrim took me further, and is possibly enough, but now Touch designer has been tempting me again very much.

Im really looking forward to visiting that machinima site, must find time tonight :)