video textures & 3d

[apu]

I have seen some posts about 3d scenes and using video as a texture on 3d objects etc etc and am wondering how this is done and what s/w works best and can it be done in real time.

Can you manipulate the video or change it completly. I guess you can if you are using some form of keying.....

can you manipulate the 3d object in real time and will the video follow suit?


or does it all have to be pre rendered and just played back.

any hint/tips/stories welcomed as I am trying to get my head around 3d.

I work pretty much with plain old avi files etc etc but some of them contain 3d stuff i have created in swift3d and i am wondering how to spruce them up a bit if possible.

[burstingfist]

For 3D with live video textures, I use visualJockey. It is a very simple process to setup a basic 3D scene with video. First you either create a .3DS file with 3DSMax or download a .3DS file from the web. Then open up visualJockey and start a new composition. Drag the pdoom 3DSLoader onto the composition. Double click on the loader to configure it. Browse to locate your .3DS file. All distince textures will be listed, click on the texture you want to have the live video and select 'input screen' for the tecture source. A new node will appear, drag the 'Live Video ' effect onto the new node (Make sure you have either a WDM or VFW compliant capture device connected). The last thing you need to do is drag another resource effect onto the background node of the 3DSLoader, just use a paintbucket for simplicity. And voila! 3D textured with live video.

[MoRpH]

As far as I'm aware the main softwares with this capability @ the momment are VisualJokey and Moonster. I'm sure there are others floating around as its a function many VJ apps are putting in there first versions these days, although few of them are as good, easy to use and fast as the Moonster engine.

Oh and I'm sure its in all that Pilgrem/SVD1/etc constant beta/vapourware

[burstingfist]

I would really like to try out Moonster, but it will not work on Win2K unless you already own the Moonster CD. I might just buy it, the interface seems pretty user friendly and it supports a wide range of media types

[hamageddon]

that "vapourware" svd-1 works quite nice
here: u can map textures, clips and live video to 3D Objects and manipulate these objects in realtime.

pd/gem can map video to 3D objects too
maybe more flexible tru modular programming, but less intuitive and live usable so IMHO less fun.

and, hey, talking bout real vapourware
how's AJ's AVS and "what the" HeyVJ? (still sounds like a barbie bundle to me) ;)

[MoRpH]

Not sure about Hey VJ as thats james' baby. As for AJs AVS I think he is still well stuck on the final release name and the dongle contruction issues... pitty the software kicks major ass, although it doesn't have the 3d functions we are talking about on this thread.

Ooops just checked their site and its actually SVD2 that I have been waiting months for screen shots of...... basically if the UI look like 3ds max (like Pilgrim) I'll walk past it... I want a UI that works live and is the sort of layout I like to work with (eg, SVi, TZT, AJs AVS, etc) thats why I think customizable UI is the way to go. Besides you can't even get the old version from the website FFS

Oh and I'll admit my bias now I think MOST 3d is SHIT and video mapped onto 3d = just as shit.... just my opinion, it ain't my thing

[hamageddon]

that svi interface doesn't have an inituitive feel 4 me, frankly it's a mess :).
pilgrim looks like resolume compared to this....

but anyway, talking of biases, i've found full satisfaction with the gui of svd1, which looks like this btw:
http://www.audiovisualizers.com/toolshak/vjprgpix/channel3/svd1_1.jpg


but talking of the worst vj-gui, nothing ever beats the vjo-interface.....:D

[apu]

thanks guys for your input i will have to check out each s/w for its merits etc etc and try some stuff out.

also not to mention learn something about 3ds. i've seen it once and thought yuck. very technical ui. a college over here runs short part time courses on 3ds and lightwave. might check them out so i can get a head start.

[elbows]

3D is one of those things which I have found if you dont have the right kind of brain for it then there is no way for software to make it much easier for you if its proper "create models from scratch" type stuff. I mean Ive been interested in using 3D stuff but I remember that at school I was awful at 3d drawing and being able to draw perspective properly and stuff like that, and so far it has proved to be hopelessly hard for me to make anything that looks remotely good using proper 3d modelling tools.

So Ive been trying to use tools such as the demo of 3dmenow for creating 3d heads, poser for character animation, ulead cool3d for 3d text etc etc. Or download existing models from the internet. The problem withthis approach is that by the very nature of these tools, the content has a similar look and can often go stale and be overused by too many people. But if used cleverly witht he right combination of effects it certainly has its place.

Other problems that spring to mind with VJing with 3D:

Most of these apps support 3ds format, and this is now an old format which doesnt support a lot of advancements in 3d techniques. So for example visualJockey and others can handle the animation built into imported 3ds files, but can I find a package that allows good animations to be exported in this format? No :( Im such a 3d cretin that I probably wont even get the terminology right, but I believe 3ds basically cannot handle mesh deformations, so you are limited to animation consisting of different elements of the 3d model moving position, not changing shape. 3D Studio Max uses .MAX format as its main file format now, 3ds is a relic of older versions. So for example I havent found a way to take poser animations and export them nicely to 3ds, or any way to get decent facial animations intoa format vj packages can use well.

3D stuff is more prone to show just how low a resolution most of us currently have to work in, via pixely edges, so use of some nice filters to soften this stuff up, or clever use of high res for the 3d elements with lower res for video so that you dont reduce your framerate to unacceptable levels.

The other biggie that springs to mind is the often mentioned "cheese factor" which 3d can bring unless its done well. Im sure we've all seen some of the horrendously bad low budget childrens tv thats made with 3d computer graphics, and the sad thing is that even that pap took a whole lotta man hours to do.

I cant overemphasise how important quality textures are for helping to avoid the cheese factor, and ive been very impressed by the results obtainable by for example using live video or video clips as textures can be. I mean its quite clear just by playing some modern computer games that 3d can look so classy if done right, but again we can all imagine the development time that goes into such things.

One other thing is Ive been looking for some kind of virtual puppeteering software thats computer based, I imagined there must be some by now but I havent seen anythig that looked good. Which is crazy considering how great games can look, and theres really not much difference needed in terms of technology, engine etc, in some ways pupettering stuff would be simpler than todays 3d shooing games. Maybe Ill be lucky and someone whose about to embark on writing yet another piece of VJ software that does much the same as all the others, but with a few different features and a different UI style, will read this and decide to make a pupeterring tool instead. If I cant export cool animations to a format that current VJ apps can use effectively in terms of triggering parts of the animation using midi controllers, then it strikes me that theres a gap in the market. Id pay ?500 for such a thing anyways. Ahh go on ;)

[eXhale]

Originally posted by elbows
Most of these apps support 3ds format, and this is now an old format which doesnt support a lot of advancements in 3d techniques. So for example visualJockey and others can handle the animation built into imported 3ds files, but can I find a package that allows good animations to be exported in this format? No :(
The 3DS format is usually handled by Windows softwares because Direct3D has built-in 3DS support. I don't know if Microsoft added support for other formats in later versions, but apparently not since VisualJockey only supports that format.

[burstingfist]

Originally posted by eXhale

The 3DS format is usually handled by Windows softwares because Direct3D has built-in 3DS support. I don't know if Microsoft added support for other formats in later versions, but apparently not since VisualJockey only supports that format.

Actually visualJockey uses OpenGL, not Direct3D. The reason many softwares do not support the more advanced formats is for simplicity. 3DS files only support translation, rotation, camera animation, and lighting animations. It is up to the host software to handle more complicated operations like mesh morphing and mesh distortions. I just purchased Moonster because it supports a wide variety of 3D formats so I am hoping it will support some of the cooler 3D effects.

[elbows]

Oops sorry for getting carried away there, having read your original post I think I should have like tried to answer you specific questions rather than go off on one about my 3d dreams :D

If you pre-render your 3d footage to a video file, then none of my rants about lack of good animation etc are applicable. And if you are clever with the way you break up the video file and assign it to controlelrs etc, you can make it look like you are controlling the animation realtime, when really you are just triggering different video clips that are different parts of the animation.

The reason I wanted more than that is for example take a human model and say you have it doing 4 different body animations. I can render 4 different avi's of these, plus an avi of the character doing nothing. But then I have to be clever live to get the timing right so that the animation follows on smoothly, and I cant do stuff like go from one pose to another one witht he intermediate steps in the animation occuring properly (well I probably could with hundreds of clips of all possible combos but thats too much for me)

Also its nice to not loose realtime control of any video etc you may be using as a texture, which obviosuly you will if youve prerendered the whole thing.

In terms of software out there, yes Pilgrim R1 (still in beta at the moment) looks promising though the design part of the UI is very much like a 3d modelling app, but then again I dont know how you can avoid that if you need as much power to create scenes as a 3d modelling package has. visualJockey has a simpler way of getting started using 3d objects for sure, and certainly doesnt lack power in terms of what you can do with textures etc. But then quite a lot of people dont like the VJ side of the UI for visualJockey.

There is another option, called Touch Designer from Derivative Inc, and thats very powerful, but again theres a fairly steep learning curve and also a steep price for it, but a trial is available.

What software (if any) are you currently using to VJ?

http://www.derivativeinc.com/home/home.asp (Touch Designer)

http://www.loosegoose.nl/PilgrimMain.htm (Pilgrim R1 but not available yet unless you can get on the beta program)

http://www.visualjockey.com (VisualJockey)

http://www.xgrind.com (components needed for 3d still in development i think but 3d text is there for a start)

http://www.xynth.net (definately uses 3d but I dont think you get to choose much or use your own stuff yet, dont know what their future plans are in this regard)

[apu]

the more i play with some basic 3d stuff via vjo the more i dont actually like it (not yet anyway) and i dont want to start a forum war over what is best etc etc.

at the moment being so new at it i kind of prefer to use my own footage and fuck it up as best i can rather than relying on 3d although i am sure a balance can be found after enough playing around. which is all time behind the wheel ya know.

i agree with elbows that if you aint got the h/w to pull of some hires stuff then you are going to struggle to look good.

i cant help but love the feel of electronika which has such a live feel to it. i have spent about 2 hour now with vjo and would not like to use it live(on its own like i can with elektronica) but i think it would crank if you ran it on another pc feeding into your h/w mixer with another app like electonica. that way you could get the best of both worlds.

all $$$ though

[eniac]

New version of XYNth will be out soon. Video texturing will be enabled for those with nVidia based cards as well as an all new audio engine and spectrum analysis. XYNth has a unique visual style and is very reponsive, stable and fast.

There is a video clip on the site with a short clip where you can see a bit of the video texturing in action. There are 4 different video processing modes for b&w, color, blurry, & dichromic.

Lots of new stuff, and exciting developments will be announced soon!

http://xynth.net

[pdoom]

Originally posted by eXhale

The 3DS format is usually handled by Windows softwares because Direct3D has built-in 3DS support

are you sure? I've never heard about something like that. AFAIK direct3d has it's own format (.x).

btw: 3ds DOES have morphing information, which is actually supported by visualJockey, but I don't know of any recent 3d tool that can export to morphing 3ds files. The only app I know to support it is the original 3D-Studio (for DOS!).

[eXhale]

Originally posted by pdoom
are you sure? I've never heard about something like that. AFAIK direct3d has it's own format (.x).Hmmm now you ask me, I'm not sure anymore. :confused: I cleary remember having somehow managed to import a plane model made on 3D Studio Max on a Direct3D (RM) app I coded, but I don't remember if I had converted it before to the .x format. I remember it wasn't a difficult process anyway.

[many2]

Believe me (and this is a question I've had to answer on many game productions) there is only one way to get the most out of 3d : you have to build an exporter.

All 3d engines have different features, different optimizations, different data paths, and you have to be able to build custom content for your engine. Since you don't want to code a whole new 3d software package like 3ds Max the only solution is to code an exporter plugin for Max. Once this is done, you can even add custom data to your 3d scenes with the 3ds Max (or other app) interface or enance this interface with custom plugins. You can also add new info in the form of flags attached to any model.

Many-2

[pdoom]

many2: yes, an exporter would be a very nice solution of course. But a game differs from a VJ app in that VJs probably want to be able to do their own 3d scenes, so what about the people that don't use max? (e.g. Maya users will be pissed).

I mean, a game company can easily stick to 3dsMax, but there's more than that out there.

[burstingfist]

Originally posted by pdoom
many2: yes, an exporter would be a very nice solution of course. But a game differs from a VJ app in that VJs probably want to be able to do their own 3d scenes, so what about the people that don't use max? (e.g. Maya users will be pissed).

I mean, a game company can easily stick to 3dsMax, but there's more than that out there.

So what exactly are you saying pdoom? I am all ears. Some sort of universal importer? I would love to see an importer that supported mesh morphs and mesh manipulations of any type. Maya is an ill program, it is just more difficult to get a "usable" copy than 3DSMax. I totally respect pdoom for whatever he has to say regarding 3D, but this is this most verbose I have seen him in a while (mind you I am quite wasted drunk and just got let off by the police)...

[pdoom]

Well, there was no big philosophy behind my words :)

I just want to say that I don't know, actually. How do we get 3D files in a VJ app? Some approaches:

1. support one specific format (3ds): not easy, because your 3d-engine certainly doesn't support everything in the file (you want to render it in realtime, so you can't support bump-mapping, lighting, shadows,...). And certainly your VJ app has more functionality than a 'simple' 3D Editor (realtime parameters, stuff that happens to textures and so on).
Another problem is that not all 3D editors support everything of that format (e.g. some editors can't export animations/morphs etc to 3ds).

2. support man different 3D formats: lot of work... and all the problems in 1) again
3. define your own 3d file format and write a plugin for a specific app (e.g. 3dsmax): would be a nice solution, flexible and user friendly. but this would mean that users that haven't got that specific app (e.g. 3dsmax) can't produce their own 3d files.
4. write your own 3d editor. Probably the best solution, but almost impossible for a bunch of low budget VJ-soft coder :)


visualJockey does approach 1) and a bit of 4)

@burstingfist: sober yet? :)

[burstingfist]

Originally posted by pdoom
Well, there was no big philosophy behind my words :)
...
2. support man different 3D formats: lot of work... and all the problems in 1) again
...

@burstingfist: sober yet? :)

OK, I just purchased Moonster because I am sick and tired of pussyfootin' around. Supposedly it supports 3DS, COB, DXF, LWO, MD2, OBJ, RAW file formats, though I am not sure if these will help at all. I really want to be able to do realtime 3D FX that are not your normal fare. I figured that Moonster was reasonable ($60 USD) so I said "hey, what the fuck?" and purchased it. Unlike MoRpH, who is for some reason vehemently opposed to 3D stuff, I think that 3D objects can be used wisely so as to enhace the overall appearance of your performance. Yeah, I have no real information to add to this thread, but hey, what the fuck, I'm wasted ;) ...

Peace

[MoRpH]

I own Moonster so I'm not ANTI-3d, I'm ANTI cliche 3d... and admittedly 99% of it is LIKE THAT.

[elbows]

Then perhaps the solution would be to make plugins for the most popular 3D apps rather than just 3DSMAX.

Also, as Blender is now changing into free opensource software, maybe that would be something to look at in terms of creating a VJ friendly editor?

[LEVLHED]

burstingfist...you're running the updated W2K/XP version of Moonster? Please let us know how the stability is...
I'd used moonster back when it was new, but there were definately stability issues....I didn't have W2K at the time so haven't tried that v.

[many2]

In fact coding an exporter isn't that hard, and it's even easier if you coded the 3d engine yourself since you'll know what to export and how. There aren't that many popular 3d packages out there : beside Maya and 3ds Max I know no other software that can be described as a standard (and I know about all of them). Once you've done exporter for one (or both) of these apps your work is done : if someone is using something else (like XSI, Blender, or Lightwave) he will probably have a way to export his files to the app you've chosen. Being able to use the full power of our 3d chips is very important to be able to get "creative" with it, and to achieve that we really need to be able to add custom content : this is why we can't stick with an old format like the 3ds format. With a lot of VJs out there buying new systems with GeForce 4 GPU we have to use that hidden power : vertex and pixel shaders seems to be made for us VJs much more than for game developpers, and beside Touch, I know no VJ software that is actually using that power.

Many-2