Pixel shader session at Siggraph

[petewarden]

Wearing my Apple hat, I'll be giving a talk on implementing pixel-shaders for real-time video effects, with examples from Motion, together with Frank Jargstorff of NVidia:

Image Processing with the GPU
Monday, 10:30 am ? 12:15;
Tuesday, 3:45 ? 5:30 pm;
Wednesday, 1:45 ? 3:15 pm
Room 401

You'll need a full pass to get in, but I should get the slides online after siggraph if you can't make it.

Pete

[many2]

People working with me on a new project are going to Siggraph so I could send them listen to your talk. Is it made exclusively for programmers of will there be some eye-candy for artists and designers ? There is no programmer on the team going to Siggraph so I wonder if you think it would be a good lecture for them.

[dongbamage]

Not gonna get there without a 5000 or so mile swim, so sildes and stuff would be greatly appreciated ;)

[klumsy]

yeah, i'd be interested in that sort of stuff (those slides), since very soon chaning mostly all software effects into pixelshaders..

so pete,
what do you do at apple?

[elbows]

Hello, hope it all goes great for ya at siggraph Pete.

To say I am dissapointed at the slow progress in terms of vj software moving to support shader technology, would be an understatement. A couple of apps have appeared that have a few shaders in there, but not reaching shader potential yet by a long way.

Im hoping that as VJ software does emerge with this functionality, that it will be possible for shaders to be shared like with freeframe, its still unclear to me which shader standard is the best for this mission?

Anyway I cant program any of this fancy stuff so I cant really help, but I am going to spend the next hour or so updating myself on whats new in the world of shaders, and see if I can find any online resources that might inspire more people who can actually program to do something VJshadertastic.

[LEVLHED]

I agree elbows! this power is still largely untapped....how long ago did I come here ranting about this stuff? seems like two years now!

[Rovastar]

Let's be honest her who has a nvidia 6800ultra/ATi X800. From what I understand pixel/fragment shaders are only *reallY* useful for powerful cards.

Once the devolpers get decent new generation cards like this then we will see stuff happening. But these cards haven't been out for long. Then the devolpment time comes in.

And besides a big chunk of the VJ market is laptops and it will be what 2 years before they have those sort of cards in laptops.

Pixel shaders are powerful and will truely be lovely when done but lets give it time.

[Exavior]

Would love to hear how the talks go Pete. I may be at siggraph a bit this year (if I can find the time) but I've not really been too active there since 'nothing real' was purchased by apple and my free passes stopped being available.

Cards that support pixel shaders have been out for well over a year. Two of my systems have cards that support it, one ATI, on nVidia. I have been enjoying Quake3 and its delicious baked normals for a couple days now. If only we could take advantage of that type of power.

And yes, the newer shader effects are for newer cards, that?s the reality of real time effects. New effects need new cards. You can?t expect per-pixel shading to run on a old voodoo3 card. ;)

Thankfully the prices in these cards should drop a great deal once more games start to utilize these features. Soon the market will flood with them. And eventually VJ software will catch up.

[MoRpH]

Originally posted by Rovastar
And besides a big chunk of the VJ market is laptops and it will be what 2 years before they have those sort of cards in laptops.

Here here, fix the problems with the current range of VJ softwares running on laptops b4 trying to add something NEW, don't let VJ apps become the new windows (trying to add new stuff b4 you fix the bugs in the old)

[klumsy]

And yes, the newer shader effects are for newer cards, that?s the reality of real time effects. New effects need new cards. You can?t expect per-pixel shading to run on a old voodoo3 card.

yep but there are different generations of pixel shaders,
with the earlier generations, while useful for certian effects typical to a 3d object/scene, aren't useful as a way to build generic effects we like to use in the VJ scene.

[InsideUsAll]

wish i was there to attend this. bit too busy at the moment to make the jaunt across the pond. if you do get chance to upload those slides somewhere dude i'd be very interested!

best regards

Dave :D

[elbows]

Arghh noo lol, I completely disagree. If there is no bleeding edge, innovation slows, surely?

In the context of where specific developers of specific software should be focussing their attention, I agree. Im not suggesting for a minute that any bugs etc in existing VJ apps be ignored in favour of shader-related developments.

Anyway each to their own, but if VJ related shader developments began tomorrow then I dont see how that would be too soon, they were useful to me in KazetachiAD a year ago, they're only going to get more useful. But yes I am prepared for several more years wait before these things take off bigtime (hopefully).

Didnt find too many exciting things about this on the net yesterday. Found an intreguing "build your own shaders using a visual tree of components" software, but its $2000 lol.

http://www.rtzen.com/product/detail,1-1-4.html

Originally posted by MoRpH
Here here, fix the problems with the current range of VJ softwares running on laptops b4 trying to add something NEW, don't let VJ apps become the new windows (trying to add new stuff b4 you fix the bugs in the old)

[klumsy]

if you want good vj system with pixelshaders being a key component, buy a hippo.

[sleepytom]

why? - the hippo is software processing only - it has no 3d rendering afaik.

[klumsy]

ok, my mistake, maybe i'm getting in confused with xynth or something.

[nkm]

It is not that strange that pixel shaders haven't really kicked off yet in the vj scene.

Only since the advent of consumer graphics cards supporting the ps 2.0 specification has pixel shader programming become really interesting. Earlier graphics cards are restricted to a maximum pixel shader instruction count of 12. There is not much you can do with 12 instructions i can tell you :)

ps2 compatible graphics cards are still quite high- end. How many vj's actually have such a graphics card?

nkm

[elbows]

Well I know what you mean, but I worry about a catch-22 situation. Many times on these forums people are asking for graphics card advice, and the advice is heavily based on the fact that most VJ apps dont use 3D much or at all. Unless VJ software is developed or they use the computer for games and other 3D purposes, why ever upgrade to a powerful card?

Am I right in thinking ps2.0 came in with the Radeon 9500/9700 and GeForceFX series cards? If so, then this stuff is moving into the mid and even lower ranges of 3D cards now. Im not saying its time for someone to launch a pixelshader base d VJ software right now, but development takes time, so I would hope a few developers are at least looking at this stuff right now.

[Rovastar]

Graphics cards companies don't help in this either. Sure PS2.0 and PS3.0 *can* run on FX series but a FX5200 is not much quicker than a GF2MX/GF4MX. A FX5600 is probably the same a low end GF4Ti series. FX5700 higher end GF4Ti series. Confusion is rife.

Yeah I *can* run Doom3 on a GF Ti4200 but only at 640x480 without significant slowdown and with many of teh options turned off. And it slows down loads if things get too hectic.

Like I said it is only rerally the new generation (nvidia 6800ultra/ATi X800) that have the horsepower to give give new PS stuff a go. Even they can slow to 20-30fps in stuff like Doom3. at a decent 1280x1024 res with full AA.

But look at comuter games how many released actually use a decent amonut of Pixel Shaders in them? Not many, not many at all. Maybe a handful. And these are companies that have millions of pounds to throw at them.

Computer games for me show what can be done real-time and they are slowly geting into it. We will be a bit behind.

There is a lot of hype surrounding PS bit no-one is actually using them much to great effect. No-one is using them to any great detail until recent months and that is to due agian to technology not keeping up. Just now people are starting to use them.

Besides OpenGL was slow to get there pixel/fragment shaders off the ground. DX leads teh way and then PS3.0 stuff came out as well. :/

I see PS more useful for realtime 3d worlds than your traditional VJ used (clip mixing and effects). In fact I see my own stuff more useful than most 3d software esp at this stage.

[Exavior]

Yeah, there are not that many computer games right now that support the full gamut of pixel shading. But that?s the way it always starts in the gaming world. A couple big games will be released that boast a ?new? technology, which encourages the more ?hard core? gamers to go out and get the latest cards. That gets those cards out on the market, opening it up for new games to be developed by smaller game development houses.

New games (like doom3, or the mythical halflife2) are beyond ?bleeding edge.? Often they release features and resolutions that can not even be achieved with current consumer graphics cards. Their existence is what pushes the development of new graphics hardware. So what you see in gaming today is what you will see in other real time graphics applications a few years down the road. With some VJs out there being game developers, I?m sure that the physical engines of these games will be modded for real time visual use long before VJ specific applications come out that take advantage of this technology.

The potential use for these engines in real time 3D visuals is great. All the great new effects like reflection/refraction/lighting, handling of more detailed models, virtual camera/display systems, and other new features could really provide a lot of power if used properly.

I just can?t wait to see what we are able to create with these new tools. They have the potential; we have the talent... let?s make something great.

[trotskythecat]

The new version of Pixelshox, now called 'Quartz Composer' uses pixel shaders - to an extent that half of the effects won't run on my Radeon 8500

[klumsy]

rovastar,
so you do know something other than how to trash people :) good info though.. however even low end new card can process pixelshaders fast enough for a few levels of good effects at say 640x480 (which most VJ's would be more than happy with)

everybody,
other than useed in 3d scenes, pixel shaders can and will be used in very different techniques than used in games, and will be able to produce pretty much any 2d software effect, but faster due to GPU architecture..

[Rovastar]

Originally posted by klumsy
rovastar,
so you do know something other than how to trash people :)

? Who was I thrashing now? I could start on you if you like. ;)

[klumsy]

? Who was I thrashing now? I could start on you if you like.


NOBODY that was my point :) :)

though you can start with me if you like, since a couple of my projects might rile you up some..
(1) interfacing r4 with visual jockey
(2) hooking all/any applications calls to directX/openGL so i can redirect output of many programs such as milkdrop to rather not display on the screen but be used by freeframe/VisualJockey.

[Rovastar]

I know you would be too easy. :)

[elbows]

For sure the issues of which hardware has what capabilities is a pain, and obviously its not nice to think of all the people who wont be able to run certain apps on their current hardware. eg I feel the pain of people who like the look of Motion but balk at the spec requirements.

But at the end of the day my view remains that its better that some people have access to use this technology through great software, than nobody have it. Im not advocating all existing VJ apps to suddenly become shadertastic, Im simply eager to see it grow where appropriate.

Being unable to program in any anguage useful for realtime VJing, I am in no position to criticise in any way any developer or the scene in general. But purely out of interest in how things work, I do find myself wondering how the specific programming skills of the developers who work on VJ apps affects things.

So this got me wondering, how much the lack of shader stuff is down to user-issues as opposed to programmers areas of talent. If 80% of developers were well versed and interested in shader stuff, would it be in a lot more apps already, regardless of all the other issues discussed so far? Or have some of those issues themselves caused programmers to keep away from this tech for now?

[klumsy]

So this got me wondering, how much the lack of shader stuff is down to user-issues as opposed to programmers areas of talent. If 80% of developers were well versed and interested in shader stuff, would it be in a lot more apps already, regardless of all the other issues discussed so far? Or have some of those issues themselves caused programmers to keep away from this tech for now?


up until the more recent generation of shader cards, shaders didn't interest me much, other than to create some basic effects for 3d scenes.. modern shaders do though.. however shaders are just one tool.. i've yet to see a VJ software that blends a nice rendering pipeline combining software and 3d hardware effects together in a fast/efficent, and most important flexible and powerful way yet.. irregardless of the shader issue. most 3d accelearted vj apps are nice eye candy (because of the hardware), let lack the flexibility and power of some of the cpu-bound software ones..
so really what we need is a good flexible architecture, that puts 'shaders' as part of the picture..

with Xpak for visual jockey this is in the pipeline.. though i'm not sure if i want to make the directX things in Xpak to be part of the normal chain of things, or rather just add this functionality to the new mixer app, allowing different chains for compositions to be as textures in a 3d scene with flexible control, different effects and shaders etc.

[Toolmaker]

Originally posted by petewarden
Wearing my Apple hat ...Hi Pete,

I saw the Tuesday 3:45 show. Where was your hat? :(

Jargstorff?s stuff was dragging so I walked out and came back. (Do people really learn DSP fundamentals standing in a sweaty conference room?)

Enjoyed your part a lot. Does Jargstorff get pissed when you dis Nvidia?s NAN implementaton?

BTW: Over here in the Microsoft world Bill Gates wants everyone to program shaders using HLSL. Bill says that if I program in assembler that I am not part of his club. :mad: Bill intends on wrapping and abstracting shaders with his own language much like the way he is wrapping executables with ?managed code.?
Excuse my violent adult content, but Bill (the figurative meme) sucks and needs shot. :trashcomp

BTW #1: Thought for sure we?d talk at James? ?VJ: The Art of Live Video Performance? thing. Where were you?

BTW #2: Was this your first time pitching at a trade show?

It?s tough work isn?t it?

Don?t underestimate those sales weenies. They work hard.

- David Miller
(the one in Venice, not dm@ms)

[many2]

Originally posted by klumsy

with Xpak for visual jockey this is in the pipeline.. though i'm not sure if i want to make the directX things in Xpak to be part of the normal chain of things, or rather just add this functionality to the new mixer app, allowing different chains for compositions to be as textures in a 3d scene with flexible control, different effects and shaders etc.

Adding the functionality to the vJo mixer app would make a lot of sense. The data path wouldn't have to come back from the GPU if it was built this way.

[klumsy]

Adding the functionality to the vJo mixer app would make a lot of sense. The data path wouldn't have to come back from the GPU if it was built this way.

if it wasn't in the mixer, but rather as components, it would work like pdooms gl 3d stuff, that if pdoom stuff are in a chain then it doesn't bring it back to memory, UNTIL it is needed in a software effect..
however with XPAK i was planning on overriding the default vjo renderiung (rather rendering to DIB normally and putting on d3d texture), and if the bottommost node was XPAK then it wouldn't have to bring anythihng back to memory (give also that it only uses XPAK transitions, and the overlays don't force it back to memory as well.. which means a bit of tedious coding.. so probably the mixerapp way with be at least the first incarnation.

[DrEskaton]

Apple's Motion is going to be the tractor app that brings pixel shaders into the mainstream.

A ton of graphic designers are now buying ATI9800XT or Nvidia 6800 Ultra cards to run motion. It brings levels of interactivity you can't even get on a $350,000 discreet flame to a $4000 box (G5 + motion).

But what Apple has also done is put a standard for reusable pixel shaders (CoreVideo, CoreImage) into the operating system AND a graphical authoring app to create compositions that use them (Quartz Composer ex Pixelshox).

When Tiger comes out it will be possible to build a custom VJ app that uses pixel shader effects with a native Apple Cocoa interface without barely writing a single line of code. QuartzComposer + Interface builder will do most of it for you.

Think MAX/MSP but with everything native and using the full power of your GPU to accelerate all effects.

Look for a ton of very fast VJ apps to come out next year on mac os x once Tiger comes out. By that time every shipping mac will have a pixelshader capable 3d card. As it is only the emac and ibook don't.

I don't know enough about the PC tech to know if Microsoft has anything exactly quivalent to CoreVideo, I know DX9 supports PS but I don't think they are as easily accessable or as easy to share between Apps as Apple has done with CoreVideo/

As for being too expensive a Nvidia FX5200 has pixelshader support and is $50. On the laptop side a base Dell for $1000US is now shipping with a FX5200GO which again does pixel shaders.

The range and quality of the effects that pixelshaders enable is going to make such a huge jump in quality output from VJ apps that you'd be insane to buy a machine now that doesn't have a PS capable card.

Next year will be when all the apps come out to support this, I'm sure there are quite a few developers hacking away right now.

[MoRpH]

OK well it looks my next laptop will DEFINATELY be a mac now ;)

[DrEskaton]

Originally posted by MoRpH
OK well it looks my next laptop will DEFINATELY be a mac now ;)

ok sorry I wasn't meaning to sound like a rapid mac fanboy. What I'm assuming is that once people see the speed of motion on a mac that very quickly an equivalent program will come out on pc and that will drive adoptation of PS cards in the PC highend graphics world and then trickle down to VJ land.

so yeah PCs will have all this stuff but I reckon the macworld has a six month lead here.

[MoRpH]

I actually wasn't taking the piss, I have actually been considering a mac as my newxt laptop for a while now (I have 3 PC laptops and I want to experiment with mac software)... so this is actually cool interesting news :)

[Nema]

Originally posted by sleepytom
why? - the hippo is software processing only - it has no 3d rendering afaik.

that was correct for the very first version of hippotizer (about two years ago). it is no secret that it used the visualJockey kernel as its first base (which prooved to be a smart choice). but... ...did you think that the developers have been just sleeping since? i had to completely refresh my knowledge in Direct3D, DirectShow (arrrgh! it is horrible in comparison to the good old times, but well, in combination with new gfx-adapters it amazed me to be incredible powerful!) and of course pixel- and vertex-shaders (which is the biggest fun i ever had in my coder career!)

so, in short: meanwhile the whole hippotizer-engine and every plugin has been recoded from scratch, using all the newest features of modern graphic-cards. if you want a demo i can recommend to visit scene-changes exhibition stand at http://www.plasa.org/