brainstorming GPUCHAIN

[klumsy]

i'm here to brainstorm, and maybe inspire some other developers to join a lofty cause. The goal:

OVERVIEW

the main a chainable plugin framework such like Freeframe(or visualjockey, or directShow), which every stage is done within the GPU with multiple inputs and an output (maybe multiple outputs), every stage (plugin) is fully self contained other than the inputs and outputs and general framework.

WHAT TO CALL IT

the name i'm not sure? GPUCHAIN? SHADERFRAME? anybody dare to come up with a catchy and accurate name for it?

DIRECTX OR OPENGL

such a project would be not as flexible as freeframe in a sense because of its physical tie to DirectX or OpenGL.. because GL is behind the industry (because of commitees taking years )and often features are implemented using different Nvidia and ATI extentions and because i like the pipeline of directX 9 much better i'm planning to go the directX route rather than openGL - However the overall framework (as consumed by the host) can be made to work with both, except there would basically be plugins that are directX and plugins than are GL, and only DX ones will work in a DX host etc.. so i plan to design this in the context of directX, but others can be working in parrallel and in combination for non hardware specific things on GL solution.

USING SHADERS? GEOMETRY
would this be for shaders? yes and no.. one of the biggest users will be pixel shaders, where one or more input textures and transformed by pixelshader code and rendered to an output texture (which is then used by the next plugin in the framework), however certian effects are good as pure pixelshaders, while others are better served by geometry etc. so this will be for SELF CONTAINED GPU EFFECTS which take in one or more inputs (TEXTURES) and output ONE(possiblly more?) TEXTURES.. it can use whatever GPU methods inside.. PIXERLSHADERS, or geometry, vertex shaders or combinations.. for example a dirtion effect is best not produced with pixelshaders (alone), but with a mesh/grid and moving the vertexes (maybe with a vertex shader), however its self contained.. its a 3d scene that it is internal to the plugin/effect.
what version of shaders do we support? i think it will just be the latest, meaing it won't work on anything but the latest generation of hardware.. mostly because pervious generations of pixelshaders weren't really powerful enough other than for simple 3d shading effects, not really for our 'software 2d in GPU effects'

Software inputs
can you combine with software? for INPUTS you can, maybe put a freeframe plugin as the input to a GPUCHAIN plugin, however all outputs remain on the card to be used by the next PLUGIN, or for final rendering to screen.

Technical Design
here is quite brainstorming of features.

1) Library - there will be a library DLL with alot of helper/framework functionality that the plugins can use
2) INPUTS
2-1) each plugin can specify what its inputs are (how many)
2-2) each plugin can DYNAMICALLY specify whether or not an input
is needed at a particular frame (thus not having to render upchain if not needed.
2-3)each plugin can specify the resolution of a particular input, (either letting the system make the decision, specifying a specific X,Y resolution, specifying an aspect ratio, or specifying a X and y percentage of the output resolution.
3) INPUT VARIABLES
3-1) like freeframe there will be boolean and floating point input variables, but as well, integer variables, and maybe even strings? (for text effects)
4) OUTPUTVARIABLES
4-1) there will be an output variable section, where a plugin can register output variables (in a group that it named), and it can set these output variables as it wishes, and the host engine will keep track of these, and show them to its app as it wishes, consuming them as they wish, or allowimng the user to then map these to other plugins input variables.
5) CONFIGURATION SCREENS
5-1) often plugins require configuration to use them in a flexible and powerful way.. i vdon't know the best way to do this though, since it limits the application alot, tie into a specific UI model and i don't want to do that.. so probably there won't be any configuration screens, but maybe a some XML or suchlike configuration files, or a some sort of configuration varriables communicated between plugin/engine and host.
6)constructor/destructor to take care of basics
7)initialise method - when the host initialises the plugin, in here to do alot of the setup
8)render method, whether the host wants the plugin to do its business.
9) what about different texture formats? 16,32 bit etc?
10) what kind of shader techniques? using HLSL or shader ASM etc? - well since that is internal to the plugin the author can choose, but the helper library will have alot of helper functions to get you on the way, probably HLSL will be the best..
11) directshow output to inputs for our textures.
12) specific 'transition' plugins..
13) a built in copyright tools/library for commercial plugins to be able to use?
14) GPU ^2 texture resolution restraints.
this is just brainstorm #1
alot of its design is going to be changed as we progress with proof of concept apps, discover new oppertunities/limitation etc

there is still alot more issues.. but that is a start, others ideas, thoughts are welcome. who is keen on being a part of this?

in many ways this is probably similar to apple's resolution issueCoreVideo (http://www.apple.com/macosx/tiger/core.html ) but hopefully even better.

Do i have a team? if so i'll start up a wiki to work on this, and we can debate different issues, start proof of concepts and work on it.

[DrEskaton]

If you want this to be an open standard you should write the shaders using OpenGL ARB extensions not DX9. Otherwise it is limited to Windows only, wouldn't it be great to be able to recompile shaders with no changes for linux or mac os x?

Yes the ARB Extensions for shaders are different on Nvidia and ATI but as I understand it ATI and NVidia's extensions of Pixelshader 2.0 and 3.0 in DX9 are also different so you will still have this issue.

Sure there may be a slight performance hit using OpenGL ARB Extensions but wouldn't the benefit of true multi platform and a true open standard be worth it?

[klumsy]

If you want this to be an open standard you should write the shaders using OpenGL ARB extensions not DX9. Otherwise it is limited to Windows only, wouldn't it be great to be able to recompile shaders with no changes for linux or mac os x?

open doesn't mean it has to be cross platform, i'm perfectly happy with it being on windows, and also other developing a costandard with somebody else that is openGL, but the GL extentions are SOOOOO different and not just in the shader area, it takes more time to develop, has less features, on PC card vendors often don't put much effort and don't do a good job on their GL drivers etc, and Gl extentions aren't up with the latest level of shaders ,and then there is integration with directShow and such... its not my priority, though i am happy to work with some poeple.

Yes the ARB Extensions for shaders are different on Nvidia and ATI but as I understand it ATI and NVidia's extensions of Pixelshader 2.0 and 3.0 in DX9 are also different so you will still have this issue

not of you use HLSL, then its the same.. MS really got their stuff together with directX9, just released DX9c has pixelshaders 3.0 support - all ready the software for future shaders that aren't even in existence yet, compared to Gl extentions that are behind the currently level of hardware.

and DX10 pipeline rocks even more.

however as i said, everything other than the actual hardware specific things in this framework can be hte same, irregardless of GL,DX.. just like you can make freeframe plugins using PC that restrict its use to windows,, however you can make a plain C version as well for cross platform.. if we make a standard, with a GL implementation and DX implementation, people can make plugins for each or both according to their desire, but i'm personally keen to work on the DX one.. at least to start.

[nkm]

Interesting project, count me in!

I've already got some sort of internal standard for the filters used in Pilgrim. These are based on vertex- and pixelshaders up to versions 1.3 currently.

For the next version of Pilgrim (R1.5) I'd like to support chaining of filters and extending shader capability to 2.0 at least.

nkm

[sleepytom]

is this thread sposered by bill or what?

if you want to develop an open standard for GPU effects it is stupid to start off with DirectX - DirectX prevents you from producing a crossplatform app and it also seriously restricts the kind of licence you can work under.

Your OpenGL bashing is very sad to see - in what way is GL less powerfull than directx? sure it is harder to code in openGL but that doesn't make it less powerfull (in actuall fact it makes it much more flexable as your not reliant on doing things the micro$oft way)

if you were to start form an open, crossplatform standard such as openGL then you would have much more chance of getting help from some seriously tallented programers who are currently working on big GL based projects at apple (ie motion and corevideo) you would also be in a much stronger position to intergrate into existing VJ software (the vast majority of VJ software that uses 3d is using OpenGL not direct3D)

i'm sure that the msdn will tell you how powerfull and great directX is but it is often very fancifull with its discriptions of how well the technology works - GL on the other hand is very established and tested and has a bigger user base who are intrested in open source development.

[elbows]

I would hope that the issue of which standards to use are solved by practical considerations, cos I doubt we will all totally agree from a philosophical point of view! In other words I very much support the view that the framework should be designed with both opengl and dx use in mind, and which one gets developed first/most will be determined by the coding skills of the people involved with the project.

I would like to offer my help to this project, though I wont be much use Im afraid. I lack programming skills, but would love to be involved with the design or even just suggestions etc. I can just about make sense of SDK documentation etc, but I cant write any code, but if this is of any use let me know lol.

[syzygy]

A question that has been puzzling me for a while...

How do graphics cards deal with multiple pixel shaders?

Presumably there is a limit to how many pixel shaders can be running on a single card at any one time.


Some of the directX reading I've been doing has hinted that oinly one pixel shader can be used at a time (e.g. the SetPixelShader function replaces any existing pixel shader with the new shader) - I presume that, since people are talking about using chains of shader-based effects, multiple shaders can in fact be used.

Can anyone clarify this for me?

Cheers,

Dan.

[klumsy]

is this thread sposered by bill or what?

if you want to develop an open standard for GPU effects it is stupid to start off with DirectX - DirectX prevents you from producing a crossplatform app and it also seriously restricts the kind of licence you can work under.

maybe stupid, but its something i've thought and researched about. and how does it restrict the kind of license? GL is only truely crossplatform at its basically, and most complicated GL stuff has a huge part of its codebase in different versions for different platforms and cards. (DX is the same to a point with cards as well), i didn't choose DX because i'm a bill gates buttkisser or anything. Actually coming from a delphi background GL is my natural instinct with having all the headers there (directX can be a pain in delphi, because every new version the coommunity has to trial and error conver the API, and build a wrapper around D3DX libraries because its a static lib!!!arrgh), but GL is much easier to work with, has a quicker learning curve etc.. i would have rather worked with GL, but there are other factors to take into consideration, if i had posted this in the game industry all the professional developers (many of my friends, from when i worked on final fantasy 7) would agree with me, up until this last laptop i have i've never had a laptop with an ATI card that had any decent GL drivers, so many bugs and such bad preformance, the only reason the current one works is because the apps i am using are using feastures been in cards for years.. i'm not ANTI GL, i understand that committee based standards take time to get updated, do you realise how many years after 3d cards supported render 2 texture for GL to actually support it? Also the other thing, is what programs/technology am i INTERFACING with, and it is windows apps, and most of them are directX, and without hacking into each and every GL and directX driver there is no mater to interlope directX and GL WITHOUT bring the texture back to main memory, which defeats the whole purpose of a hardware pipeline. so i have to go with directX, especially if i want hardware decoding of DVD/mpeg2 inside video memory or i want to support HD inputs and outputs on pc platform within the immediate future. don't miss my main points, and purpose of this and let DX/GL like PC/MAC religious issues get in the way. for me GL is easier..

BUT REMEMBER OTHER THAN INTERNAL ASPECTS OF THIS STANDARD, IT IS CROSS PLATFORM. but i want it cross technology as well, and there that means GL and DirectX, but they won't work together, but can support the same framework, just as plugins will expose properties of what their minimal requirements regards hardware features, shader language level etc, they will also specify whether they are directX or GL... i'm going to start on the directX 1) because i have limited time, and programming in HLSL means i don't have to make and test nvidia and ati veersions of each effect. 2) because I have to interface with directX mostly, 3) because of greater driver support and
4) because its fun getting you all riled up (just joking)
I AM MORE THAN HAPPY FOR OTHERS TO JUMP ON BOARD AND PINEER THE HELPER LIBRARY AND EXAMPLES IN GL? are you keen?


Your OpenGL bashing is very sad to see - in what way is GL less powerfull than directx? sure it is harder to code in openGL but that doesn't make it less powerfull (in actuall fact it makes it much more flexable as your not reliant on doing things the micro$oft way)

i am not blinding bashing it, its just its only been patched on (in the same horrid way windows has) and was initially designed on a framework of very different hardware than we have today, one of the only advantages of direct3d is significant rewriting of the t echniques to keep up with card technology. i dislike MS, but it is true. if i didn't want to be reliant on MS i'd be writing my own OS like in the olden days, i don't have time for that.



if you were to start form an open, crossplatform standard such as openGL then you would have much more chance of getting help from some seriously tallented programers who are currently working on big GL based projects at apple (ie motion and corevideo) you would also be in a much stronger position to intergrate into existing VJ software (the vast majority of VJ software that uses 3d is using OpenGL not direct3D)

i doubt apple would really want people to work on a competitor to their own CoreVideo. If i wanted help from my game programming buddies directX would be more successful in getting help.. but the point is this is to be BOTH, but I am going to start on the overall thing, and specifically directX implementation.
as for existing VJ apps, point taken. i'm not up with all the different softwares out there whati know off the top of my head (could be wrong on some) can people add to this list. i don't know about mac ones.

visualjockey - directX
hippotomiser - directX
TZT - directX
resolume - directX?
pilgrim - i assume directX from authors post above?
vjamm - openGL
xynth - openGL
i know there are more openGL stuff, just can't recall..
and then there are going to be my apps :)


i'm sure that the msdn will tell you how powerfull and great directX is but it is often very fancifull with its discriptions of how well the technology works - GL on the other hand is very established and tested and has a bigger user base who are intrested in open source development.

i don't make my decisions on MSDN, i use MSDN for reference, i make my decisions on certian implementations in games/ engines by people i know and respect in the game industry and demoscene.
i'm aiming for a VJ version of this
http://www.theprodukkt.com/werkkzeug1.html#download

and more.
lets put aside directX / GL battles and work on making something good and cool, there will be people who use and utilise both sides (and people who will use both in one app, if they feel they can afford a read from videocard to memory (if its just a one off thing), and after PCIexpress is out, many more people will probably do that.. i know you are a vjamm user and don't want to see a technology made that won't benefit your program, but the way this program will be made, with the general architecture being abstracted from any hardware, once the directX is implemented and the external interfaces set, it would only be a weekends job to make some examples in GL..
if i were just doing everything the MS way, i wouldn't be bothering to make such a standard but would just write code inside directshow filters, but alas i am not..

[klumsy]

Some of the directX reading I've been doing has hinted that oinly one pixel shader can be used at a time (e.g. the SetPixelShader function replaces any existing pixel shader with the new shader) - I presume that, since people are talking about using chains of shader-based effects, multiple shaders can in fact be used.

i'm not to sure about differenrt shaders being used on different things in a scene, however with chaining them, its basically rendering a scene to texture using shaders with the scene just being a simple billboard..
similar to rendering a sceen to texture, then next effect using that scene then rendering to texture etc

[syzygy]

Cool, I think I get it. So the app would do one pass for each effect in the chain, rendering to a texture each time, then using that texture on a surface in a minimal scene for the next pass.

[sleepytom]

visualjockey - directX
hippotomiser - directX
TZT - directX
resolume - directX?
pilgrim - i assume directX from authors post above?
vjamm - openGL
xynth - openGL
i know there are more openGL stuff, just can't recall..
and then there are going to be my apps


this isn't quite correct - there is a difrence between the video playback engine (directshow / vfw / quicktime) and the display engine (directdraw / directshow / direct3d / Opengl)

as the only apps in your list that are running a 3d display system are pilgrim, vjamm and xynth then openGL is a clear winner. :P

look at other apps that use 3d display systems and the GL side becomes even stronger, touch designer is GL as is grid+vdmx (well they use that GL extension for max), in actuall fact the only app that i know of to use Direct3d is pilgrim, everyone else uses OpenGL.

there was discussion of the concept of a shader based plugin system a while ago on the freeframe dev list. perhapse it would be timely to bring it up again there - it would be good to build on the success of freeframe as many plugin developers are allready involoved in the project.

[dongbamage]

I think directx9 is thge way to go using hlsl, looks good.

anyone play with ati's rendermonkey? shows some of the stuff we want...

[cat]

Actually vvvv is directx 9 as well, and supports pixel shaders directly within it, ie you can write hlsl code in the app itself, or use *.fx files.
Would something like that not be the way to go, ie using *.fx directly. But Im not a coder so you can ignor me if you like! Having used pilgrim and vvvv, although my gfx card dosent support all the features yet, these apps have persuaded me to upgrade! I think pixel shaders rock, put an fx on and incur no hit to the app is amazing!
I have to say of the apps you've mention Tom, the ones Ive seen and/or used that use openGL always look a little washed out and flat in the actual ouput quality. Now I dont dont know if thats openGL or the way the apps been coded, but directx has always seemed a little deeper and more contrasty. Out of interest is core video open gl based or a mac proprietry brand ie directMAC?
Ive been led to believe that the programmer for Catalyst works on the MAC development team, which would imply to me that catalyst and pixelmad use (and possibly helped develope ) core video. Which would mean that unless PC's get a kick up the bum they'll be left eating dust, so my vote is to go for the closest to hardware route you can!
But as I said I'll be cheering from the wings.....
Go team GFX!!!

Cat

[sleepytom]

the *shouldn't* be about a PC/Mac thing but about creating an open standard for realtime pixleshader video effects - crossplatform compatability should be a priority so that the standard has as much chance of become accepted as possible.

OpenGL is allready the established crossplatform 3d system - why limmit yourself to winblows / directx when starting out on a project like this?

if the new mac stuff is as impressive as people say it is then the last thing you want to do is create a video effects standard that will never work on the mac - in two years time how many VJs will own macs? it'll be a grater percentage than today if corevideo is any good at all - do you really want to exclude these potential users/ programers from this project?


as to openGL looking "soft" - umm i think your just plain wrong here OpenGL is used in very highend systems (think sgi render clusters) for doing film resolution compositing and effects - chances are your gl settings are wrong on your graphics card

oh and the catalist system is fully OpenGL

[elbows]

Well the bottom line is for some interested OpenGL programmers to raise their hands then, I suppose it is necessary to debate the pro's and con's of each here, but let keep it positive eh.

Meanwhile, cat many thanks for the vvvv tipoff, I had no idea it had shader support now, Im off to investigate.

[cat]

I've no particular axe to grind, I just want tools!
The open Gl that Ive seen look soft is from Vjo and Touch, touch being based on one of the High Open GL systems, so maybe its just the realtime optimisations? Or maybe my card, or shit drivers...?
If catalyst is open GL then its almost certain that core video will be open gl (according to my reasoning above). If you want full cross platform then surely the thing to do would be to port core video, but I think you'd get sue'd by Apple! And sure nuff if core video takes off and theres nothing comparable for PC then there will be a shift to more mac owners, I just dont have that kind of diposable income at the moment, its not that I wouldnt like one. But isnt *.fx the pc equivellent?The Hlsl code is meant to work for ati, nvidia and also there is an open gl version of it I believe(?)
Its meant to be easier than C too, but C looks like everest to me at the moment, maybe I should just start climbing....
Does anyone know if microshite are planning their own equivelent?

The shader support in vvvv made my jaw drop to be honest, the code is literally a right click away, compiles on the fly, just cant see anything cause Im too cheap skate on my gfx card, just drop another couple of quid......

[klumsy]

OpenGL is allready the established crossplatform 3d system - why limmit yourself to winblows / directx when starting out on a project like this?

HOW MANY TIMES DO I HAVE TO SAY THAT I AM NOT LIMITING IT TO DIRECTX WINDOWS. THAT THE MAIN SYSTEM IS ABSTRACTED FROM ANY HARDWARE/PLATFORM DEPENDENCIES.
the fact is , I and other coders NEED to interface with directd3d systems and directX and GL CAN'T SHARE TEXTURES IN CARDMEMORY WITHOUT they have to bring the data back to main memory which is as preformance it. basically each plugin will have self describing details saying what graphics technology (directX, openl GL etc, and the inputs and outputs passed as (VOID *) with reference a pointer to either a GL texture object or a DX one..
openGL is portable, at least 80% of it, the problem is the area i am interested in is the other20%, there is no such thing as purely portable unless you settle for the lowest common denomintor. C/C++ is MEANT To be a portable language, however how many times have you had problems getting some C++ code to compile on a different compiler let alone a different platform.. the fact is in the REAL world, useful code often has to skip the barrier of interal reference, to platforms and hardware.. How do many game makers make portable games that work on so many different platforms (and different engines, like ps2 emotion, openGL, and directX... they keep all the main code abstacted and seperate from platform dependencies, and write different platform depentant code for each platform, sometimes this takes away some features or preformance they have have to write some clever hacks as well.. GPUCHAIN IS ABSTRACTED from ANY HARDWARE IMPLEMENTATION.. though if people want to write an effect that works on both openGL and directX, they will either have to make 2 versions internally or externally... but highly optomised freeframe plugins do this all the time.. they make different versions depending on whether a computer has AMD, pentium3, pentium4 chip, whether they support MMX, 3dNOW, SSE,SSE2 insructions that give graphics code HUGE preformance benefits, and often different versions of highly optomised code for 32 bit, 24bit and various 15 and 16 bit graphics formats.. its just that YOU the consumer may be unaware of such different 'platforms' that it supports internally.. also when Gl code is ported to different systems.. you'd be surprised at hte HUGE amount of work that people do to utilise the very different GL extentions there are, test them on SOOO many different cards, make work around for driver bugs, or different GL extention creators 'ideology differences' (similar problems with DirectX as well).

WHY ARE YOU INSISTING THAT I DROP DIRECTX SUPPORT AND ONLY SUPPORT GL, when i have said so many tiems that this framework will be flexible enough to add GL in, but I PERSONALLY won't be doing this IN THE BEGINNING, this standard will take a few PROTOTYPE revisions anyway.. its its probably better for GL to wait until the spec is solid..
WHY WOULD I WORK ON A GL SYSTEM ONLY WHEN IT DOESN'T FUFILL MY PERSONAL NEEDS?? if this standard didn't include DX , i personally wouldn't be working on it.

MAN I CAN'T BELIEVE THE GL-protectionistic manifestations in this thread, especially from in the beginning i said GL would be part of the standard.

if the new mac stuff is as impressive as people say it is then the last thing you want to do is create a video effects standard that will never work on the mac - in two years time how many VJs will own macs? it'll be a grater percentage than today if corevideo is any good at all - do you really want to exclude these potential users/ programers from this project?

I AM NOT EXCLUDING ANYBODY!!!!! if coreVideo is so good, (which it probably is) mac developers wouldn't really want to use GLGPUCHAIN unless it is much better and then people would have to write wrappers on mac around corevideo for that as well.. i'm making a competeing product to corevideo that is for wintel, but flexible enough to be cross platform.. and if it takes off and people write various DX and GL plugins, i'm sure a host would be implemented on a mac..

OK, NOW LETS STOP THIS BICKERING..
IF MY FIRST PORT, i wastn't running down GL, i like it personally as a preference, i know many weaknesses of DirectX, but i also can admit some of its strengths, especially in recent years, to the point that DirectX is often directing the direction that hardware manufactureres are going in.. i pointed out the weaknessses of GL to explain my reasons for my personally implementation being around directX , (Though scalible enough to include GL) (For example openGL 2.0 has been released very recently, and the games industry is very dissapointed.. it was meant to be a rewrite completely bringing up some antiquidated methods to modern day standards, but it basically just put a few extentions into the main standard and added some stuff. but hey GL works for many things, and i actually got personal GL projects i'm working on (integrating visualjockey with R4). if you want to see the quality i am looking at getting in 3d, download
http://www.theprodukkt.com/werkkzeug1.html
and the 'the popular demo' work file and see that in action, and that is using OLD CARD technology.

i am now going back to edit my first post, and put it in a new thread, that can be more constructive rather than an argument.

Karl

[sleepytom]

WHY ARE YOU INSISTING THAT I DROP DIRECTX SUPPORT AND ONLY SUPPORT GL, I'm not :confused: i'm mearly pointing out that GL support is vital if you have any intretest in this ever being a crossplatform / cross compiler project.

it seems that you understand this - however i'd point out that passing responciblity to plugin developer to code for both GL and directX makes plugin dev a lot more complex than doing one or the other.

one of the reasons freeframe has been so sucessfull is because it is very simple to write plugins - hence more plugin developers - hence more plugins - hence more pressure for apps to support freeframe -- it all feeds round into itself strengthening the project and making it into an established standard.

[klumsy]

it seems that you understand this - however i'd point out that passing responciblity to plugin developer to code for both GL and directX makes plugin dev a lot more complex than doing one or the other.

i'm not forcing people to write 2 versions of plugins, just like in freeframe there are plugins written in delphi that aren't ported to linux/mac.. there are plugins that use winapi calls (i.e drawing text with GDI, that limit to windows), plugins that use MMX thus again restricting to windows.. or ones that could be ported, but aren't.. there are plugins that only support 32 bit , not 16 bit (will fail miserably).. the point is plugin developers will only support the features they need, and will overlook or fudge the rest on most of the cases..
i'm not planning on making it complicated.. i plan on making extensive d3d helper library , and hope for such an extensive GL one as well, with code examples in the SDK on building shaders with CG, gl_arm, HLSL, nvidia and ati shader assembler etc, as wel as vertex shaders, and plugins that don't use shaders at all, but rather distort etc with a self contained 3d effect, as for a developer the only NEw thing they need to know is the general framework issues - seperate from any hardware, they will already have to know d3d or GL and a pixel shader technology to make any such code anyway, however the helper libraries should even cut down on that requirement

i plan on making some generic plugins, that are fully script based.. such as using a cross platform LUA script to specify the inputs and all that, and deal with processing the input variables into constants used by a pixel shader, (sos users can just specify a LUA script, and also specify a HLSL script as well.

also i plan on making such a similar 'generic' plugin to deal with LUA , and CG.. such a plugin could have 2 versions made, a GL and a D3d version, with the lua, and CG scripts working in BOTH.. there is fully cross platform support there.. you can do alot of stuff in it, however its just a subset of the whole game, and i don't want to restrict GPUCHAIN to that scenario (i.e i want people to say, hey i can turn my d3d, or GL effect that renders to a texture into this.




one of the reasons freeframe has been so sucessfull is because it is very simple to write plugins - hence more plugin developers - hence more plugins - hence more pressure for apps to support freeframe -- it all feeds round into itself strengthening the project and making it into an established standard.

that is true, but you also have to admit the reality that alot of the plugins aren't portable, and you'll get the same with CPUCHAIN whether or not i make it d3d GL or both, you'd get people making stuff that works only on nvidia, or only on d3d, or get people who make code that adapts to older cards, but is even cooler on newer cards.. the this is, this gives everybody flexibility..

[johnnylocust]

If the focus of the application is on the shaders, whether you use GL or dx is rather moot. The shader language for GL and directx are so similar that nVidia's CG language can be used to target both platforms :yep: If I understand correctly, you're working torwards usign the GPU for visuals, and not so much 3D. Rendering a square using a series of shaders and a video feed can be made with a handfull of lines of code for either GL, or directx. My recommendation however would lean more torwards directshow. The vmr9 component is amazingly powerfull. Contrary to popular belief, it's very simple to implement. The multivmr9 demo in the SDK has everything done for you already. It's a com object, so it doesn't get any easier.

[klumsy]

If the focus of the application is on the shaders, whether you use GL or dx is rather moot.

its still valid since it has to be hosted in one enviroment or another.. as i posted above though , i do plan a plugin for this architecture that allows fully scripted scenarios (lua for pre and post variable stuff) and CG for the actual thing, this would work in both scenarios


If I understand correctly, you're working torwards usign the GPU for visuals, and not so much 3D.

yes and no, each stage if 3d will be self contained.. alot of '2d looking effects' are best achieved with 3d geometry. so plugins can do that.. or just plain pixelshaders etc


. My recommendation however would lean more torwards directshow. The vmr9 component is amazingly powerfull. Contrary to popular belief, it's very simple to implement. The multivmr9 demo in the SDK has everything done for you already. It's a com object, so it doesn't get any easier.

directshow is nice, however it doesn't have the same parameter/variable system that i want, and it is fully asymetrical, which is fine for video generators etc, but very annoying for graphic filters, but the graphic filter thenselves are software based, on in the GPU, what iam basically trying to make in like a Dshow filter architecture within GPU .. vmr9 is nice, but is just the final putting to screen that is done in hardware (unless its using hardware to decompress video).. i want to chain lots of effects ontop fo that video, once its in hardware, this plugin architecture if for that purpose. you can be sure that there will be a vmr9 directshow plugin as a 'generator' for this architecture :)

[dongbamage]

I'm loving pixelshaders but I'm wanting 3d too!

Klumsy give yerself a hug mate! :yep:

I'm glad that ppl are taking it seriously. Both sides of the argument must be said so that we can get the best prog that we can make.

I've been impressed with the directx9 implentations I have seen, and I know that they have been working with hardware vendors. The api is also nice...

Lets get this rolling together as a community WE can make something that WE want! Its a real opportunity lets not waste it,

[DrEskaton]

just a queston? Doesn't DX9 have "directshow filters" which would bascially do what you guys are proposing. i.e. resuable 2d image processing filters using the GPU?

Sorry to be ignorant but how would GPUCHAIN filters be different from directshow filters?

[murk]

Originally posted by DrEskaton
just a queston? Doesn't DX9 have "directshow filters" which would bascially do what you guys are proposing. i.e. resuable 2d image processing filters using the GPU?

Sorry to be ignorant but how would GPUCHAIN filters be different from directshow filters?

This is the same question I was having. I am sure Klumsy has reasons, but it would be good to make sure that the project isn't reinventing the wheel.

[klumsy]

just a queston? Doesn't DX9 have "directshow filters" which would bascially do what you guys are proposing. i.e. resuable 2d image processing filters using the GPU?

Sorry to be ignorant but how would GPUCHAIN filters be different from directshow filters?

you mustn've read half of my posts :)
i like directSHow, but directshow filters are software based, in main memory, think of directshow like a freeframe plugin, that has more features (deals with audio) has output pins, is realtime anad not realtime as well.. dshow also is async with each fliter in the filter graph managing lots of threads and is difficult to throttle..
GPUCHAIN will be differnt in the sense it will be visual only, will be completely inside the GPU, can contain 3d scenes at each stage, can use pixel shaders, but also its interfaces with apps will be different (i.e the variables past into it, as well as the variables that a plugin can generate that cana be consumed by the host app.. CPUCHAIN will have powerful ways to turn off chains not in use (for preformance) and throttle things by being syncronous by nature, but asynronous for certian features..
using directShow is more tedious than other directX, and making directShow filters is even more tedious.. (if it wasn't so, dshow would be used instead of freeframe already).. i'm wanting to give a framework that is powerful, flexible, inside the GPU, siuted for realtime visual applications, VJ specific, and crossplatform,open.

[syzygy]

I think this would be a really positive companion to freeframe.

Freeframe would continued to be used for effects that target specific layers within VJ apps.

GPU chain would be used for post-mix effects on the completed composition, right?


I think the important things are:

* Keep it simple! As people have pointed out, the fact that Freeframe has become a success is down to its initial simplicity. Don't be scared to leave features to version 2 of a spec in order to bootstrap plugin development and host support more quickly.

* Make it as widely compatible as possible, but don't pursue compatability at the cost of complexity or never-ending development. (i.e. stick to your guns and pursue the D3D line at first if that will get something useful working)

* Consult widely with VJ app developers, plugin developers and users (VJs) throughout the process to make sure the standard is what everyone wants and needs.

* Make sure the project works in tandem with the freeframe project, supporting each other.



I don't have any D3D or OpenGL experience, but I'm interested in being involved with this project. The availability of a standard for shader and 3D based plugins would give me the incentive I need to learn GPU programming. I don have experience at reviewing API specs, so maybe I can be useful at some stage...

[DrEskaton]

Originally posted by syzygy
I think this would be a really positive companion to freeframe.

Freeframe would continued to be used for effects that target specific layers within VJ apps.

GPU chain would be used for post-mix effects on the completed composition, right?


why split it up like this? gpu effects let you do things like really nice realtme blurs and glows. You should be able to use these on individual layers as well as the final comp.

wouldn't GPUCHAIN just be an alternative API to freeframe that places more emphasis on implementing the effects as realtime GPU pixel shaders. Freeframe is more generic and includes software rendering and OpenGL effects.

[sleepytom]

no it wouldn't be an alternative API to freeframe it would be complimentory.

for example a host might choose directshow video playback > freeframe (cpu processing) > GpuChain (gpu processing) > Screen (output)

- GPU chain would be an extra layer of effects at output - it would depend upon the host implementation as to wether the effects were available to apply to each layer/ clip or just as a master output effects chain.

if we use resolume as an example then it would be relitivly simple to add GPUChain as a master output effects chain - however adding GPUChain effects to individual layers or individual clips would require a total rewrite of the host as it is not sencible to send data to GPUChain only to get it back into the CPU for further processing (ie mixing, keying, compositing etc).

really GPUChain only makes sence if you are using OpenGL or Direct3d for the mixing / compositing stage of the app - then the data is up on the graphics card allready and you don't have to deal with the agp bottle neck to get access to the GPUChain effects.

hmm a block diagram would be very handy at this point...

[syzygy]

Originally posted by DrEskaton

wouldn't GPUCHAIN just be an alternative API to freeframe that places more emphasis on implementing the effects as realtime GPU pixel shaders. Freeframe is more generic and includes software rendering and OpenGL effects.

As sleepytom mentioned, the problem with using graphics card based techniques such a pixel shaders on individual layers in the vj app is that lots of data will have to be sent up and down to the video card and back to main memory.

As long as the video card is rendering to its own memory, things can be nice and effcient. As soon as the data is sent back down for the vjapp to combine with other layers or freeframe effects, you'd lose the efficiency in the transfers.

This kind of system would also be much more complex for vj app developers to support, making it harder for the standard to get into widespread use.

Dan.

[DrEskaton]

Originally posted by syzygy
As sleepytom mentioned, the problem with using graphics card based techniques such a pixel shaders on individual layers in the vj app is that lots of data will have to be sent up and down to the video card and back to main memory.

As long as the video card is rendering to its own memory, things can be nice and effcient. As soon as the data is sent back down for the vjapp to combine with other layers or freeframe effects, you'd lose the efficiency in the transfers.

This kind of system would also be much more complex for vj app developers to support, making it harder for the standard to get into widespread use.

Dan.

Ok I understand the going back through main memory is bad but as I understand it basic compositing operations like Blend / Addmix / screen are pretty easy to implement as pixelshader effects so wouldn't it make sense and be faster for all layer compositing operations to be done as part of GPUCHAIN on the card.

This would mean Toms diagram would become:

Directshow Video -> Freeframe (individual layer effects) -> GPUCHAIN layer effects -> GPUCHAIN compositing of layers -> GPUCHAIN final effects on rendered output.

Don't a lot of newer VJ apps eg Vjamms OpenGL compositing, Modul8, Pilgraim do all layer mixing by rendering movies to billboard textures so they could easily do GPU effects just on one single layer texture?

Apple's CoreVideo will definately include this kind of functionality so if you are making a workalike then it needs to allow for this kind of workflow as well as effects on rendered output.

[syzygy]

agreed that vj apps could move to doing pretty much everything on the graphics card, just decompressing video and providing frames to an effects/mixer chain that is executed in the GPU.

The problem is that the harder it is for vj app developers to support a standard, the longer it will take for them to do so and the more possibility is that some will not bother.


supporting a simple standard for post-mix GPU-based effects will be relatively easy.

Moving to using the GPU for layer effects, layer mixing and global effects will require considerably more reengineering of existing applications.

Using the GPU for mixing will also up the hardware requirements for VJ apps - every user will need to have a graphics card that can handle the pixel shaders needed for the mixing modes.

Why not take this one step at a time? The approach of keeping things simple at first has worked really well for Freeframe and I think it would work well for GPUChain too.

Dan.

[petewarden]

The bare minimum needed for a GPU effect plugin API is a render function that looks like this:

render(inputTexture, width, height);

In OpenGL and C this would be:

void render(GLuint textureID, int width, int height);

and the simplest implementation would look like this:

void render(GLuint textureID, int width, int height)
{
glEnable(GL_TEXTURE_2D);
glBindTexture(textureID);

glBegin(GL_QUADS);

glTexCoord2f(0,0);
glVertex2f(0,0);

glTexCoord2f(1,0);
glVertex2f(width,0);

glTexCoord2f(1,1);
glVertex2f(width,height);

glTexCoord2f(0,1);
glVertex2f(0,height);

glEnd();
}

This function simply copies the input to the output, with no changes.

A filter running a simple pixel shader using ARB_fragment_program would be a little more complex:

char* invertProgram =
"!!ARBfp1.0\n"
"# Pixel shader to invert the RGB channels of the input \n"
"TEMP input; \n"
"\n"
"# Read the input color from the texture
"TEX input, fragment.texcoord[0], texture[0], 2D; \n"
"# Subtract it from 1 to invert all the channels \n"
"SUB result.color, {1, 1, 1, 1}, input; \n"
"END";

GLuint invertProgramID = 0;

void render(GLuint textureID, int width, int height)
{
if (invertProgramID==0)
{
glEnable(GL_FRAGMENT_PROGRAM_ARB);
glGenProgramsARB(1, &invertProgramID);
glBindProgramARB( GL_FRAGMENT_PROGRAM_ARB, invertProgramID);
glProgramStringARB( GL_FRAGMENT_PROGRAM_ARB, GL_PROGRAM_FORMAT_ASCII_ARB, strlen(invertProgram), invertProgram);
}

glEnable( GL_FRAGMENT_PROGRAM_ARB);
glBindProgramARB( GL_FRAGMENT_PROGRAM_ARB, invertProgramID);

glEnable(GL_TEXTURE_2D);
glBindTexture(textureID);

glBegin(GL_QUADS);

glTexCoord2f(0,0);
glVertex2f(0,0);

glTexCoord2f(1,0);
glVertex2f(width,0);

glTexCoord2f(1,1);
glVertex2f(width,height);

glTexCoord2f(0,1);
glVertex2f(0,height);

glEnd();
}

These are typed from memory, and so may not compile straight off, but are both similar to code I've used extensively in production software. It looks like the forums chewed up the formatting though. They make some assumptions, including power-of-two input texture sizes, so GL_TEXTURE_2D can be used rather than one of the rectangle extensions, and that a context has already been set up to draw into before the render is called, with a projection matrix so that (0,0) is at one corner of the screen and (width,height) is at the opposite, they do no error checking, and the shader inverts the alpha channel.

I'm sure similar examples can be created for DirectX.

The most important point is that the interface can be very minimal, and doesn't need to make assumptions about how the effects are implemented (shaders or not) or where they will be used in the rendering chain. As Dan says, a lot of host apps may need re-engineering to cope with more GPU stuff, but that shouldn't affect the plugin standard.

Any standard should be extremely thin, and set out the bare minimum needed for plugins and hosts to cooperate. Anything more complex will be tough for hosts to implement, and prone to going out of date very quickly.

Pete

[elbows]

OK after spending a few hours learning the basics of how vvvv works, I managed to butcher one of the examples so that Ive got an avi or mpeg being sent through a shader and rendered, which is nice.

So now Im thinking that it might pay to study how the .fx support works in this application in more detail. Has anybody else looked at this yet? I'll put another few hours in tonight and hopefully get a bit of detail and report back.

[nkm]

I agree with Pete Warden. In Pilgrim the filters only receive the layer dx surface (layer output-> filter input) and the dx surface to render to (filter output) and that's it. Couldn't be more simple.

However, what makes it difficult for a videocard based filter/ plugin is that it has to take the current state of the vga into account, like renderstates, timings etc. Hence, a plugin is never completely stand- alone, at least not as stand-alone as a Freeframe plugin i think.

nkm

[petewarden]

Yes, for instance the shader example doesn't disable the shader program at the end, so if the host tries to draw into the same context, it will end up using the shader, unless it explicitly calls glDisable( GL_FRAGMENT_PROGRAM_ARB ); first.

Management of state like that is tricky, I've ended up with a combination of resetting some commonly used state within the host after render calls, and requiring plugins to clean up after themselves by resetting any state they change.

[klumsy]

Ok I understand the going back through main memory is bad but as I understand it basic compositing operations like Blend / Addmix / screen are pretty easy to implement as pixelshader effects so wouldn't it make sense and be faster for all layer compositing operations to be done as part of GPUCHAIN on the card.

it would make more sense, but its not going to happen at first.. its going to ussually be some sort of software or freeframe to GPUCHAIN, to screen.. (with GPUCHAIN plugins also to do the mixing etc..) i know for some there will be a frustration factor when they have a particular freeframe effect they want to do AFTER a certian GPUCHAIN effect.. but slowly there will be more GPUCHAIN plugins made and availible, and it will get to the point that most CPU effects will become obsolete and replaced.


Don't a lot of newer VJ apps eg Vjamms OpenGL compositing, Modul8, Pilgraim do all layer mixing by rendering movies to billboard textures so they could easily do GPU effects just on one single layer texture?

i'd assume so, i've love to hear which ones do though.

as for the AGP bottleneck, for some users they might be willing to pay the read back to memory bottle neck ONCE for a smallish size texture at agpx8 or PCIexpress, but it is up to the host how they use it.. there is definately a penalty and speed cost.

[klumsy]

thanks pete for the heads up, the whole state management thing was something that i have been dreading.


The most important point is that the interface can be very minimal, and doesn't need to make assumptions about how the effects are implemented (shaders or not) or where they will be used in the rendering chain. As Dan says, a lot of host apps may need re-engineering to cope with more GPU stuff, but that shouldn't affect the plugin standard.

would you recommend a system like freeframe though that plugins take in parameters, and also have ability for multiple inputs or such, or would you keep it really to a small library of functions, and then have a more extensive (freeframe) like wrapper around that?


Yes, for instance the shader example doesn't disable the shader program at the end, so if the host tries to draw into the same context, it will end up using the shader, unless it explicitly calls glDisable( GL_FRAGMENT_PROGRAM_ARB ); first.

Management of state like that is tricky, I've ended up with a combination of resetting some commonly used state within the host after render calls, and requiring plugins to clean up after themselves by resetting any state they change.

this is where i see potential problems, its going to be neccisary for plugins to follow certian rules as far as state changing, however with released (but no source ) plugins, a plugin could introduce subtle bugs that might seem to be caused by a different plugin in the chain. its almost like all state changing calls should go through the helper library that can keep track of things, but then that would put too much work into the helper library, mean programmers have to do things diferently than they are used to , and would go out of date so quick.. i suppose we just have to assume the plugin does well, (just as we assume they don't look up a system anyway), and maybe a special 'testerhost' could be made that you can run a plugin through to see if it cleans up after itself..


They make some assumptions, including power-of-two input texture sizes,

this is the biggest thing that has been bothering me.. how do you deal with content thats height and width is not ^2.... i haven't worked with 3d cards for a few years, and i'm not up with the latest and greatest in 3d apis.. but lets say i was making a pixelshader for 640x480 that does a simple gradient... quite easy code..
however if i have to split it up into a various 'tiles' of various size.. and do a pixelshader on each of those, the precalculations would get a bit complex (and even more so for a more complex formula)..
i sure hope that you can do pixelshaders on bigger textures and such.

b.t.w thankyou EVERYBODY For all your input and enthusiam

[klumsy]

i just came across this opensource project that looks really interesting, i haven't given it a good look yet.

http://www.jahshaka.com/

[elbows]

Jahshaka isnt doing much for me at the moment, but I cant tell whether the shader stuff is supposed to work yet using the windows binaries, as some documentation suggests the shaders only work in linux at the mo. But the documents might be out of date.

So far from a practical "see some shaders, see how the variables are exposed to the vj app for realtime control etc" perspective, vvvv is proving most useful. Trouble is that I cant just randomly start copying & pasting examples of their shaders here, so I dont really know what to say about vvvv's way of doing things until some others have had a look.

Where Jahshaka seems to be wonderful so far is these 2 documents, which I think contain some very interesting & valuable thoughts on shaders, and that seem intensly relevant to this thread:

http://www.jahshaka.com/downloads/JahshakaGPGPU.pdf

http://www.jahshaka.com/downloads/JahshakaEffects.pdf

[dongbamage]

Nice.

btw, The AGP bi-directional bandwith restrictions have been sorted with the advent of PCI express...

[klumsy]

yeah PCIexpress is nice, but it is only double the bandwidth of agp8x. you couldn't afford to do a real back to system memory for every slot in the chain.. howver you could afford to do it once or twice (thus being able to combine open GL and directX plugins a little in the chain..
even with agp8x you can afford to do it alittle (just as visualjockey can afford to bring back a low resolution rendering of a 3d scene from pdooms 3d stuff.)
looks like i should check out this vvvv sometime.

[sleepytom]

yeh pci express should improve things a bit, but as ever we need to be looking to the future and designing systems to run at much higher res than we are dealing with today. - one of the main advantages of GPU processing is the possiblity of realtime HDTV res effects - when you start to shif that many pixels around you need really serious bandwith.

visualjockey is a good example - you can do ok by having a 3d node at the end of the chain - this video goes up>down>up the agp. But put in a multilayer node with a 3d node above each branch and it soon grinds the framerates as multiple layers have to go up>down>up (also bear in mind that by defalt VJo processes at 1/4 of your selected resolution)

[klumsy]

if my researching i came accross openMl
http://www.khronos.org/openml/images/openml_75px.png
http://www.khronos.org/openml/

it looks quite interesting and somebody said SGI at SIGGRAPH mentioned they are going to opensource their code..

anyhow i'm going to go a direction most people won't like for now.. i'm not even just going directX, but managed DirectX, in C#, thus really restricting the reuse.. THIS IS FOR A TEST APPLICATION.. i just want to develop a very good object orientated framework, in C# , to refine my concepts, while i exercise my skills (or now lack of them) in shaders... i will CONCURRENTLY design the architecture of GPUCHAIN on a wiki with all those who care, GPUCHAIN though i would like it to be a C++ standard will be just plain DLL (.SO) function exports.. so that non C++ languages can make plugins etc.. however i for directX version at least there will be a nice baseclass with alot of functinality for MSVC peps to decend from.

but GPU# is going to be a whole nother story..
hopefully some people will step up to the plate and help with the implementation of the GL GPUCHAIN.

[t2k]

Hey GPUCHAIN threadsters,

I had this idea almost 3 years ago and have been working on it ever since. The name of the software is "Salvation", it uses OpenGL, and is a fully modular development environment written in as-portable-as-possible C++. It is being developed on Windows but I have already had success porting portions of the renderer (not the editor) to Mac OS.

I am on vacation and don't have access to my dev machine. There are no screenshots online of constructions using shaders, but here is a picture of the "water" simulation module for now:

http://www.treyharrison.com/salvation/screenshots

Modules all reside in external .dlls, which means anybody with VisualC++ 6.0 or better can create their own. The inter-module API is also extremely flexible, to a degree which lets 3rd party developers define their own inter-module API, requiring no change to the core code base. Modules can be grouped inside of modules, every module UI can be customized by the user ("skinned"), demonstrations can be recorded and played back in realtime. In addition to the OpenGL modules there are basic vector math modules, midi note/controller modules, .tga, .avi, .ai (Adobe Illustrator), .md2 (Quake2) import modules... the list keeps growing, and will continue to grow as fast as I can manage =)

I have incorporated Cg shaders already and intend to incorporate GLSL shaders soon. You can even visually construct shader programs using modules that represent shader operations and results, and it will auto-compile your design into ARB_fragment or ARB_vertex programs.

The internal design has taken a long time to get right. Along the way I've spun-off a few other little things.. a Fruityloops plugin which allows you to sequence your Salvation visuals in sync with Fruityloops... already available on my website is a Windows screensaver which can run any design created with Salvation. A Winamp plugin has not been started but surely wouldn't take long.

My vacation is over in a few days, after which I intend to beta test for awhile and then begin selling the software. If you'd like to beta test, please contact me (info available on my website).

trey

[nkm]

I'm currently working on a flowdiagram GPU based filter framework for PilgrimR1.5. It's more like an experiment geared towards an open GPU framework structure like GPUChain. I'll post some regular updates and findings if people are interested.

nkm

[johnnylocust]

I'm certainly interested on helping out. The software I'm designing is based on VMR9 and directx though.
Butt Naked Broadcasting (http://buttnakedbroadcasting.blogspot.com)

But at some point in the near future I'm going to be looking at some sort of architecture to chain shaders for post and pre rendering.