calling all OpenGL VJ software developers

[t2k]

http://www.treyharrison.com/salvation/screenshots/shader_small.jpg http://www.treyharrison.com/salvation/screenshots/glsl_fpbuffer_small.jpg http://www.treyharrison.com/salvation/screenshots/watersim_small.jpg http://www.treyharrison.com/salvation/screenshots/MidiKnob_small.jpg http://www.treyharrison.com/salvation/screenshots/knob_small.jpg

I've been working on an OpenGL content creation tool called "Salvation" for several years. There are screenshots, movies, and info about the software here:

http://www.slvtn.com

The growing # of OpenGL VJ apps out there has inspired me to start working on a library that other software authors could use to import and manipulate content created with my program (similar to the way many VJ apps import movies and flash files).

Integrating accelerated OpenGL plugins into a VJ program is (as developers have seen) a lot more complicated than integrating video or video processing plugins such as FreeFrame. For GL content to display at the high resolutions and high framerates that all of these OpenGL VJ apps promise, plugins have to share GL resources with the host application as well as other GL plugins that may be loaded... resources like texture handles, pbuffer handles, display context handles, access to available GL extensions, access to current GL state, etc. All of these things have to be shared and accounted for carefully, otherwise a texture may show up in 1 window but not another, a vertex shader may work when rendering to the screen but not when rendering to a pbuffer, a texture created as TEXTURE_RECT_NV might get bound to TEXTURE_2D and not show up.. there's all kinds of crap that can go wrong.

After several years (heheh) I have solved all of these problems within my software, and I believe I have created a stable, flexible, and useful platform for accelerated GL content creation.

So, if you are writing an OpenGL VJ tool, I'd like to work with you and figure out how to get content created with my software running and mix-able within your application. The easiest way I can think of off the top of my head would be to share pbuffers between our code - ie, my content would render to a pbuffer managed by your application, your application would render to a pbuffer managed by mine, etc. Thats a good starting point I think, so if anyone out there is interested let me know =). I'd especially like to find out which of the already existing and popular VJ apps out there are using OpenGL, so that I can contact the authors of those apps directly with my proposal.

thanks,

trey

[VJamm]

Hi

VJamm 2.5 is OpenGL - we have been talking about this for a while now. We considered OpenGL when we started FreeFrame, but as you said it is SO complicated so we went for a simple framebuffer approach with FreeFrame 1.0, which was a good decision I think as it allowed everyone to join easily. I am trying to encourage people to do hardware OpenGL plugins in FreeFrame1.0 at the moment as the up n down the AGP is working so well now cf. games GL preprocessing.

I've always talked about a possible future FreeFrameGL project of actual hardware plugins.

I'll point our GL team to your post.

best wishes

Russell Blakeborough

[jaw]

great that some developers are around :) me and Trey are already having contact about this - me and my team are also working on a software pretty much like salvation (VSXu). The conclusion is that we would all benefit from a standardized module/freeframe-like standard for GL effects. One idea is that the freeframe team and the interrested parties could form some kind of standard group to flesh this out..

On what implementation level to put this standard is a different question though. In any case, having interoperability is a very good thing, then none of us have to focus as much on modules but more on providing a great GUI. :nod:

[Rovastar]

pbuffers are a good way forward now esp that many cards work with them (Ati have been a pain 6 month to a year ago.)

Are you thinking of standdard sizes for pbuffers (512x512; 1024x512, etc)?

I am thinking there wlll always be problems/issues when converting to actual output reses.

[t2k]

Mr. VJamm,

You said "I am trying to encourage people to do hardware OpenGL plugins in FreeFrame1.0 at the moment as the up n down the AGP is working so well now cf. games GL preprocessing."

I didn't understand this very well. Are you saying that the AGP bus speed is fast enough to upload lots of video footage?

RovaStar:

I was thinking that the host application would choose what size pbuffer to share with the plugin, then the plugin would have to render to that size. The plugin would never actually see the pbuffer handle, the host would deal with making the pbuffer the active context and calling a render method in the plugin. The plugin would know and could make use of the output resolution but it wouldn't be able to change it.

The host would take into account a variety of factors in choosing the size of the pbuffer for the plugin.. basing it on the final output resolution, the final aspect ratio, the plugin's preferred resolution and/or preferred aspect ratio, etc. The host could also have a hidden manual override option to set a specific resolution, if needed.

trey

[sleepytom]

what russell means is that he feels the AGP bus can handle multiple up down passes so people could write a freeframe plugin that renders the video to a null gl surface - do gl effects on that surface and then bring the video back down the agp buss to output into the host app

this is how the PDOOM 3d gl effects work in VJO for example - however its not an ideal solution and you will run into bandwith issues once you start to try and process multiple effects like this (depending on host architecture the agp buss can be very busy allready without this extra load)

[jaw]

please, no up/down on the AGP port, that is terrible design considering the current hardware architecture. doesn't matter if we have agp32x or something, OpenGL is not a bat or a magic box that hit with things (like used in visualjockey) when you want to generate effects,you must use it as intended. What t2k is suggesting is that other apps that generate visuals from be it movies or whatever have access to a pbuffer (or even better in special cases: a pure rendering pipe) for use in accelerated effects. Then if you still wanna invert the colors (or do some other freeframe-like effect) you can do so using fragment shaders. But like, first freeframe chain, then up to GL, then into trey's/our software [magic box doing openGL] or whatnot, then maybe fragment shader then out to screen.

[t2k]

sleepytom, its definitely possible for something like that to work, but its not something i'm particularly interested in - the purpose of my proposal is to work towards a standard that would allow external GL code to make use of and contribute to GL resources inside the host application.

Suppose the host app has loaded two of these plugins that you/russell are talking about. Both plugins take their input video frame, upload it to the graphics card, do some GL processing, download the frame and giving it back to the host. Suppose the user has connected these two plugins serially, so that the processing path is host->plugin1->plugin2->host. Between plugin 1 and 2, there would be 2 full frames of video (with the same contents) unnecessarily transferred across the bus (once from the GPU to the CPU at the end of plugin 1, then from the CPU back to the GPU at the beginning of plugin 2).

This "works" and can be "fast" if you spend enough money on your computer, but its a waste of resources. The obvious optimization here is for plugin 1 to render to a pbuffer, pass the pbuffer to plugin2 as a texture, then have plugin2 pull the result back down to the CPU. This is the kind of solution I'd like to work towards. In fact I imagine that plugin2 shouldn't have to download the frame to the cpu because the host could access plugin2's results by texturing from plugin2's pbuffer.

I don't mean to shoot down the existing VJO solution, in the end what matters with a VJ app is whether it can put cool moving pictures on a screen and I'm sure those VJO gl plugins are cool, even though they don't make optimal use of system resources.

[sleepytom]

i never said it was a good idea

a proper effects chaining system that is fully GL and takes place on the GPU is a much more desireable idea.

[djMidnight]

Originally posted by sleepytom
a proper effects chaining system that is fully GL and takes place on the GPU is a much more desireable idea.

This is exactly how Midivid GPU works, though it's all DirectX 9. I chose DirectX because it seems to be more compatible across various graphics cards. ATI & nVidia both support the various pixel shader sets in DX9, but can't settle on a GL extension set, so I'm told. (Have you found this to be true?)

Images are passed around as textures and all the effects happen through pixel and vertex shaders. It is certainly possible to pull them back down to main memory for the CPU to work with (my video sampler does this) but it's not ideal, and you can take a pretty good speed hit if you're not very careful. In contrast, I can run a bunch of complex pixel shaders in a row, using render-to-texture, and maintain 60fps at 640x480 on an nVidia 5200.

Jason Dorie

[t2k]

GL extensions are in a constant state of evolution, part fo the process requires that the hardware vendors and driver writers get together and "disagree" about what the next best step is for the extensions and the api in general. But any vendor can add their own extensions, and they all do (ATI,NVidia,3dlabs,etc).

GLSL shaders are working on the big 3 lines of cards, and ARB_fragment and ARB_vertex shaders are working on them too. PBuffers are implemented with different os-specific extensions on Windows, Mac, and Linux - thats disappointing, but its nothing that a smart programmer can't handle with with the right level of abstraction.

Extensions will have to play a role in the plugin standard we are working towards. Plugin authors can of course query for extensions themselves, but there are several libraries out there that will handle it for you. Within my own software I have a class that is used to keep track of what extensions are available, and that is passed around all over the place for whenever non-standard GL methods are needed.

[tho]

Originally posted by t2k
So, if you are writing an OpenGL VJ tool, I'd like to work with you and figure out how to get content created with my software running and mix-able within your application. The easiest way I can think of off the top of my head would be to share pbuffers between our code - ie, my content would render to a pbuffer managed by your application, your application would render to a pbuffer managed by mine, etc. Thats a good starting point I think, so if anyone out there is interested let me know =). I'd especially like to find out which of the already existing and popular VJ apps out there are using OpenGL, so that I can contact the authors of those apps directly with my proposal.
Hi, I am currently maintaining Smode Studio (www.galago.fr/smode). It is originally a demo tool and we are working hardly on it to make it more and more VJ friendly.
I have seen your app and it is impressive, especially the flexibility.
Smode Studio is based on openGL and most of the effects are done with it.
I agree with the pbuffer approach as a starting point for sharing output, the only difficulty should be to share the opengl context. Anyway, I send you a mail for you to contact me directly if you want.