Video Decoding Questions for Software Developers

[Esotic]

Just cuz I'm curious about this...

Isn't MPEG1/2 decoding built into most graphic chipsets? If so, wouldn't MPEG be the video format of choice for speed reasons?

Can any of the software developers out there comment on this? Does your program make use of hardware acceleration or is there some reason that it can't?

Enquiring minds want to know,

-Esotic

[Esotic]

Hmmm... here's some DirectX info on the MSDN.

http://msdn.microsoft.com/archive/default.asp?url=/archive/en-us/directx9_c/directx/htm/aboutdirectxvideoacceleration.asp

[Toolmaker]

Originally posted by Esotic
Just cuz I'm curious about this...

Isn't MPEG1/2 decoding built into most graphic chipsets? If so, wouldn't MPEG be the video format of choice for speed reasons?

Can any of the software developers out there comment on this? Does your program make use of hardware acceleration or is there some reason that it can't?

Enquiring mind want to know,

-Esotic I think I know what you mean. And if so, the answer is yes (with a few twists).

You are thinking that any system that pumps content in kernel mode directly from the HD to the video card will have lower latency than one that thunks to user mode and uses a software based codec.

Right? :idea:


Of course if you want to scrub, or play it backwards, then every frame must be a key frame. No problem. Write your own off-line compressor. Don?t abandon MPEG 2.

Right? :idea:

There is another video standard built into most video cards called SD-DVCR (aka SMPTE 314M / IEC 61834) that IMO, is also a very good choice, because every frame is always a key frame. Some folks will stumble over the interlace issue, but again, the solution is to write your own off-line compressor (to guarantee 100% correlation between fields).

I would love to connect with any other software developers who understand what I?m talking about. I?ve already done a great deal of work in this area.

The trouble I have is this: I?m a 44 year-old adult, and I can?t afford to work for free. :(

[Esotic]

I'm just curious as to whether or not we as the VJ developer community are missing some hardware acceleration options.

I do all my VJ programming as a hobby, so I'm all about kickin' it open source. :)

And I love garlic cheese bread,

-Esotic

[Toolmaker]

Originally posted by Esotic
I'm just curious as to whether or not we as the VJ developer community are missing some hardware acceleration options.

I do all my VJ programming as a hobby, so I'm all about kickin' it open source. :)

And I love garlic cheese bread,

-Esotic I wish my landlord loved garlic cheese bread.

As long as VJ apps continue to use Quicktime or Jitter as their rendering engine, they will be missing some hardware acceleration options.

The next big leap (at least in the Microsoft world) is to begin writing virtual kernel-streaming device drivers and wire them directly to the graphics card with DirectShow.

[VJamm]

mmm - well on windows most people are using DirectShow MPEG decompression, and without specialist hardware, I'm not convinced there's any hardware MPEG decompression to be had there, certainly not if you want to do anything with the frames.

The DirectShow architecture does allow for stuff to be dropped onto hardware when it's installed in your windows - I think the guys up at Camart in Cambridge have some hardware decompression / compression stuff going on in the version of the VJamm ScratchPad project they have running on a Perception video editing system there - but practically speaking I don't think we're missing much - stop me if I'm wrong - as regards normal VJsoftware running on laptops / shuttles etc.

Russell

[elbows]

I didnt think there were many VJ apps that used Quicktime or Jitter as rendering engine on the PC?

[elbows]

Originally posted by VJamm
mmm - well on windows most people are using DirectShow MPEG decompression, and without specialist hardware, I'm not convinced there's any hardware MPEG decompression to be had there, certainly not if you want to do anything with the frames.

The DirectShow architecture does allow for stuff to be dropped onto hardware when it's installed in your windows - I think the guys up at Camart in Cambridge have some hardware decompression / compression stuff going on in the version of the VJamm ScratchPad project they have running on a Perception video editing system there - but practically speaking I don't think we're missing much - stop me if I'm wrong - as regards normal VJsoftware running on laptops / shuttles etc.

Russell

Oops my last post was written before yours appeared, sorry about that.

Anyway I broadly agree with you there, and as CPUs etc have got faster I dont see video decompression as a major use of my system resources anymore, its become less of an issue as the years fly by.

Still having said that, Id like to know if there is anything real to be gained in practical terms from latest nvidia graphics spec waffling on about decoding and encoding features, or whether its mostly useless hype.

[sleepytom]

i guess its the old up down up thing again...

if there is hardware decompression on GPU then thats all good and well but to get the frames back to do any software processing would mean getting all the decompressed frames back down the agp before sending them up again for output.

i guess nvidia would like us to avoid any cpu effects processing and do it all up on the graphics card so the file decompression would take place before gpu / pixelshader effects and youwould just send it up the agp once.

but nvidia don't have any intrest in older hardware where we might like to be able to support graphics chipsets that are a few years old (like most laptops have)

the current developments in nvidia land seem to be geared towards low CPU use HDTV playback

[Toolmaker]

Originally posted by VJamm
...certainly not if you want to do anything with the frames.Hi Russell.

I guess it depends on what you want to do.

Look at the Cube sample in the DX9 SDK.

What?s going on there?

Who?s doing the thinking?

Professor ATI, or Professor Intel?

[Toolmaker]

Originally posted by VJamm
... certainly not if you want to do anything with the frames.What is HLSL all about? :rolleyes:

When ATI says they support DirectX 9 hardware acceleration (Pixel Shader 2.0), what are they talking about? :rolleyes:

[VJamm]

... as far as i know in the cube the DS decompression is software, the spinning of the cube and 3D texture mapping of the video is hardware. I'm more familiar with an OpenGL sample DS texture mapping sample in Delphi though ...

Russell

[klumsy]

streaming video through directshow onto textures in the way to go,
and HLSL etc and pixelshaders can be used to pretty much replace all our 2d 'software' effects, and they run soooo fast.. (actually if the majority of vj software was inside the 3d pipeline it would allow alot of CPU for decompressing video in memory :) and whatever else

the problem with directShow is that you can't easily throttle it (i.e say don't give me frames unless i want one.) as easy filter in the filtergraph has its own threads and all and will just keep on pumping out frames according to itself.

[Toolmaker]

Originally posted by klumsy

the problem with directShow is that you can't easily throttle it (i.e say don't give me frames unless i want one.) as easy filter in the filtergraph has its own threads and all and will just keep on pumping out frames according to itself. That?s not a problem for me because my virtual video capture device (my kernel driver) does all the ?throttling.? VMR-9 doesn?t get a new frame until I say so.

:whip:

[klumsy]

thats cool. how does that work? just with a simple filtergraph , or does it still work with complex filtergraphs?

b.t.w that is a sick smiley
:moon:
h

[VJamm]

We do a similar thing in VJamm, except that we take video from DS / VFW and stream it to OpenGL textures.

The problem with Direct3D and the VMR is that they lock you into micro$oft black boxed stuff. That's why we opted for software effects in the first instance with FreeFrame, then they can work for everyone. Direct3D pixel shader effects are gonna be pretty app specific - not much chance of building up a good open system to share effects there it seems to me.

Hardware is good though, we haven't looked back since building the VJamm OpenGL compositor - it can handle 16 layers of 8bit blended keying in realtime at massive framesizes with no problem - and lots of mad 3D effects - unimaginable in software.

Russell

[Nema]

ou can query filtergraph for several interfaces:

IID_MediaEvent
IID_MediaSeeking
IID_MediaControl

with them, you get all control you need. however, it is not easy to get used to directshow (especially if you are not used to COM objects).

if you use filtergraph and lead your visual streams into the VMR9 filter, your videos will automatically take advantage of hardware-accelerated decoding (in dependency of the filters you have installed). the difficulty is then in grabbing the stream-output in a best way from your vj-app or vj-app-plugin.

the first thing to start if you are new to directshow is the nice little tool "graph edit" within the directx sdk.

[Nema]

ah, and i forgot to mention: the predestinated combination with DirectShow is Direct3D but not OpenGL. if you seriously think about divin ginto all the benefits of DirectSHow you better concentrate on Direct3d rather than OpenGl too.

[DrEskaton]

Originally posted by VJamm

The problem with Direct3D and the VMR is that they lock you into micro$oft black boxed stuff. That's why we opted for software effects in the first instance with FreeFrame, then they can work for everyone. Direct3D pixel shader effects are gonna be pretty app specific - not much chance of building up a good open system to share effects there it seems to me.
Russell

I know it's Apple specific but isn't the new CoreVideo stuff in Mac OS X 10.4 supposed to be a fix for this? i.e a standard for pixelshader filters that can be shared between all Mac VJ apps (or editing or visual effects apps or whatever). They seem to have a lot of success convincing developers to adapt Audio Units as a cross application standard for audio filters so I'm assuming the same will happen with video after 10.4 Tiger comes out.

Does Microsoft not have an equivalent way of defining a pixelshader filter in such a way it can be reused between programs in DX9? Genuinely curious not meaning to start a platform war, I just want to know what the equivalent MS technology is.

[Toolmaker]

Originally posted by DrEskaton

Does Microsoft not have an equivalent way of defining a pixelshader filter in such a way it can be reused between programs in DX9?In Longhorn, Avalon ?looks like? a D3D9 app running in full-screen mode, and will use pixelshaders if the card has them.

Read this:

download.microsoft.com/download/1/8/f/18f8cee2-0b64-41f2-893d-a6f2295b40c8/TW04006_WINHEC2004.ppt (download.microsoft.com/download/1/8/f/18f8cee2-0b64-41f2-893d-a6f2295b40c8/TW04006_WINHEC2004.ppt)

Do you understand that ?cooperatively sharing pixelshaders with other D3D apps and the Desktop? is a bad thing for games and VJ apps because frame rates take a hit? :nono: :confused: :scared:

[klumsy]

Do you understand that ?cooperatively sharing pixelshaders with other D3D apps and the Desktop? is a bad thing for games and VJ apps because frame rates take a hit?
hey microsoft is constantly researching new ways to slow down their operating system so you as a consumer keep in the cycle of buying new hardware because your computer runs to slow, and the new computer comes with a new OS, etc etc.

[Toolmaker]

Originally posted by klumsy
hey microsoft is constantly researching new ways to slow down their operating system so you as a consumer keep in the cycle of buying new hardware because your computer runs to slow, and the new computer comes with a new OS, etc etc. But the point I made holds true for the Mac too.

[DrEskaton]

Originally posted by Toolmaker


Do you understand that ?cooperatively sharing pixelshaders with other D3D apps and the Desktop? is a bad thing for games and VJ apps because frame rates take a hit? :nono: :confused: :scared:

ok, please explain why this is so, do you mean that because the base OS is constantly using the GFX card that an app can't use it's full performance?

I believe that would depend on the way it's done. Mac OS X has had 3D accelerated windowing since 10.2 and it doesn't seem to affect 3D performance, your app can just switch to a full screen mode and all the windowing stuff is suspended and doesn't draw any resources.

Even in windowed mode, unless you are dragging windows around while attempting to VJ it shouldn't have an affect?

Under a modern multitasking OS a sleeping thread shouldn't drain any CPU or GPU resources. or am I missing something?

The thing is even if there is a slight hit, wouldn't the benefit of lots of developers being able to share filters be worth it. If your argument was true we would all be writing VJ apps in DOS to have the fastest possible hardware access!

still curious you don't seem to say if MS has a way of sharing pixelshader filter implementations or not?

[klumsy]

sure there is, there is MS effect file, plus High Level Shader Language is compiled applied at run time, so different apps can just share HLSL files..
however there isn't neccisarily a point to it, because different people use shaders for different things, 'shaders' is some magical word it seems that makes people go wow.. just one feature of a 3d card, used various ways, normally in 3d effects.. i'm not very familiar with macs, but is this 'shared pixelshader library or whatever' general purpose use (i.e 3d games would use it, 3d modelling programs etc).. or it is rather 'pixel shaders in the context of video streams' like a directShow filter (but using pixelshaders'.. such a library in this limited scope (where basically there are inputs and outputs like freeframe, and each 'plugin' took those inputs and outputs and did processing.. basically 2d textures, just using pixelshaders to do the '2d' processing..
such a framework would be quite nice IMO, but its ontop and beyond 'pixelshaders' and actually a limited scenario of their use.

anybody can point me to more info on this CoreVideo.. it would seem useful, maybe we could make a standard like freeframe for this sort of thing..
problem is trying to do it all in pixelshaders itself is good for certian effects, but is still limiting..
alot of effects would best in combination of 3d geometry, vertex shaders AND pixel shaders.

[DrEskaton]

Originally posted by klumsy
i'm not very familiar with macs, but is this 'shared pixelshader library or whatever' general purpose use (i.e 3d games would use it, 3d modelling programs etc).. or it is rather 'pixel shaders in the context of video streams' like a directShow filter (but using pixelshaders'.. such a library in this limited scope (where basically there are inputs and outputs like freeframe, and each 'plugin' took those inputs and outputs and did processing.. basically 2d textures, just using pixelshaders to do the '2d' processing..
such a framework would be quite nice IMO, but its ontop and beyond 'pixelshaders' and actually a limited scenario of their use.

anybody can point me to more info on this CoreVideo.. it would seem useful, maybe we could make a standard like freeframe for this sort of thing..


There's a fluff PR page here:
http://www.apple.com/macosx/tiger/core.html

All the good stuff is still covered by NDA unfortunately.

The gist of it is the next version of OS X out in Jan next year will include a bunch of GPU accelerated 2D image filters that any program can use and define a standard for developers to make their own reusable "Image Units" much like Audio Units have become a standard for audio filters in the mac world. The idea is for all Image processing applications, whether for print, video editing or live performance to share these Image Units plugins.

If you become a member of developer.apple.com you can access more info depending on your membership level.

[klumsy]

ok, so now i really get what different people are saying, and how the term pixelshaders have been used and abused.
really its a video stream achirecture that happens to use pixelshaders internally for the actually transformation of data.

[Nema]

what is described at this mac-page as "Image Unit" is nothing else than a mac-version of a MS "directshow filter". such filters can use advantages of pixelshaders, but it is optional. such a filter is one component in a whole chain of filters if you process a stream. you can for example place an "bump-filter" just behind the "mpeg decoder" filter, and the result of your stream (the final picture of your stream which will be passed to your app) will then get bumped automatically.

however, for a VJ app it makes no big sense doing everything with async streams, you better code a bundle of pixelshader filters yourself, and together with a primitive 3d engine you can then map them on everything (not only onto streams from live-video and media files, but also on any cool generic effect, over a 3d-rendered output...)

[klumsy]

woah, i never knew that you could make shader directShowfilter that don't bring the result back into main memory,
that is cool.. however dshows async architecture is not cool
i wonder if coreVideo is async or syncronous or both?

[Nema]

i don't know eighter, but i guess it is async too. if you have a mediaclip encoded with 25fps, it is better to process it async to your main-app which runs with (maybe) 100fps.

[klumsy]

thats true, the prob i have is with things processing when its not up to our framerate, i'd personally have something SYNC, with ability to have it bypass (i.e if its teh same frame of the 25 fps