Software Shoot out Proposal

[wellREDman]

As a way of really being able to compare the looks of different softwares I'd like to run three or 5 pc/beamer/screen set ups in parralel , each running a nominated piece of software, but ALL using the SAME clips,

Ideally the machines would be the same spec, tho this will take some guesstimation for Mac/pc comparisons

I think it would be also cool for each soft to have a chosen "champion" to show it off to the best of its ability

Although its a competition of sorts, there wont be any judging, or winners or shit, its just a way of really seeing the difference in styles and feels that you get from different softies,

obviously itll be only open to clip playing softs


so what do ppl think?

and what softs do ppl want to see side by side?

and i spose what range of clips will give a decent workout ?

[sleepytom]

i've been thinking about this for a while and have devised methods to objectivly test midi triggering latency and consistancy.

i doubt people are intrested in such objetive tests though and any subjective test is flawed by its subjectivity (ie your going to be more judging the style of the diffrent operators than the softwares themselves)

i would like to see a setup where we have say 6 pieces of software in a row all with the same clips and an attendent (ideally from the company that makes the software) who can demonstrate the software to people as the move from one to the other - then people will be able to judge for themselves by asking the operaors to perform things that they are intrested in doing themselves.

[wellREDman]

yeah i like the idea of having ppl being able to have a play on the setups,

but also like the idea of having a masterclass performance by each soft

but the two arent mutually exclusive , we could have it running all day, for people to check out , then in the early evening a 6 soft simultaneous performance.

im really intersted in seeing the differences in the looks of the different softs, side by side

for example at Avit Leeds, both vdmx and resolume had quite distinctive feels to the way they keyed, so i could tell when someone was using them, this year, i didnt ever really tell which soft was being used just from the output

[alangeering]

Originally posted by sleepytom
i've been thinking about this for a while and have devised methods to objectivly test midi triggering latency and consistancy.


Trigger all 6 machines from one controller. Would be difficult to give accurate judging of triggering below 1/5 of a second (5 frames) as we'd have to allow 1 frame each way for the scans of each of the projectors/video outs, and the camera you'd record it all with would be running at 25 fps. so give or take a bit anything within 1/5 of a second would have to be called a dead heat.

Actually, as persistance of vision is less than this it might be able to watch the screen and judge it.

I do like the idea. For a fair test we would need equal spec machines, same installs of codecs and OS. Also, same scan converters would be nice. All projectors colour corrected to avoid problems there...

maybe this kind of detail is too much effort...

[sleepytom]

no, its much more involoved that that - and gives measurable results...

won't bore anyone with the detail here - last time i explained an objective test i got flamed for making everything too complex:confused:

[wellREDman]

I dunno if im into going into all that complexity,

it gets a bit too much like judging them on their technology,

im more after being able to see the subtle differeces, and maybe even the ways that different softs colour ppls styles etc.

[PilotX]

yeah, I'm with Red on this one.. I'm more interested in how different softs treat the same clips, although it's also highly dependant on how the person controlling the softs does things, I expect that if we had 5 different crews using one soft with the same clips we would see very different things..

but I'm less interested in which is technically 'better' in terms of latency, more interested in watching how different softs work differently with the same set of video..

[elbows]

Im interested in latency but I dont think it should be studied in realtime at avit. It sounds more like a project that should have the test results published on the internet.

[wellREDman]

but thinking about it, that kind of practical high tech benchmarking would be really best done whilst we have a bunch of identical systems set up

so if someone wants to write up the proposal for the latency etc. testing ill include that in my proposal :)

[sleepytom]

it doesn't need to be done on "identical" machines - its better done on just one machine with all the softwares on it

i might get round to doing this test at some point - if i do then i'll publish the results

i'm intrested in the measurable performance difrences between software as it is the one area where they are directly comparable in a non subjective fashion

i'm also keen to see software in use with the same set of clips and i'd be keen for these clips to span a range of diffrent styles from sampletastic archive.org clips though to super high res (1024x768) 3d renders

[juxta]

this seems like a weird to me,

clip production is often vj soft dependant i think

everything from codec to length to level to compositing and fx all depend on what you will be playing with live.

[sleepytom]

yes but a load of simular clips in diffrent software will give an idea of the strengths and weakness of the diffrent softs

it is very subjective though