IPB

Welcome Guest ( Log In | Register )

8 Pages V  « < 6 7 8  
Reply to this topicStart new topic
128kbps Extension Test - OPEN
[JAZ]
post Jul 29 2003, 20:21
Post #176





Group: Members
Posts: 1710
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



QUOTE (Gecko @ Jul 29 2003, 09:05 PM)
A question which hasn't been answered yet: what about WMAPro 2 pass? By using 2 pass you are forcing a vbr codec to a certain bitrate; tell it "to make the most out of 128k" which is unfair because you are artificially limiting the codec.

WMAPro in VBR variated too much, and a setting couldn't be choosen. That's why the test uses the ABR mode (2-pass).
Go to the top of the page
+Quote Post
ff123
post Jul 29 2003, 20:29
Post #177


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (Gecko @ Jul 29 2003, 11:05 AM)
A question which hasn't been answered yet: what about WMAPro 2 pass? By using 2 pass you are forcing a vbr codec to a certain bitrate; tell it "to make the most out of 128k" which is unfair because you are artificially limiting the codec.

WMA9Pro 2-pass VBR is a difficult case. The normal WMA9Pro VBR had wild bitrate swings from album to album, and we had to rule out using that in this comparison because the average bitrate across different albums just wasn't stable enough.

So we figured 2-pass VBR would be the next best thing. However, in the ideal world, the 2-pass would be calculated by encoding an entire album, or at the very least, an entire song, and then the sample would be sliced out afterwards.

Note that there is at least one critical difference between WMA9Pro 2-pass and limiting the bitrate on the other VBR codecs -- Microsoft actually provides a 2-pass mode, and the other codecs don't. So 2-pass is a mode that listeners might actually choose to use, whereas limiting the bitrates on the other VBR codecs would not be how they would normally be used.

ff123
Go to the top of the page
+Quote Post
rjamorim
post Jul 29 2003, 20:44
Post #178


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Just wanted to inform everyone here that the test pages have been moved to Paul's server for practicity reasons.

The new url is:
http://audio.ciara.us/test/

RareWares will stay here at HA, only the Listening Tests subsection will go.

Sorry for any inconvenience.

Regards;

Roberto.

This post has been edited by rjamorim: Jul 29 2003, 20:46


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
phong
post Jul 29 2003, 21:02
Post #179





Group: Members
Posts: 346
Joined: 7-July 03
From: 15 & Ryan
Member No.: 7619



QUOTE (Gecko)
So what does this tell you about the quality of the motor?

Not much, but we're shopping for a whole car, not just a motor. Who cares how efficient your motor is if you loose all that efficiency in a crappy transmission.

QUOTE (Gecko)
The vbr boxer reacts and pulls out a heavy shield that can deal with even the most forcefull blows, but the cbr boxer simply collapses. Again, what does this tell you?

It tells us that our glassjaw CBR boxer should find a new line of work before he gets himself killed.

In this analogy, if defense represents adaptability of bitrate, what does "ring generalship" stand for? Which codec should be accused of ear biting?


--------------------
I am *expanding!* It is so much *squishy* to *smell* you! *Campers* are the best! I have *anticipation* and then what? Better parties in *the middle* for sure.
http://www.phong.org/
Go to the top of the page
+Quote Post
fileman
post Jul 29 2003, 21:42
Post #180





Group: Members
Posts: 144
Joined: 14-July 02
Member No.: 2585



@rjamorim

Thank you for your great work with that test!

Regards, fileman.
Go to the top of the page
+Quote Post
puntloos
post Jul 31 2003, 14:02
Post #181





Group: Members
Posts: 149
Joined: 20-July 03
Member No.: 7881



I will just say one more thing before I wave my white flag and/or get banned for being too stubborn, whichever comes first wink.gif Please, spend some time on my little rant, one last time.

Another 'hypothetical situation' for you:
Suppose we have a test between 2 encoders, both VBR and only 2 samples.
    - Bachpsichord, total playtime say 10 seconds.
    - Hardrock, also 10 seconds playtime.
Now we run the encoders with 'quality settings such that they will average at 128kbit, or to put it differently, such that in the end the sum of all filesizes add up to exactly the same: 327680 bytes. Both encoders have used the same amount of bytes to 'play with'.

Now. The results: *drumroll*
Encoder A:
    - Bachpsichord, encoded at 192kbit average.
    - Hardrock, encoded at 64kbit average
Encoder B:
    - Bachpsichord, encoded at 128kbit average.
    - Hardrock, encoded at 128kbit average.


Now we run the listening tests, and suppose it turns out that overall, the Encoder A and Encoder B versions of both Bachpsichord and hardrock tracks get the same score. (they don't sound the same, they just sound equally 'pleasing' when you consider overall score). Or if you wish, more skewed: Say that Encoder A gets a slightly higher score than B on bachpsichord, but a slightly lower score than B on hardrock and the average score is the same.

And now my point is:

The result: "well, that's that, Encoder A and Encoder B are equal for 128kbit." is what would be the conclusion of this test like everybody supports here. And I agree! It is Fair.

But.

If you weigh the averages, you can say MORE. There is more information in the results! You can then say:
- Encoder A probably has an inferior coding method for bachpsichord, and needs more bits to reach its target quality.
- Encoder A probably has a superior coding method for hardrock and needed fewer bits to reach its target quality

And this is useful information, wouldn't you say? Isnt this what this test is all about? Finding out as much as we can from listening tests we do, trying to compare the various files? Wouldn't it be nice to be able to say "Well, for Classical music, mpc scored highest, while for hardrock ogg vorbis and AAC are the preferred choices? (just making something up here).

Plus, of course my example is highly stylized. With a lot more encoders, a lot more samples, filesizes and so on, it is not inconceivable that interesting extra results can come to the light. Again, I'm not saying this weighing scheme is rock solid, it, like the test itself, gives some extra data to think about. I really don't see why you would want to ignore this.

As a last comparison, if you are choosing which car to buy, and a test gives Car A and Car B *exactly* the same score and same price, then you don't just pick one blindfolded. No. You go and check what exact features both cars have that you consider important. One car will go faster while the other is safer. A USEFUL car test won't just say "Car A and B are equal", no, a useful test will say "Car A and B are equal, Car A is faster and more dangerous, car B is slower and safer". Pick what you prefer.
Go to the top of the page
+Quote Post
bawjaws
post Jul 31 2003, 14:20
Post #182





Group: Members
Posts: 173
Joined: 10-December 02
Member No.: 4043



QUOTE (phong @ Jul 28 2003, 09:20 PM)
This reminds me of a benchmark I recently heard about comparing a Pentium 4s to a G5.  They disabled hyperthreading on the P4 to make the test "fair".  That's like saying if one CPU had an extra fast cache, it should be disabled to make the test "fair".  Nonsense of course.

This is sort of off-topic (except that it shows how difficult benchmarking and testing can be) but:

They disabled hyperthreading on the P4 because the P4 got *worse* scores when it was turned on. What you think was done to make things 'fairer' for the G5 was actually done to benefit the P4.

Weird, huh?
Go to the top of the page
+Quote Post
AstralStorm
post Jul 31 2003, 14:42
Post #183





Group: Members
Posts: 745
Joined: 22-April 03
From: /dev/null
Member No.: 6130



QUOTE (puntloos @ Jul 31 2003, 03:02 PM)
- Encoder A probably has an inferior coding method for bachpsichord, and needs more bits to reach its target quality.
- Encoder A probably has a superior coding method for hardrock and needed fewer bits to reach its target quality

No, it might mean that encoder A has a superior psychoacoustic model, which detected, that first sample will be hard to encode and used less bits for easier second track. It is right if the same codec fails at bachpsichord using less bitrate.

Do you think that 64kbps MP3 is superior to 128kbps AAC, because the latter uses more bits? rolleyes.gif

Anyway, we're testing QUALITY, not QUALITY/SPACE tradeoff. Go figure.
In the second case, 48kbps MP3 might be better than 64kbps one,
because quality loss is substancially smaller than filesize loss.

This post has been edited by AstralStorm: Jul 31 2003, 14:49


--------------------
ruxvilti'a
Go to the top of the page
+Quote Post
guruboolez
post Jul 31 2003, 15:32
Post #184





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (puntloos @ Jul 31 2003, 02:02 PM)
Wouldn't it be nice to be able to say "Well, for Classical music, mpc scored highest, while for hardrock ogg vorbis and AAC are the preferred choices? (just making something up here).

Yes, conclusions are fine to me.
But you forget one thing. Roberto's test had a precise purpose : building an approximative hierarchy between many audio format, at ~130 kbps, and for general purpose. You can't conclude anything reliable and precise on musical genre, with only 12 samples. If you wan't to know which codec is the best for harpsichord, encode at least three different harpsichord sample (harpsichord didn't have a big dynamic range, but with recording conditions, manufacturing, and music content, results may change). For piano, you should take at least 6 samples : low/mid/high volume, noise or not, precise recording or more distant one... For orchestra, damn, it will be more difficult : strings, wind, full-orchestra, solo instruments, recording conditions, etc... are many variations of a same reality. And you to continue with string quartets, solo recorder, trumpets, wind ensemble, voice of course, organ... And I'm only talking here about "classical" genre.

In Roberto's test, there are two classical samples : macabre (orchestra) and bachpsichord. It doesn't make sense to conclude that mp4 is the must accurate encoder for piano and voice, just because mp4 was the best for macabre.wav. There's more sense to conclude than sample X, winner of the bachpsichord test, should be a privileged choice for solo harpsichord discs (nice : it's <0,01% of the whole discs of this planet), simply because another harpsichord recording should differ too much. For orchestra, I wouldn't conclude anything on macabre.wav results. Of course, blade failed horribly, and had 99,99% chance to be a bad choice on other orchestra parts. But the winner on these 20 seconds, how many chances to keep the first place on a long clarinet solo ? On a tutti ? A string introduction ?

Roberto's test conclusion will give us less answers than lead. If an encoder will fail on electronic sample (#5), and be the best on metal (#10), we can't conclude that this encoder is good for metal and useless for electro. We just have to investigate further to be sure, by testing other similar but different samples. It's just a preliminar hypothesis, that need to be proved by user experience, or by isolated tests.


(Note : for harpsichord, my choice is made. I know for a good time what encoder is the best for my ears. I'm just waiting the end of the test before any comments).
Go to the top of the page
+Quote Post
bond
post Aug 3 2003, 15:52
Post #185





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



rjamorim,
i am still listening to some samples, so dont hurry to close the test wink.gif


btw. does anybody know from where the macabre clip was taken (i mean from which composer, etc...)


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
rjamorim
post Aug 3 2003, 16:05
Post #186


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Hehe. Well, the test will be closed tonight at midnight brazilian time (4AM of Monday GMT)

So, please send in your results until then wink.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
rjamorim
post Aug 4 2003, 02:00
Post #187


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



TEST CLOSED!

No more mails accepted from this moment on.

Expect results in a few hours.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
rjamorim
post Sep 10 2003, 04:18
Post #188


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (AstralStorm @ Jul 24 2003, 05:57 PM)
Here is the sample batch - you need to make an executable from it!

And how do I do that?

I tried a tool called bat2exe, it didn't work (batch execution stops at the middle)
http://nlsn.free.fr/batch-down/Bat2Exe.ZIP
That tool actually creates .com files. tongue.gif

Can somebody help me out with a good bat to exe tool?


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
rjamorim
post Sep 10 2003, 06:25
Post #189


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Just tried this one:
http://www.bdargo.com/info/batcomp.htm

Compiles the batch files well and these .exe from .bat work perfectly here at home. But they generated GPFs on ff123's PC, and output some very weird errors (could not find C:\bdtmp) sad.gif

This post has been edited by rjamorim: Sep 10 2003, 06:27


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post

8 Pages V  « < 6 7 8
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st April 2014 - 13:28