Multiformat Listening Test @ 128 kbps - OPEN Options

[skelly831]

I have already placed my bets on who will win, but such comments are unwelcome until the test ends, so I'll keep them to myself

The above should be recommended to all who take the test, somebody always comes and ruins the uh... fun.

[jido]

Ok, so who wrote a *nix script to decompress all the samples automatically?

C'mon shouldn't be difficult...

[ronnie_t]

I will be hosting the files on razorback2

CODE

ed2k://|file|Sample01.zip|8509511|66679758AA8A2A35A1E84A689A1ECD53|h=LWZMHFG5CETFRHK6WLGEJPIXDQP4EBL6|/
ed2k://|file|Sample02.zip|7640515|AC5CDCBB42A2BD07EF814ABC32B48AFF|h=O2MEA225C5HB4I6JHAZXRUYDJTNM22CO|/
ed2k://|file|Sample03.zip|4628261|9F030183C15114A63E99AD1974B15BC9|h=SP3J4IZK7HPPYQUDI3DCYBJIGNTXC4XH|/
ed2k://|file|Sample04.zip|7034692|F793E41485B4B344306A6F3D3807529E|h=R3FA2SO5S725S2W2TIL7QOAQOVAVYR5K|/
ed2k://|file|Sample05.zip|8916801|86CA6334E2C822671BAA95E0F7DFFD8A|h=4AFNVZAADWZ6BMYY2GBWXKUP7KSRRJP2|/
ed2k://|file|Sample06.zip|3292659|82AA542444E3222E018EF0D1C65EF1D3|h=UMXYXWBJ57G45EFNTN3KA5QWSCFKJK6V|/
ed2k://|file|Sample07.zip|5947370|2C3A582CD5B0AB6BD8B0BDD21DDE2764|h=P7GV6IJG7TDUTN2OXE7BW2YJOZC4HDF5|/
ed2k://|file|Sample08.zip|7929888|BA0764358B80A8A67DDD00F385B15844|h=4E4BSA6X66LXAX5V7IQY7H2DS4TXCC3Z|/
ed2k://|file|Sample09.zip|8757039|D88B3A64A4C029FC250FA0A12C266906|h=JH2UTISCN5FDVXR4TKLTIP3IQZ2TBLIA|/
ed2k://|file|Sample10.zip|6620142|7F74993053571111DD9CFD38E9E31D98|h=355SD4TZTSYWZ6VIYGP5DNJKZXJERS4P|/
ed2k://|file|Sample11.zip|5358761|D0FAD27E576091EA0207B71BC47BD502|h=LGPPNNNCR7R2DQOQLGZTHJ44MVIB3XSW|/
ed2k://|file|Sample12.zip|9849316|DDBF5E2F93A0835DC197CFB4471957F0|h=PZDEHZT334ASEH7GBZSGSDB3RNHTK37D|/
ed2k://|file|Sample13.zip|8844385|A2F0D44C5ECCD1C00C92298FF902C7EA|h=ARSPD4C2B2YAKJG4HGEKSCNB33WMMRIM|/
ed2k://|file|Sample14.zip|6126723|6B18903D8DC559A9EE2523937A352D68|h=AVU5KZO7SHHKUHQRB3LNRE3LMRZXB23Z|/
ed2k://|file|Sample15.zip|5303587|8590704FDB32DC9074DBF0F49E3D8763|h=FY56ERKEKGQ36F5IF4YKLXC5Y3GCRG2C|/
ed2k://|file|Sample16.zip|4895823|40CA8C35902CA1033338C7F11A1881FB|h=DGOL3XMWKJ7SEHP6T6LNBORP4OLWIHPX|/
ed2k://|file|Sample17.zip|10289090|B662CD9548D880C6A9165C9E3C4B640E|h=6CE6UXJ5KBKFSCVTMM5OAE3IM5IMGB7P|/
ed2k://|file|Sample18.zip|9455856|A7727AAF3DC1AD2B384DC6056B63712E|h=ERZQI6KELOFJYHNUUCJCYUXD67GDVRUV|/

[ErikS]

Ok, so who wrote a *nix script to decompress all the samples automatically?

C'mon shouldn't be difficult...

I did for myself and sent a copy to Sebastian. Feel free to use... Place it in a file .../ABC-HR/bin_nix/SampleNN

It works for me so don't complain on my script skillz, m'kay.

CODE

#!/bin/bash

pushd .
cdir=`pwd`
bin=`dirname "$cdir/$0"`
cd $bin/..
nr=`echo $0 | sed 's/.*Sample\([0-9]*\)/\1/'`
oggdec=$bin/oggdec
faad=$bin/faad2
lame=$bin/lame
wvunpack=$bin/wvunpack

if [ -f "Sample${nr}.zip" ]; then
unzip Sample${nr}.zip
pushd .
cd Sample${nr}
for i in *
do
case $i in
*.mp3)
$lame --decode $i `basename $i mp3`wav
;;
*.mp4)
$faad $i
;;
*.m4a)
$faad $i
;;
*.ogg)
$oggdec $i
;;
*.wv)
$wvunpack $i
;;
esac
done
#remove compressed files
rm *.ogg *.wv *.mp3 *.mp4 *.m4a
popd
else
echo "Please download Sample${nr}.zip first."
fi

popd

then this in another file in the same directory:

CODE

#!/bin/bash

cdir=`pwd`
bin=`dirname "$cdir/$0"`

${bin}/Sample01
${bin}/Sample02
${bin}/Sample03
${bin}/Sample04
${bin}/Sample05
${bin}/Sample06
${bin}/Sample07
${bin}/Sample08
${bin}/Sample09
${bin}/Sample10
${bin}/Sample11
${bin}/Sample12
${bin}/Sample13
${bin}/Sample14
${bin}/Sample15
${bin}/Sample16
${bin}/Sample17
${bin}/Sample18

edit: suggested changes made...

This post has been edited by ErikS: Dec 7 2005, 01:43

[martythestickman]

hey guys.

just finished setting up all the files needed for the test including installation of java, and i have a problem. the wav's dont play. what am i doing wrong?
reference file doesnt play, ABX option playback doesnt work either. checked settings and my sound card is selected.....
need help if you want some results

thanks all

[Sebastian Mares]

...

Thanks, I'll add it to the ABC-HR_bin.zip when I am back home if you don't mind.

This post has been edited by Sebastian Mares: Dec 6 2005, 09:15

[caligae]

Ok, so who wrote a *nix script to decompress all the samples automatically?

C'mon shouldn't be difficult...

Wrote this script yesterday. This does no unpacking, error checking and so on. Simple go to directory SampleXX and run the script. This assumes all binaries are in $PATH. I think it's a bit easier extensible for future listening tests than ErikSs script so maybe we could merge them.

CODE

#!/bin/sh

for i in *
do
   case $i in
       *.mp3)
           lame --decode $i `basename $i mp3`wav
          ;;
       *.mp4)
           faad -o ` basename $i mp4`wav $i
          ;;
       *.m4a)
           faad -o `basename $i m4a`wav $i
          ;;
       *.ogg)
           oggdec $i
          ;;
       *.wv)
           wvunpack $i `basename $i wv`wav
          ;;
   esac
done

[Sebastian Mares]

Could someone announce the test on D9 and /. please?

[kurtnoise]

Could someone announce the test on D9 and /. please?

I did it for D9...

[kwanbis]

whats D9?

[Alex B]

http://www.doom9.org/
http://slashdot.org/

[kwanbis]

i was think in of digg.com ... we should digg it

[eisa01]

Ok, so who wrote a *nix script to decompress all the samples automatically?

C'mon shouldn't be difficult...

Wrote this script yesterday. This does no unpacking, error checking and so on. Simple go to directory SampleXX and run the script. This assumes all binaries are in $PATH. I think it's a bit easier extensible for future listening tests than ErikSs script so maybe we could merge them.

Thanks, I'm pretty new to commandline stuff, so I had no idea how to do this. But that didn't stop me from modifying this line

CODE

for i in Sample??/*

Now, you just place it in the directory containing the sample folders and execute it.

edit: Maybe not a very good idea, because all the files are in the top folder now

This post has been edited by eisa01: Dec 6 2005, 15:15

[naylor83]

...
Carbonelli has two samples that have the same effect as dr4 in http://ff123.net/training/training.html

I really can't find the artifacts you're talking about. Are they at one particular position in the samples?

[Atreus]

Digg it! Let's try to get this test on the front page...

http://digg.com/music/Multiformat_Listenin...128_kbps_-_OPEN

[Shade[ST]]

,Dec 5 2005, 10:27 PM]...
Carbonelli has two samples that have the same effect as dr4 in http://ff123.net/training/training.html

I really can't find the artifacts you're talking about. Are they at one particular position in the samples?

Near the beginning, IIRC. I had volume quite loud on my headphones (hd-280 pro) It's subtle, but recognizable.

[ep0ch]

Doesn't telling everyone where the artifacts are kinda defeat the point of a listening test?

[Sebastian Mares]

It is.

[IgorC]

VBR can produce different size. It will be normal if codec A will produce bigger size than codec B on one half of samples and codec B will produce bigger size on another half of samples.

However in this test. Itunes AAC always produces bigger size than Nero AAC.

1. +10%
2. +7.7%
3. +4.6%
4. +15%
5. +12%
6. +17.3%
7. +1.76%
8. +11.85%
9. +13.3 %
10. +9%
11. +8.3%
12. +12.6%
13. +10.37%
14. +15%
15. +9.3%
16. +8.57%
17. +9.1%
18. +1.74%

Neither one Nero AAC sample has a higher bitrate than Itunes one.
Probability that it happens only on this 18 samples is quite low.
It's not hard to assume iTunes will be better on this test due to quite big oversize.

[sTisTi]

Doesn't telling everyone where the artifacts are kinda defeat the point of a listening test?

The whole procedure of a listening test kind of defeats itself if you think like this. Training yourself to hear artifacts also shouldn't be done then, because you then might recognize new artifacts you missed before, which would change the notation of the codecs. I think the important thing is whether you can hear artifacts are not. Without knowing what to listen for and preferably also in which parts of the music, I am pretty sure practically everybody would have to rate all codecs as 5.0 in this test.

This post has been edited by sTisTi: Dec 6 2005, 20:38

[Sebastian Mares]

Doesn't telling everyone where the artifacts are kinda defeat the point of a listening test?

The whole procedure of a listening test kind of defeats itself if you think like this. Training yourself to hear artifacts also shouldn't be done then, because you then might recognize new artifacts you missed before, which would change the notation of the codecs. I think the important thing is whether you can hear artifacts are not. Without knowing what to listen for and preferably also in which parts of the music, I am pretty sure practically everybody would have to rate all codecs as 5.0 in this test.

Well, specifying the exact position and sample where an artifact is noticable is a bit unfair IMHO.

[Sebastian Mares]

VBR can produce different size. It will be normal if codec A will produce bigger size than codec B on one half of samples and codec B will produce bigger size on another half of samples.

However in this test. Itunes AAC always produces bigger size than Nero AAC.

1. +10%
2. +7.7%
3. +4.6%
4. +15%
5. +12%
6. +17.3%
7. +1.76%
8. +11.85%
9. +13.3 %
10. +9%
11. +8.3%
12. +12.6%
13. +10.37%
14. +15%
15. +9.3%
16. +8.57%
17. +9.1%
18. +1.74%

Neither one Nero AAC sample has a higher bitrate than Itunes  one. 
Probability that it happens only on this 18 samples is quite low.
It's not hard to assume iTunes will be better on this test  due to quite big oversize.

And that's why only a difference of 10% is tolerated. If one encoder is smart and assigns more bits - well - it's smart.
Also high bitrate doesn't automatically mean good quality.

[IgorC]

And that's why only a difference of 10% is tolerated. If one encoder is smart and assigns more bits - well - it's smart.

In the last Guru's test difference also was about 10% between result of Nero and Itunes.
So if Nero will changes their codec in the way to produce bigger size it will be a new revolutional codec, won't it?

This post has been edited by IgorC: Dec 6 2005, 20:53

[Alex B]

The sample bitrates are completely irrelevant and should be ignored. The VBR settings used with the other encoders besides Nero were chosen because they are likely to produce the same average bitrate with a library of complete tracks. The encoders are free to use any bitrate for the samples. Actually, we have no idea about the peak bitrates inside the samples. The only thing what matters is quality. Different encoders use different means.

The Nero developers who are aware of this criterion instructed the settings for this new untested Nero encoder.

EDIT

The older Nero 7 encoder produced the same average bitrate with the other encoders when the "Internet" VBR setting was tested by guruboolez and myself.

EDIT 2

I am going to add the new Nero version to my bitrate table soon.

This post has been edited by Alex B: Dec 6 2005, 21:09

[Sebastian Mares]

So if Nero will changes their codec in the way to produce  bigger size it will be a new revolutional codec, won't it?

As I said, bitrate is not the only factor when it comes to audio quality. Shine at 128 kbps CBR has the same bitrate as LAME at 128 kbps CBR. They both produce MP3s, but the resulting files sound totally different.