Listening test at ct

Topic: Listening test at ct (Read 9103 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

Listening test at ct

2002-08-22 19:15:54

URL:

http://www.heise.de/newsticker/data/vza-22.08.02-000/

Files I got (uncompressed, MD5):

8efccf81e782a3b6cbd4045cb2a1ff52  6140.wav

573fd2d049eea544d92071c10ba74581  6141.wav

def5908bba7a2fe10f734d45f2a07914  6142.wav

b2247fd675effb9c7b0ccc6da0f0f529  6143.wav

36ef5eb879307cbccda08eba52b75255  6144.wav

a4893be56ee50c9856514e99503dd098  6145.wav

fc62e90eca5b00d8f5d98d76842d9d60  6146.wav

448fc4495c4949703b274a058bc8ba23  6147.wav

fc62e90eca5b00d8f5d98d76842d9d60  6148.wav

ba0afbf7df12029cd5470f3afc596178  6149.wav

38e901bd629dbc748eb14b98d112e791  6150.wav

fcd9c4b8f3f02c3bd481e7dea0d734ae  6151.wav

e0e35e134bccb8dcdd79be264f332382  6152.wav

7fc3f2747220f4cb4ceb04f24ad516a5  6153.wav

07b9d1a50a55f7300c71269cc2918026  referenz.wav

It's only possible to sort the files in the order of quality, no further quality accessment is possible.
AAC was used with 44.1 kHz also at 64 kbps.

I hope the result of the test will not be that 64 kbps = CD quality. 128 kbps MP3 are ugly enough.

Listening test at ct

Reply #1 – 2002-08-22 19:34:08

I second that motion!

I think we have heard enough of the BS that 64kbps or 128kbps MP3 is CD quality.

Cheers
AgentMil

Listening test at ct

Reply #2 – 2002-08-22 20:14:47

So it looks like there are two sets of 7 files, one set at 64 kbit/s and the other set at 128 kbit/s (approximate bitrates).

There is one musical sample being compared? Is it a medley of different music styles, or is it one style? Do different people get different samples of music?

ff123

Listening test at ct

Reply #3 – 2002-08-22 20:47:45

Apparently there are fragments of three songs: a pop song, a jazz recording and a crescendo from a classical concert.

BTW, the Heise forums have turned into something similar to Slashdot; many ignorant people commenting on this test...

Listening test at ct

Reply #4 – 2002-08-22 21:11:08

Quote

Originally posted by CiTay
Apparently there are fragments of three songs: a pop song, a jazz recording and a crescendo from a classical concert.

BTW, the Heise forums have turned into something similar to Slashdot; many ignorant people commenting on this test...

The test has really many flaws.

The test sample consists of theree parts with remarkable loudness differences between the titles.
May be someone can load it into CEP and show the spectral shape/envelope.

See also:
http://www.heise.de/ct/leserbefragung/hoer...830ce5b0612b12b

1. Kylie Minogue: "Love At First Sight", Track 1 von der MCD "Kylie Minogue: Love At First Sight", Katalog-Nr. 7243 550986 0 3. © 2002 EMI Records Ltd.

2. Jazz feat. Emma Lanford: "Horny", Track 8 von der CD "Mousse T: Gourmet De Funk", Katalog-Nr. PJCD0004. © 2002 Peppermint Jam Records.

3. Giuseppe Verdi: Extrakt aus der CD "Orchestre et Chœurs de l' Opéra Bastille: Otello", Dirigient Myung-Whun Chung, Katalog-Nr. 439 805-2. © 1997 Deutsche Grammophon GmbH.

Listening test at ct

Reply #5 – 2002-08-22 21:29:37

Quote

Originally posted by CiTay
BTW, the Heise forums have turned into something similar to Slashdot; many ignorant people commenting on this test...

Well, I hope somebody doesn't point back to r3mix to say how a "real" test should be performed. That was just about too much for me to take! At least this test makes people use their ears.

ff123

Listening test at ct

Reply #6 – 2002-08-22 21:30:30

Original: http://www.uni-jena.de/~pfk/mpp/audio4/referenz.pac

Musepack 1.1: http://www.uni-jena.de/~pfk/mpp/audio4/referenz-1.1.mpc
Decoded: http://www.uni-jena.de/~pfk/mpp/audio4/referenz-1.1.pac

Musepack 1.9: http://www.uni-jena.de/~pfk/mpp/audio4/referenz-1.9.mpc
Decoded: http://www.uni-jena.de/~pfk/mpp/audio4/referenz-1.9.pac

Version 1.1 uses --quality 4.65, version 1.9 --quality 4.75

--
Frank Klemm

Listening test at ct

Reply #7 – 2002-08-23 06:51:51

I tried both the 64 and 128 tests. I think a clear winner and a clear loser will emerge from the 64 kbit/s test. However, I don't think that enough music was tested to be a fair comparison of the relative quality of these codecs.

On the 128 test, I could only tell the difference on two of the seven files. I used abchr.exe to present the files, so I heard that the time offsets were not the same for at least one file. Using a difference in time alignment is of course not a valid way to distinguish an encoded file from the original.

The cT web page forced me to rank all of the files, even though I could not tell the difference for 5 of them. I think that this test (at 128 kbit/s) will have too much listener "noise" in it to generate useful results.

ff123

Listening test at ct

Reply #8 – 2002-08-23 07:53:21

heise.de forums are worse than slashdot, period!
I just read through the discussions about the listening test and hardly anone has a clue about psychoaccoustic audiocompression.

The test sample is a real disaster and in fact not usable for a listening test, but I'll try it anyway.

dev0

Listening test at ct

Reply #9 – 2002-08-23 08:36:56

Quote

Originally posted by ff123
I tried both the 64 and 128 tests. I think a clear winner and a clear loser will emerge from the 64 kbit/s test. However, I don't think that enough music was tested to be a fair comparison of the relative quality of these codecs.

On the 128 test, I could only tell the difference on two of the seven files. I used abchr.exe to present the files, so I heard that the time offsets were not the same for at least one file. Using a difference in time alignment is of course not a valid way to distinguish an encoded file from the original.

The cT web page forced me to rank all of the files, even though I could not tell the difference for 5 of them. I think that this test (at 128 kbit/s) will have too much listener "noise" in it to generate useful results.

ff123

Be attentively. Every test contains 6 lossy encoded files and a lossless (but time shifted) sample.

Start analysis of my abcx:

Code: [Select]

Reading referenz.wav ...

Removing DC (left=8, right=8)

Delay Ch1 is   +0.0000 samples

Delay Ch2 is   +0.0000 samples



Reading 6140.wav ...

Removing DC (left=10, right=9)

Delay Ch1 is +3294.9926 samples

Delay Ch2 is +3295.0105 samples



Reading 6141.wav ...

Removing DC (left=5, right=5)

Delay Ch1 is  -68.0108 samples

Delay Ch2 is  -67.9864 samples



Reading 6142.wav ...

Removing DC (left=2, right=1)

Delay Ch1 is +632.9907 samples

Delay Ch2 is +633.0059 samples



Reading 6143.wav ...

Removing DC (left=15, right=10)

Delay Ch1 is -196.0176 samples

Delay Ch2 is -195.9857 samples



Reading 6144.wav ...

Removing DC (left=4, right=3)

Delay Ch1 is  -56.3828 samples

Delay Ch2 is  -56.3529 samples



Reading 6145.wav ...

Removing DC (left=-6, right=-3)

Delay Ch1 is -194.8469 samples

Delay Ch2 is -194.8217 samples



Reading 6146.wav ...

Removing DC (left=8, right=8)

Delay Ch1 is -196.0000 samples

Delay Ch2 is -196.0000 samples



Reading 6147.wav ...

Removing DC (left=8, right=8)

Delay Ch1 is -1507.0081 samples

Delay Ch2 is -1506.9901 samples



Reading 6148.wav ...

Removing DC (left=8, right=8)

Delay Ch1 is -196.0000 samples

Delay Ch2 is -196.0000 samples



Reading 6149.wav ...

Removing DC (left=8, right=9)

Delay Ch1 is -196.9993 samples

Delay Ch2 is -197.0012 samples



Reading 6150.wav ...

Removing DC (left=6, right=5)

Delay Ch1 is  +46.9878 samples

Delay Ch2 is  +47.0113 samples



Reading 6151.wav ...

Removing DC (left=10, right=13)

Delay Ch1 is -196.0112 samples

Delay Ch2 is -195.9903 samples



Reading 6152.wav ...

Removing DC (left=8, right=8)

Delay Ch1 is +145.9976 samples

Delay Ch2 is +146.0029 samples



Reading 6153.wav ...

Removing DC (left=8, right=8)

Delay Ch1 is +1799.3871 samples

Delay Ch2 is +1799.4048 samples





Delaying A by +1507 samples

Delaying B by +4802 samples

Delaying C by +1439 samples

Delaying D by +2140 samples

Delaying E by +1311 samples

Delaying F by +1451 samples

Delaying G by +1312 samples

Delaying H by +1311 samples

Delaying I by    +0 samples

Delaying J by +1311 samples

Delaying K by +1310 samples

Delaying L by +1554 samples

Delaying M by +1311 samples

Delaying N by +1653 samples

Delaying O by +3306 samples



r=1.000000  sx=6528.991786  sy=6528.991786  x=-1.282357  y=-1.282357  b=1.000000

r=0.978621  sx=6528.991786  sy=6465.721047  x=-1.282357  y=-0.959502  b=0.990309

r=0.987094  sx=6528.991786  sy=6589.209510  x=-1.282357  y=0.020073  b=1.009223

r=0.981366  sx=6528.991786  sy=6312.741815  x=-1.282357  y=-1.077922  b=0.966879

r=0.956254  sx=6528.991786  sy=6642.194215  x=-1.282357  y=-1.608282  b=1.017338

r=0.969342  sx=6528.991786  sy=6370.966923  x=-1.282357  y=-0.675616  b=0.975796

r=0.973751  sx=6528.991786  sy=6269.816132  x=-1.282357  y=-1.011746  b=0.960304

r=1.000000  sx=6528.991786  sy=6528.991786  x=-1.282357  y=-1.282357  b=1.000000        Files are id

entically

r=0.996787  sx=6528.991786  sy=6455.728107  x=-1.282357  y=-0.348965  b=0.988779

r=1.000000  sx=6528.991786  sy=6528.991786  x=-1.282357  y=-1.282357  b=1.000000        Files are id

entically

r=0.988670  sx=6528.991786  sy=6473.581921  x=-1.282357  y=-1.436466  b=0.991513

r=0.992270  sx=6528.991786  sy=6562.837880  x=-1.282357  y=-0.090919  b=1.005184

r=0.992311  sx=6528.991786  sy=6581.994369  x=-1.282357  y=-1.368739  b=1.008118

r=0.989275  sx=6528.991786  sy=6490.267108  x=-1.282357  y=-0.966215  b=0.994069

r=0.990399  sx=6528.991786  sy=6332.658015  x=-1.282357  y=-1.023793  b=0.969929



Multiplying A by 0.82655

Multiplying B by 0.83464

Multiplying C by 0.81900

Multiplying D by 0.85486

Multiplying E by 0.81246

Multiplying F by 0.84705

Multiplying G by 0.86072

Multiplying H by 0.82655

Multiplying I by 0.83593

Multiplying J by 0.82655

Multiplying K by 0.83363

Multiplying L by 0.82229

Multiplying M by 0.81989

Multiplying N by 0.83148

Multiplying O by 0.85218



2*16 bit 44100 Hz*19.843 sec

Listening test at ct

Reply #10 – 2002-08-23 15:34:10

Quote

Originally posted by Frank Klemm

Be attentively. Every test contains 6 lossy encoded files and a lossless (but time shifted) sample.

Ah, there is a hidden reference. No wonder I couldn't tell the difference on one of the files in the 64 test. So there may not be a clear winner on that test after all

Too late for me to apply any time offsets. I already submitted my results.

ff123

Listening test at ct

Reply #11 – 2002-08-23 17:41:04

Quote

Originally posted by ff123

Ah, there is a hidden reference. No wonder I couldn't tell the difference on one of the files in the 64 test. So there may not be a clear winner on that test after all

Too late for me to apply any time offsets. I already submitted my results.

ff123

For those who want to cut the files:

Part 1 (Heavily distored Kylie): 0...331972
Part 2 (Vocal + Jazz): 331973...621765
Part 3 (Classic): 621766...End

Listening test at ct

Reply #12 – 2002-08-23 17:42:46

Quote

Originally posted by ff123

Ah, there is a hidden reference. No wonder I couldn't tell the difference on one of the files in the 64 test. So there may not be a clear winner on that test after all

Too late for me to apply any time offsets. I already submitted my results.

ff123

I hope that the c't will name the used encoders. Because sound quality is related to
encoders, not to file formats. File formats have some intrinsic limits, but that is another
problem.

Listening test at ct

Reply #13 – 2002-08-23 18:37:40

Quote

Originally posted by ff123
I tried both the 64 and 128 tests. I think a clear winner and a clear loser will emerge from the 64 kbit/s test...

That's what I think.

Quote

On the 128 test, I could only tell the difference on two of the seven files.

I will have to do some training with pcabx. Sadly, I couldn't recognize ANY differences
I think even the referenz.wav is of baaad quality. Only the 64kbit-Test made it worse.

Listening test at ct

Reply #14 – 2002-08-23 20:16:03

Just did the 64kbit test and my results were nearly the same as the average rating so far (except for a change of ranks 5 and 6).

Two files (hence one lossy) were even transparent to me - Maybe because you get tired after all this listening. So I'll expect to fail with most or at least some of the 128kbit samples.

As ff123 stated the statistical noise will be a big problem especially with the 128kbit test. If I can only distinguish 2 files from the reference i have to give the other files random ranks.

I'm also hoping for the encoders and settings - otherwise the results would be pretty useless. Anyway, I'm very curious which codec the winner/loser is.

Listening test at ct

Reply #15 – 2002-08-23 20:33:09

Quote

Originally posted by caligae
Just did the 64kbit test and my results were nearly the same as the average rating so far (except for a change of ranks 5 and 6).

Two files (hence one lossy) were even transparent to me - Maybe because you get tired after all this listening. So I'll expect to fail with most or at least some of the 128kbit samples.

As ff123 stated the statistical noise will be a big problem especially with the 128kbit test. If I can only distinguish 2 files from the reference i have to give the other files random ranks.

I'm also hoping for the encoders and settings - otherwise the results would be pretty useless. Anyway, I'm very curious which codec the winner/loser is.

Two files beside the "referenz" file contain lossless audio.

Listening test at ct

Reply #16 – 2002-08-23 23:02:36

Quote

Originally posted by Frank Klemm

Two files beside the "referenz" file contain lossless audio.

Are you referring to the 128 set now? Because I clearly heard a difference on 6 of the 7 files for the 64 test.

ff123

Listening test at ct

Reply #17 – 2002-08-24 01:36:50

Quote

Originally posted by ff123

Are you referring to the 128 set now? Because I clearly heard a difference on 6 of the 7 files for the 64 test.

ff123

See output of abcx.

It shows (my) file names, delay differences and values from linear correlation.
A correlation quality of r=1.00000 shows (nearly) identically files.
A "files are identically" shows that the synchronizated part is bitwise identically.
The 'b' shows level effects. Equal leveled files have a b of 1.00000 or -1.00000 if they are
out of phase.

Listening test at ct

Reply #18 – 2002-08-24 02:12:06

Ok, one hidden reference for the 64 set and another hidden reference for the 128 set. BTW, people who plan to take the c'T test should not look too closely at Frank's output if they wish the reference to remain hidden!

ff123

Listening test at ct

Reply #19 – 2002-08-24 14:28:38

Quote

Originally posted by ff123
Ok, one hidden reference for the 64 set and another hidden reference for the 128 set. BTW, people who plan to take the c'T test should not look too closely at Frank's output if they wish the reference to remain hidden!

ff123

I'm currently adding some additional files to the c't files:

128 kbps: Ogg, Lame, Musepack, Psytel AAC, tooLame
64 kbps: Ogg, Psytel AAC

and dividing the files into the three parts. The results are a little bit strange.

Listening test at ct

Reply #20 – 2002-08-24 19:43:29

Just for kicks, I tried the 128 test a second time after waiting a while to forget what I was listening for before. I found the same two, but I described the artifacting a little bit differently ("stereo bloom" vs. "flange", and "stereo collapse" vs. "extra noise in center"). I think I can guess which one was the ogg. I also heard a third difference this time around, an obvious glitch which I noticed on the 64 test, but didn't look for on the 128 test the first time around.

Even if the encoder actually caused this glitch, I don't know if it's really fair that I penalized this file at either bitrate, because otherwise I wouldn't have noticed a difference, and maybe in most music ripped from CD I wonder if the glitch would be there at all.

I'm beginning to think that the files prepared for these listening tests should have both the beginnings and ends sliced off them.

ff123

Listening test at ct

Reply #21 – 2002-08-24 20:34:18

Quote

Originally posted by ff123
Just for kicks, I tried the 128 test a second time after waiting a while to forget what I was listening for before. I found the same two, but I described the artifacting a little bit differently ("stereo bloom" vs. "flange", and "stereo collapse" vs. "extra noise in center"). I think I can guess which one was the ogg. I also heard a third difference this time around, an obvious glitch which I noticed on the 64 test, but didn't look for on the 128 test the first time around.

Even if the encoder actually caused this glitch, I don't know if it's really fair that I penalized this file at either bitrate, because otherwise I wouldn't have noticed a difference, and maybe in most music ripped from CD I wonder if the glitch would be there at all.

I'm beginning to think that the files prepared for these listening tests should have both the beginnings and ends sliced off them.

ff123

***64 kbps***
---best---
6140
6144, 6145,ogg
ogg(managed),6141,6143
6142
--worsed--

***128 kbps***
--best--
Musepack 1.1, 1.9
Ogg, 6147, 6153
lame aps128, 6151, 6153
6152, Psytel AAC, 6149
6150. toolame -mj
--worsed--

Note that Musepack is tuned for my hearing.

Listening test at ct

Reply #22 – 2002-08-25 08:25:09

Quote

Originally posted by Frank Klemm

***128 kbps***
--best--
Musepack 1.1, 1.9
Ogg, 6147, 6153
lame aps128, 6151, 6153
6152, Psytel AAC, 6149
6150. toolame -mj
--worsed--

Hmm... shouldn't one of the numbered files be the best one (=sample-moved wave)?

What was your Musepack command line?

Listening test at ct

Reply #23 – 2002-08-25 11:58:00

Quote

Originally posted by Continuum
What was your Musepack command line?

Quote

Originally posted by Frank Klemm
Version 1.1 uses --quality 4.65, version 1.9 --quality 4.75

Listening test at ct

Reply #24 – 2002-08-25 13:13:55

Quote

Originally posted by Continuum
Hmm... shouldn't one of the numbered files be the best one (=sample-moved wave)?

Do you miss any numbers?

Notice