Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: mppenc ABX test #1 (Read 6489 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

mppenc ABX test #1

This is the first of a few short and very simple ABX tests I want MPC users to perform. I provide a BitTorrent link to a .zip file (a fast HTTP address will serve as a backup, but please use BT) with 3 48 KHz WavPack files (SSRCed from 44.1 to 48 to avoid aliasing issues I've confirmed with some of the testers in the past) that you should unpack and encode with mppenc 1.14 and 1.15v (both are on the official Musepack site). Use whichever program you prefer for the ABX test, but please post the log showing your success rate at telling the .mpc from the original .wav (WAV vs. 1.14 and also WAV vs. 1.15v). I want to shape some sort of a profile so we'd know what most users hear and don't hear. I don't mind if you expose your results for others to see. In the spirit of openness, we should share all info and explain what flaws we heard. If you can provide successful results at --extreme or above, great. If only --standard, post as well. All results are meaningful. In 2 weeks the test will be over and new samples will be posted.

I'll post the results I had with one of the samples. You should provide as much info as you can about your equipment (headphones, sound card) and the nature of the flaws you hear. Notice that most of the samples I'll post had to be shortened to avoid copyright infringement issues. The problematic range of the sample we (the devs) hear is isolated and the short sample should be easier to work with.

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 15:08:02

File A: file://D:\Test\golem2-cut-48.wav
File B: file://D:\Test\golem2-cut-48.mpc

14:13:54 : Test started.
14:14:21 : 01/01  50.0%
14:14:39 : 02/02  25.0%
14:15:04 : 03/03  12.5%
14:16:25 : 04/04  6.3%
14:18:37 : 05/05  3.1%
14:18:45 : 06/06  1.6%
14:18:52 : 07/07  0.8%
14:20:18 : 08/08  0.4%
14:20:19 : Test finished.

Comments: mppenc 1.15v, M-Audio Revolution 7.1 sound card, Sennheiser HD495S headphones. Quality setting: --extreme

----------
Total: 8/8 (0.4%)
------------------------------

Test sample suite #1 - BitTorrent Link
Test sample suite #1 - HTTP Link
And if Warhol's a genius, what am I? A speck of lint on the ***** of an alien

mppenc ABX test #1

Reply #1
Ok, here you go.
Amesia @ quality 5,6 & 7. mppenc 1.15v
Headphones: Sennheiser HD 600, soundcard: E-mu 1212m.
Artifacts: 'Unstable' transients, background noise
Mood: Lazy and unconcentrated (!)   

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 14:38:03

File A: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.q5.MPC

14:38:04 : Test started.
14:38:24 : 01/01  50.0%
14:38:30 : 02/02  25.0%
14:38:33 : 03/03  12.5%
14:38:44 : 04/04  6.3%
14:38:54 : 05/05  3.1%
14:39:00 : 06/06  1.6%
14:39:06 : 07/07  0.8%
14:39:17 : 08/08  0.4%
14:39:18 : Test finished.

----------
Total: 8/8 (0.4%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 14:48:14

File A: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.q6.MPC

14:48:15 : Test started.
14:48:34 : 01/01  50.0%
14:48:40 : 02/02  25.0%
14:48:44 : 03/03  12.5%
14:49:02 : 04/04  6.3%
14:49:06 : 05/05  3.1%
14:49:13 : 06/06  1.6%
14:49:29 : 07/07  0.8%
14:49:47 : 08/08  0.4%
14:49:49 : Test finished.

----------
Total: 8/8 (0.4%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 14:51:24

File A: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.q7.MPC

14:51:24 : Test started.
14:51:34 : 01/01  50.0%
14:51:38 : 02/02  25.0%
14:51:44 : 03/03  12.5%
14:51:50 : 04/04  6.3%
14:51:56 : 05/05  3.1%
14:52:02 : 06/06  1.6%
14:52:11 : 06/07  6.3%
14:52:16 : 07/08  3.5%
14:52:21 : 08/09  2.0%
14:52:26 : 09/10  1.1%
14:52:32 : 10/11  0.6%
14:52:39 : 10/12  1.9%
14:52:43 : 11/13  1.1%
14:52:47 : 12/14  0.6%
14:52:58 : 13/15  0.4%
14:53:17 : 14/16  0.2%
14:53:19 : Test finished.

----------
Total: 14/16 (0.2%)


-tm/
I can stay for hours listening to all the sounds... (Samuel Beckett, Rough for Theatre I)

mppenc ABX test #1

Reply #2
Some more results, for the hex sample at quality 5,6,7,8 (mppenc 1.15v)
Same listening equipment, but better concentration this time 

General comments: imprecise tonality of the micro-attacks in the background of this sample, some added noise (pre-echo?) in the same region

Remark for the sample at quality 5: there was a very easy to spot channel imbalance at the beginning of this sample at quality 5, it seemed like the channel balance was moved to the right. This problem disappeared at higher quality settings.

Results for q5, q6, q7, q8:

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 15:34:13

File A: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.q5.MPC

15:34:14 : Test started.
15:34:38 : 01/01  50.0%
15:34:44 : 02/02  25.0%
15:34:54 : 03/03  12.5%
15:35:01 : 04/04  6.3%
15:35:04 : 05/05  3.1%
15:35:13 : 06/06  1.6%
15:35:18 : 07/07  0.8%
15:35:25 : 08/08  0.4%
15:35:26 : Test finished.

----------
Total: 8/8 (0.4%)


foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 16:16:57

File A: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.q6.MPC

16:16:57 : Test started.
16:17:07 : 01/01  50.0%
16:17:15 : 02/02  25.0%
16:17:20 : 03/03  12.5%
16:17:26 : 04/04  6.3%
16:17:34 : 05/05  3.1%
16:17:57 : 06/06  1.6%
16:18:02 : 07/07  0.8%
16:18:11 : 08/08  0.4%
16:18:13 : Test finished.

----------
Total: 8/8 (0.4%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 16:25:41

File A: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.q7.MPC

16:25:42 : Test started.
16:25:56 : 01/01  50.0%
16:26:03 : 02/02  25.0%
16:26:11 : 03/03  12.5%
16:26:49 : 04/04  6.3%
16:26:54 : 05/05  3.1%
16:27:01 : 06/06  1.6%
16:27:08 : 07/07  0.8%
16:27:19 : 08/08  0.4%
16:27:20 : Test finished.

----------
Total: 8/8 (0.4%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 16:28:21

File A: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.q8.MPC

16:28:21 : Test started.
16:29:00 : 01/01  50.0%
16:29:06 : 02/02  25.0%
16:29:11 : 03/03  12.5%
16:29:22 : 03/04  31.3%
16:29:29 : 04/05  18.8%
16:29:35 : 05/06  10.9%
16:29:42 : 06/07  6.3%
16:30:04 : 07/08  3.5%
16:30:52 : 08/09  2.0%
16:31:04 : 09/10  1.1%
16:31:14 : 10/11  0.6%
16:31:35 : 11/12  0.3%
16:31:49 : 12/13  0.2%
16:32:16 : 13/14  0.1%
16:32:40 : 14/15  0.0%
16:32:54 : 15/16  0.0%
16:32:55 : Test finished.

----------
Total: 15/16 (0.0%)


-tm/
I can stay for hours listening to all the sounds... (Samuel Beckett, Rough for Theatre I)

mppenc ABX test #1

Reply #3
I used AKG K66 headphones and Creative SB PCI soundcard.

amnesia-48 - horrible quality even at q10.

A file: C:\x\amnesia-48.wav
B file: C:\x\amnesia-48 1.14 q10.wav

05:39:14    1/1  p=50.0%
05:39:18    2/2  p=25.0%
05:39:22    3/3  p=12.5%
05:39:27    4/4  p=6.2%
05:39:30    5/5  p=3.1%
05:39:34    6/6  p=1.6%
05:39:37    7/7  p=0.8%
05:39:44    8/8  p=0.4%
05:39:47    9/9  p=0.2%
05:39:52  10/10  p< 0.1%


Some better results with 1.15v, but not convincing.

A file: C:\x\amnesia-48.wav
B file: C:\x\amnesia-48 1.15v q10.wav

05:42:32    1/1  p=50.0%
05:42:36    2/2  p=25.0%
05:42:40    3/3  p=12.5%
05:42:49    4/4  p=6.2%
05:42:53    5/5  p=3.1%
05:42:56    6/6  p=1.6%
05:42:59    7/7  p=0.8%
05:43:02    8/8  p=0.4%
05:43:07    9/9  p=0.2%
05:43:10  10/10  p< 0.1%


hex-48

A file: C:\x\hex-48.wav
B file: C:\x\hex-48 1.14 q10.wav

05:57:02    1/1  p=50.0%
05:57:07    2/2  p=25.0%
05:57:14    3/3  p=12.5%
05:57:20    4/4  p=6.2%
05:57:26    5/5  p=3.1%
05:57:31    6/6  p=1.6%
05:57:37    7/7  p=0.8%
05:57:41    8/8  p=0.4%
05:57:47    9/9  p=0.2%
05:57:52  10/10  p< 0.1%

Same results with 1.15v.

Quote
Remark for the sample at quality 5: there was a very easy to spot channel imbalance at the beginning of this sample at quality 5, it seemed like the channel balance was moved to the right. This problem disappeared at higher quality settings.

The problem did not disappear to me. The only "artifact" I hear is the bad stereo separation.


golem2-cut-48

I can ABX it only at q4.

mppenc ABX test #1

Reply #4
hex-48 with 1.14 and 1.15v, both q5.
Equipment: "Consumer hardware". No soundcard (Intel 82801AA chipset) and $3.75 earphones.

Code: [Select]
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname: hex-48 - orig. vs. 1.15v vs. 1.14, --standard

Tester: znode

1R = H:\Incoming\Test\hex-48.1.14.standard.wav
2L = H:\Incoming\Test\hex-48.1.15v.standard.wav

---------------------------------------
General Comments:
No other artefacts heard besides stereo shift :(
---------------------------------------
1R File: H:\Incoming\Test\hex-48.1.14.standard.wav
1R Rating: 3.5
1R Comment: Stereo shift very obvious in the first 2 seconds, and continues up to 4.
---------------------------------------
2L File: H:\Incoming\Test\hex-48.1.15v.standard.wav
2L Rating: 3.9
2L Comment: Locations of stereo shift same as sample 1, but intensity of shift is... less? Unable to prove that by ABX testing.
---------------------------------------
ABX Results:
Original vs H:\Incoming\Test\hex-48.1.14.standard.wav
   8 out of 8, pval = 0.004
Original vs H:\Incoming\Test\hex-48.1.15v.standard.wav
   13 out of 13, pval < 0.001
H:\Incoming\Test\hex-48.1.14.standard.wav vs H:\Incoming\Test\hex-48.1.15v.standard.wav
   5 out of 13, pval = 0.867

mppenc ABX test #1

Reply #5
I've tested amnesia with 1.14 and 1.15v at quality 4-7

I'm not a well trained ABXer so i started at 4 and moved up.
I used my Beyerdynamic DT531 plugged into the headphone out of my creative inspire system which is connected to my onboard Nvidia sounstorm

Playback through foobar2k with resampling at 48khz due to the soundstorm

q4: very obvious ringing which is a bit better with 1.15v. thats the only artefact that i could consentrate on, i didn't find another thing that sounded wrong, especially so wrong that i could tell it apart from the original.

q5: the ringing is much better, normal people probably wont hear it if they play it back through speakers.
again 1.15v is better

q6: very difficult for me to hear, sometimes i can tell it apart, sometimes it's impossible, 1.15v seems to be better but that's not trustworthy 

q7: with 1.14 i was able to tell it apart sometimes, didn't test 1.15v because it was at q6 nearly impossible...

hope this helps, here are he logs:

Code: [Select]
foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 15:21:58

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.14 q4\amnesia-48.mpc

15:21:58 : Test started.
15:22:19 : 01/01  50.0%
15:22:28 : 02/02  25.0%
15:22:31 : 03/03  12.5%
15:22:34 : 04/04  6.3%
15:22:37 : 05/05  3.1%
15:22:40 : 06/06  1.6%
15:22:45 : 07/07  0.8%
15:22:48 : 08/08  0.4%
15:22:51 : 09/09  0.2%
15:22:54 : 10/10  0.1%
15:22:56 : Test finished.

----------
Total: 10/10 (0.1%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 14:40:19

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.14 q5\amnesia-48.mpc

14:40:19 : Test started.
14:40:45 : 01/01  50.0%
14:40:51 : 02/02  25.0%
14:41:08 : 03/03  12.5%
14:41:20 : 04/04  6.3%
14:41:32 : 05/05  3.1%
14:41:42 : 06/06  1.6%
14:41:56 : 07/07  0.8%
14:42:09 : 07/08  3.5%
14:42:26 : 08/09  2.0%
14:42:33 : Test finished.

----------
Total: 8/9 (2.0%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 14:49:47

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.14 q6\amnesia-48.mpc

14:49:48 : Test started.
14:52:16 : 00/01  100.0%
14:53:03 : 00/02  100.0%
14:53:14 : 01/03  87.5%
14:53:27 : 01/04  93.8%
14:54:06 : 01/05  96.9%
14:54:25 : 02/06  89.1%
14:54:47 : 03/07  77.3%
14:55:38 : 04/08  63.7%
14:55:55 : 04/09  74.6%
14:56:05 : 05/10  62.3%
14:56:19 : 06/11  50.0%
14:56:58 : 07/12  38.7%
14:57:23 : 07/13  50.0%
14:57:28 : Test finished.

----------
Total: 7/13 (50.0%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 14:57:51

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.14 q7\amnesia-48.mpc

14:57:52 : Test started.
14:58:30 : 01/01  50.0%
14:58:50 : 02/02  25.0%
14:59:47 : 03/03  12.5%
15:00:34 : 03/04  31.3%
15:00:49 : 04/05  18.8%
15:00:55 : 04/06  34.4%
15:01:39 : 04/07  50.0%
15:02:38 : 05/08  36.3%
15:02:52 : 06/09  25.4%
15:03:02 : 06/10  37.7%
15:03:06 : Test finished.

----------
Total: 6/10 (37.7%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 15:03:20

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.15v q4\amnesia-48.mpc

15:03:21 : Test started.
15:04:06 : 01/01  50.0%
15:04:09 : 02/02  25.0%
15:04:13 : 03/03  12.5%
15:04:20 : 04/04  6.3%
15:04:28 : 05/05  3.1%
15:04:31 : 06/06  1.6%
15:04:38 : 07/07  0.8%
15:04:44 : 08/08  0.4%
15:04:50 : 09/09  0.2%
15:04:54 : 10/10  0.1%
15:04:56 : Test finished.

----------
Total: 10/10 (0.1%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 15:05:36

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.15v q5\amnesia-48.mpc

15:05:37 : Test started.
15:06:20 : 01/01  50.0%
15:06:27 : 02/02  25.0%
15:07:13 : 03/03  12.5%
15:07:46 : 03/04  31.3%
15:08:33 : 03/05  50.0%
15:12:54 : 04/06  34.4%
15:13:07 : 05/07  22.7%
15:13:22 : 05/08  36.3%
15:14:18 : 06/09  25.4%
15:14:42 : 07/10  17.2%
15:14:47 : Test finished.

----------
Total: 7/10 (17.2%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 15:15:53

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.15v q6\amnesia-48.mpc

15:15:53 : Test started.
15:16:32 : 00/01  100.0%
15:17:18 : 01/02  75.0%
15:17:27 : 02/03  50.0%
15:17:44 : 03/04  31.3%
15:18:00 : 04/05  18.8%
15:18:31 : 05/06  10.9%
15:18:57 : 05/07  22.7%
15:19:19 : 05/08  36.3%
15:19:36 : 05/09  50.0%
15:20:15 : 05/10  62.3%
15:20:19 : Test finished.

----------
Total: 5/10 (62.3%)
member of the "i have a cat-avatar"-group ;)