IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Randomizing file names, to compare lossy files etc.
Brand
post Dec 26 2012, 21:05
Post #1





Group: Members
Posts: 312
Joined: 27-November 09
Member No.: 75355



As you may know, comparing various lossy files (by ABXing each of them against the lossless file or something else) can be subject to some bias, if you know which of the lossy files you're testing.
You might have a preference for one lossy format or encoder and so you might (subconsciously) do the tests differently.

There's a way to eliminate this potential bias, by randomly renaming the lossy files.
Of course, at the end, you also need to know which file is which.
For this purpose, I found a simple script (Windows) that does just that: http://www.howtogeek.com/57661/stupid-geek...in-a-directory/
Put some files into the folder with the script, run the script and you will get: renamed files + a txt file that tells you which file is which, so you can check when you're finished.

To then use those files, it helps if they're of the same size/duration/metadata (easier with simple CBR files like WAV), so that Explorer (or whatever file manager you're using) doesn't give you any hints.
Even with different file sizes, you can select the icon view in Explorer, so that unless you hover over a file for a second or two, you won't see the details. You can then Ctrl+A on the files, unselect the ones you don't need and add the selected ones to Foobar without getting any extra information. You'll see what I mean when you try it in practice.


I found this an effective way to randomize files for blind testing on your own.
But if there's an easier way, let me know.

There's at least one thing that would improve this, though: copy the randomized files to clipboard. This would eliminate the need to select the files carefully, since they could simply be pasted into Foobar. Is there a way to add this to the script?
Go to the top of the page
+Quote Post
probedb
post Dec 26 2012, 22:40
Post #2





Group: Members
Posts: 1121
Joined: 6-September 04
Member No.: 16817



I thought ABXing meant you had no idea what file was which anyway?
Go to the top of the page
+Quote Post
Porcus
post Dec 27 2012, 02:42
Post #3





Group: Members
Posts: 1779
Joined: 30-November 06
Member No.: 38207



QUOTE (Brand @ Dec 26 2012, 21:05) *
But if there's an easier way, let me know.


http://www.foobar2000.org/components/view/foo_abx ?


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
AndyH-ha
post Dec 27 2012, 07:03
Post #4





Group: Members
Posts: 2191
Joined: 31-August 05
Member No.: 24222



The goal here is to compare a variety of different kinds of lossy encoding against the original, one at a time. Each time, the original is one of the two in the comparison. ABX software prevents one from knowing which is the original, which is the encoding.

HOWEVER, ABX software does not prevent one from knowing which lossy encoding was selected for the test each time. This later is the problem the OP wishes to overcome and has attacked with random file renaming.
Go to the top of the page
+Quote Post
greynol
post Dec 27 2012, 08:09
Post #5





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



http://en.m.wikipedia.org/wiki/MUSHRA
http://wiki.hydrogenaudio.org/index.php?title=ABC/HR


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Brand
post Dec 27 2012, 11:30
Post #6





Group: Members
Posts: 312
Joined: 27-November 09
Member No.: 75355



As AndyH-ha explained, this is for doing several separate ABX tests.

Just a quick practical example: you want to determine which is (more) transparent at 150kbps, MP3 or Vorbis. So you do an ABX test with the MP3 against the lossless and then another ABX test with Vorbis against the lossless.
If you know whether you're ABXing MP3 or Vorbis, you could be (subconsciously) biased and perform the tests differently. Randomly renaming the files first prevents you from knowing this.
(It would make sense to convert the MP3 and Vorbis to WAV before randomizing.)


Randomly renaming the files could also be used for simply playing back the files on their own (no ABX) to subjectively evaluate which file 'sounds better'.
ABC/HR mentioned above is more sophisticated for this kind of task, but it requires a dedicated tool or an external tester.

This post has been edited by Brand: Dec 27 2012, 11:40
Go to the top of the page
+Quote Post
greynol
post Dec 28 2012, 08:05
Post #7





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



To me the idea of being more transparent is like the idea of being more pregnant. Either something is or it isn't. If you're trying to rank something then ABX is the wrong tool.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Brand
post Dec 28 2012, 10:21
Post #8





Group: Members
Posts: 312
Joined: 27-November 09
Member No.: 75355



Let me put it this way, then: some non-transparencies (or pregnancies) are easier to notice than others. In this sense, I think ABXing can be used for ranking.
And if in the process of separate ABXing of two lossy samples you notice that one of them is transparent and the other isn't, you have a clear winner anyway.
Go to the top of the page
+Quote Post
Porcus
post Dec 28 2012, 11:35
Post #9





Group: Members
Posts: 1779
Joined: 30-November 06
Member No.: 38207



QUOTE (greynol @ Dec 28 2012, 08:05) *
To me the idea of being more transparent is like the idea of being more pregnant. Either something is or it isn't. If you're trying to rank something then ABX is the wrong tool.


More visibly pregnant, I guess? Problem is, the original A may be indistinguishable from B may be indistinguishable from C, yet C might not be transparent. “ABX and ACX” could be helpful then.


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
Nessuno
post Dec 28 2012, 12:25
Post #10





Group: Members
Posts: 422
Joined: 16-December 10
From: Palermo
Member No.: 86562



QUOTE (Porcus @ Dec 28 2012, 11:35) *
More visibly pregnant, I guess? Problem is, the original A may be indistinguishable from B may be indistinguishable from C, yet C might not be transparent. “ABX and ACX” could be helpful then.

In this case would you say that B is "more transparent" than C (which simply is not)?
Or, put in another way, if a x quality level encoding of a track is proven to be transparent, a x + Dx level (same track, codec and encoder) is more transparent?


--------------------
... I live by long distance.
Go to the top of the page
+Quote Post
Porcus
post Dec 28 2012, 12:46
Post #11





Group: Members
Posts: 1779
Joined: 30-November 06
Member No.: 38207



Has less audible differences? Less “intransparencies”? (Sounds like a euphemism from politically correct American speek ...)

Point: If you can tell A from C, but not A from B nor B from C, that indicates an ordering of ABC or CBA.


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
greynol
post Dec 28 2012, 13:40
Post #12





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



Originally what Porcus said sounded different from the situation where C is transparent to A but can be distinguished from B which is also transparent to A, assuming both are lossy and A is the lossless source. Either way it is solved by performing ABX on one lossy codec at a time which still doesn't do any better at ranking what is only a binary state.

Why not just use the correct tool which automatically does all the heavy-lifting for you? It isn't like one doesn't exist.

This post has been edited by greynol: Dec 28 2012, 13:57


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Brand
post Dec 28 2012, 16:45
Post #13





Group: Members
Posts: 312
Joined: 27-November 09
Member No.: 75355



At the risk of repeating myself or stating the obvious, I think we're talking about two different tasks here:

1. which sample is harder or impossible to ABX (aka "sounds more like the lossless")
and
2. which sample sounds better (even without the lossless to compare it against)

The highest ranking sample in an ABC/HR test might not be the hardest to ABX. Yet, the latter might be what one wants.

This is why I think ABC/HR is mostly used for lower bitrates, where audible artifacts are noticeable or at least expected, while ABXing is used when testing for transparency.
Go to the top of the page
+Quote Post
greynol
post Dec 28 2012, 18:23
Post #14





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



Lol, do I have to risk repeating myself too?

You may rank any way you want just as you can load any samples you like. The reason high bitrates are avoided has to do with getting useful information from public tests which would just as well exist with ABX-style tests.

I'd also like to know if anyone here has ever personally been encumbered by knowing which codecs were being tested while performing an ABX test.

This post has been edited by greynol: Dec 28 2012, 18:57


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Brand
post Dec 28 2012, 18:57
Post #15





Group: Members
Posts: 312
Joined: 27-November 09
Member No.: 75355



Frankly, I don't know what your point is.

Is "sample A was easy to ABX, while sample B was difficult/impossible to ABX" not useful information?


QUOTE (greynol @ Dec 28 2012, 18:23) *
I'd also like to know if anyone here has ever personally been encumbered by knowing which codes were being tested while performing an ABX test.

Oh, come on. We're on HA, sub-forum Listening Tests.
I would expect us all to be familiar with the placebo effect and with taking proper precautions to avoid any possible bias.
Go to the top of the page
+Quote Post
greynol
post Dec 28 2012, 19:33
Post #16





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



To the best of my recollection I know of no issues raised about being biased based on knowing what was being tested...until now. ABC/HR and ABX already take care of that as you were already told. You were also either directly or indirectly told that ABX should not be lossy vs. lossy by at least two of us.

My point is that your needs can be met with ABC/HR better than with ABX. This is even taking each and every one of your "concerns" into account. This includes one being less transparent than another. smile.gif

I personally don't see any problems with your trying to reinvent the wheel other than you are trying to reinvent the wheel. wink.gif

If it makes any difference I've tested near-transparent (read: transparent to untrained ears) codecs professionally. The style of test used: MUSHRA not ABX. IME, the best way to bias/unbiased/train/inform yourself is to work on a codec at a time varying bitrates on various samples. Research the codecs to find thier weaknesses in order to choose the best samples to hone in most efficiently. Otherwise let ignorance be bliss.

This post has been edited by greynol: Dec 28 2012, 20:33


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Brand
post Dec 28 2012, 20:48
Post #17





Group: Members
Posts: 312
Joined: 27-November 09
Member No.: 75355



Alright, got it. I don't think it's reinventing the wheel, it's not meant to replace ABC/HR, MUSHRA or anything else. It's a specific method for specific tests and results. Whether you find it of any use or you prefer another method it's up to you.

For the record, though: I never suggested ABXing lossy vs. lossy, so I don't know why you'd mention that.
And the problem of expectation bias regarding ABX tests (when you know which encoder/bitrate you're ABXing) has been brought up before here on HA, it's not my original idea.
Go to the top of the page
+Quote Post
greynol
post Dec 28 2012, 21:35
Post #18





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



1) I didn't raise the issue of the problems of lossy vs. lossy.

2) I don't view concerns about sample bias at transparent or near-transparent bitrates with any real seriousness. I'll believe people do better in an ABX test due to bias when I see it and have a hard time seeing the possibility of doing worse as a bad thing other than it's reminiscent of a fairly common placebophile argument. I also don't think it's unreasonable to dismiss as trivial concerns about bias against codecs using settings that are obviously not transparent with codecs that are known to give sub-par results at these settings while also testing more modern codecs that are generally considered to do well in comparison (e.g. 96kbit. Mp3 vs. 96kbit aac). Biases against specific modern codecs in favor of other modern codecs? Use MUSHRA.

3) "Ease" (or lack thereof) of ABX-ing could easily be due to fatigue, or correctly guessing. For reasons like these, P-values are not going to be a reasonable gauge of definitive quality, unless they demonstrate the ability to distinguish one lossy codec but not for another consistently. I will concede that one codec can be easier to test compared to another, with both giving passing results, in which case the time it took could be used as an indicator for someone else reviewing the results. But why not rank one 5 or 10 points higher (on a 100-point scale) in a MUSHRA test???

If this doesn't move you then we will have to agree to disagree. From here on out my comments are for the benefit of others who might not know that there are already perfectly good methods to accomplish your end goal as you stated it. So long as I feel I can contribute useful information to the discussion, I will continue to do so. smile.gif

This post has been edited by greynol: Dec 29 2012, 06:53


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th April 2014 - 05:18