Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Impact of listening fatigue , during ABX test. (Read 21826 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Impact of listening fatigue , during ABX test.

Hi ,
I  thought I  could test how high bit rates could be useful for me,
by doing an ABX  test based on a sample rich on high pitched sound.
Correct me if I'm wrong, but high bit rates help to preserve high frequencies.

So I used a sample from the OST of game "sentinel return"
from John Carpenter. I precise, because it's not an artificial sample
generated by me, but a music that exists.
In attachment , there's a lossless flac version,
and one encoded at 128kps cbr (lame 3.98).

I  tried to make an ABX test with two samples (using ABX  comparator from foobar),
but I  realized that listening to the sample again and again was a bit fatiguing for my ears.
  Anyway here are the questions that arise from this test.

First are you experiencing listening fatigue like me with this sample,
and  can you ABX  them easily ? I'm not sure if that's important, but I use headphones, and with
relatively loud volume, compared to my normal level for regular listening.

For the ones experiencing listening fatigue, how would you overcome this ?
Personally, I  try to make a big pause of 30 second at least before making
a decision at each step of ABX  test , just listening at silence or quiet
music. But that's not obvious.

And finally , do you think that the impact of listening fatigue can
be ignored during regular ABX tests ?
I've done  many times  some ABX  tests, getting successful results in the 4/5  first steps ,
and then getting errors.
So either the ABX  comparator from foobar, is not truly random,
and I'm good at guessing the first steps, or I'm loosing my ability
to focus on the differences because of listening fatigue or anything else (like
distracting background noise).

Impact of listening fatigue , during ABX test.

Reply #1
You can have long pauses if you want. So listening fatigue is not a factor.

You can use speakers (instead of headphones) if you want. That's not a factor as well.

Impact of listening fatigue , during ABX test.

Reply #2
@Light Fire
The question is that if you are always aware or not of listening fatigue.
You might have wrong conclusion (i.e I'm unable to ABX)
if you don't pay attention to this problem.
Do you usually take a long  pause, if after some steps, you become undecided if A or B  is X ?
How far, are you ready to go , to be sure you are unable to hear a difference ?
Do you raise the volume when suddenly you become undecided,
or try to rest your ears ?
I think that after some times, average people just prefer to give up ,
and conclude "I'm unable to ABX".

Also, If some people want to play with the sample files and try to ABX , I'm not against some reactions.
But maybe, this sample is only significant for me, I  don't know.


Impact of listening fatigue , during ABX test.

Reply #3
Yes, sometimes I'm aware of listening fatigue, it gets harder to detect again and again the same distortion. When this happens I just stop and try again next day or at least wait several hours. I ABX with headphones and the volume is VERY important. The higher the volume the worse, but a volume too low leaves artifacts undetected. After abx-ing for some time with the same equipment I have found a kind of sweet spot of volume settings. (I usually do mid-high bitrate abx-ing)

Impact of listening fatigue , during ABX test.

Reply #4
It takes me sometimes a lot of time, without a pause, until I spot the difference and only then can I successfully ABX.
But of course, after longer period spent ABX-ing (about 30 minutes) fatigue hits me. Then I need at least 10 minutes until I recover.

Impact of listening fatigue , during ABX test.

Reply #5
Well, thanks for the answers.  I  got the clear feeling that  ABX-ing can induce fatigue, and obviously this can  be  true for other people.
I  retain that Alexxander wait at least several hours to cancel effects of fatigue,  and 10 minutes for muaddib.
I  guess that ABX-ing successfully can take sometimes much more than an hour, and not everyone  have this level of patience. Unless you are lucky and have great ears ...

Impact of listening fatigue , during ABX test.

Reply #6
I would like to know the point of ABXing at higher than normal listening levels. Isn't it more important to know if you can tell the difference at normal listening levels?

 

Impact of listening fatigue , during ABX test.

Reply #7
@pdq,
Quote
Isn't it more important to know if you can tell the difference at normal listening levels?

I  guess it depends of the purpose of the ABX test, and how much you are perfectionist  in  term of sound quality.
Once I  got no clue of which file sound better during an ABX test , and the only way for me to spot an interesting difference (level of noise), was to raise the volume a  lot.

Impact of listening fatigue , during ABX test.

Reply #8
I  guess it depends of the purpose of the ABX test, and how much you are perfectionist  in  term of sound quality.
Once I  got no clue of which file sound better during an ABX test , and the only way for me to spot an interesting difference (level of noise), was to raise the volume a  lot.


If by "perfectionist" you mean you desire audio which is unchanged from the original - any bitrate of lossy is the wrong way to go for you.
If you must go to extraordinary measures you can likely find all sorts of flaws in perceptual encoders, they must make some assumptions about "normal" listening.
Creature of habit.

Impact of listening fatigue , during ABX test.

Reply #9
First are you experiencing listening fatigue like me with this sample,
and  can you ABX  them easily ? I'm not sure if that's important, but I use headphones, and with
relatively loud volume, compared to my normal level for regular listening.

I found listening fatigue a problem with this sample, until the second half of my ABX session when I stumbled on a section of the files that was more obviously different. [The section of the files from 22.4 sec to 23.0 sec sounded obviously deeper for my hearing with the flac version compared with the mp3.] For the record, here is the ABX result:
foo_abx 1.3.4 report
foobar2000 v0.9.6.8
2009/10/10 09:55:51

File A: C:\Downloads\_extracted\carpenter.zip.extracted\Carpenter_128kps_cbr.mp3
File B: C:\Downloads\_extracted\carpenter.zip.extracted\Carpenter_FLAC.flac

09:55:51 : Test started.
09:57:53 : 01/01  50.0%
09:59:54 : 02/02  25.0%
10:05:27 : 03/03  12.5%
10:05:53 : 04/04  6.3%
10:06:28 : 05/05  3.1%
10:06:45 : 06/06  1.6%
10:07:03 : 07/07  0.8%
10:07:15 : 08/08  0.4%
10:07:27 : 09/09  0.2%
10:07:53 : 10/10  0.1%
10:08:00 : Test finished.

----------
Total: 10/10 (0.1%
)

Quote
For the ones experiencing listening fatigue, how would you overcome this ?
Personally, I  try to make a big pause of 30 second at least before making
a decision at each step of ABX  test , just listening at silence or quiet
music. But that's not obvious.

I suggest there are different aspects of listening fatigue at work including:
(1) desentitization of the hearing organ itself (organ of Corti).  [I think my high frequency hearing is susceptible to this.]
(2) the hearing analysis part of the brain becoming desentitized when repeatedly presented with what is essentially the same stimulus.
(3) the brain relying in part on its memory of its last analysis of the sound event, part on its accumulated internal modelling of what such sound events typically sound like, and part on the raw nerve impulses from the organ of Corti. [As time goes on of repeatedly listening to the same section of music, the brain may automatically place greater weight on synthesis and imagination, rather than narrowly focussing on the raw data.]

An aspect of 3 is illustrated by the fact that one word in the dialogue in a sitcom may be unintelligible to a particular listener for the first 5 playings of a section of dialogue, but if the listener realises what that word is on the 6th playing, then on any subsequent playing the initially unintelligible word is likely to be readily understood by that listener. In fact the listener may find it hard to imagine how they could possibly not have understood the initially unintelligible word.

I find the best ways to deal with listening fatigue are to:
(a) change the section of the ABX files I am listening to
(b) take a 10 minute break [if everything starts sounding the same!].


Quote
And finally , do you think that the impact of listening fatigue can
be ignored during regular ABX tests ?
I've done  many times  some ABX  tests, getting successful results in the 4/5  first steps ,
and then getting errors.
So either the ABX  comparator from foobar, is not truly random,
and I'm good at guessing the first steps, or I'm loosing my ability
to focus on the differences because of listening fatigue or anything else (like
distracting background noise).

I think that ABXing in the initial steps is artificially easy in that an A B test of a short segment of a recording can reveal extremely subtle differnces that a listener might not identify if listening to the entirety of the recording A, waiting half an hour, and then playing the enirety of recording B.  Also the listener can repeat the samples as often as they wish, before choosing to provide an answer and proceed to the next "step".

On the other hand, I find that my own hearing is often best at detecting a difference in the very first A B test of a short segment, and that "listening fatigue" sets in very quickly.  After 4 or 5 "steps" detection becomes difficult, even though initially the discrepancy between A and B can seem plain.  As I say, my way of getting around this problem is to switch to another section of the files.

Impact of listening fatigue , during ABX test.

Reply #10
@Soap

Quote
If you must go to extraordinary measures you can likely find all sorts of
flaws in perceptual encoders


I don't consider that raising the volume, or trying to avoid listening fatigue
, to be extraordinary measures. Probably it depends of what you are expecting.
In my case , I  tried  here, to see if a high bit rate could be justified  to
satisfy my ears . I used an example I  thought "obvious" at first , but that
was not the case at the end. I  was also curious to see  if codec developpers,
take in consideration the problem of listening fatigue that can occur during
AB-X  tests, when tweaking a lossy codec (I guess that's the case according to
  muaddib answer...).

Quote
If by "perfectionist" you mean you desire audio which is unchanged from
the original - any bitrate of lossy is the wrong way to go for you.


During an AB-X test, you don't necessarily compare a lossy version , with a
lossless version.  Example: two versions encoded with different lossy codec.
But yes, if you just want to listen to the most "faithful" version, you might
just listen at the source, and not bother with any lossy codec.


@MLXXX
Quote
The section of the files from 22.4 sec to 23.0 sec sounded obviously deeper
for my hearing with the flac version compared with the mp3.


Thanks for the indication, will try to AB-X  the section later.

Quote
I think that ABXing in the initial steps is artificially easy


Or artificially hard...  Being forced to focus on audible differences,
is not "natural".  I  usually like to do other stuff while listening to music.

Quote
As I say, my way of getting around this problem is to switch to another section of the files.


Great, that  it works for you. Thanks for the suggestion.

Impact of listening fatigue , during ABX test.

Reply #11
10.3-11.5 secs its easy to abx. Smearing. 1sec pauses help.

Even more disturbing around 11.8-13 secs and @ 16sec !! (BAD)  Also 13.9 secs..

This sample is too critical for mp3 encoding anyway.



Whoever gets fatigued is very very lucky as you are preecho immune.

Impact of listening fatigue , during ABX test.

Reply #12
Whoever gets fatigued is very very lucky as you are preecho immune.

The fatigue that arises from relistening to a very short segment (to obtain say 8 or more consecutive correct ABX results for that segment) doesn't stop pre-echo or other artifacts from being heard at the first hearing of the segment.

Impact of listening fatigue , during ABX test.

Reply #13
The fatigue that arises from relistening to a very short segment (to obtain say 8 or more consecutive correct ABX results for that segment) doesn't stop pre-echo or other artifacts from being heard at the first hearing of the segment.

Correct. When listening to very tiring items (such as castanets in a loop), you have to come to a conclusion quickly. And don't think you can postpone such fatigue by turning up the volume. From some investigations at work I can tell you it's counterproductive. The louder you listen, the shorter your test sessions should be. I think a good rule of thumb is:

1. At normal "conversational" listening level, don't do more than 30 minutes without pause.
2. For each 6 dB above normal listening level, reduce the test duration by a factor of 2.

Chris
If I don't reply to your reply, it means I agree with you.

Impact of listening fatigue , during ABX test.

Reply #14
Quote
The section of the files from 22.4 sec to 23.0 sec sounded obviously deeper
for my hearing with the flac version compared with the mp3.

Thanks for the indication, will try to AB-X  the section later.

Any luck with this?

Impact of listening fatigue , during ABX test.

Reply #15
@greynol
Currently I only have with me  my laptop, and Senn IE7  IEMs ; and I don't know  if that's good enough for this  ABX Test.
I  tried all the sections suggested on this thread before , and didn't find any "easy" to ABX.

I've just done now a new ABX test with the 22.4 sec - 23.0 sec section.
Normal volume level, and relatively  big pauses (at least at the end) .  I've removed and put again my IEMs,
in some steps, because I  wanted to move ( so the seal of my IEMs might not be  the same).

Here's the result:
Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.8
2009/10/14 23:48:03

File A: E:\carpenter\Carpenter_128kps_cbr.mp3
File B: E:\carpenter\Carpenter_FLAC.flac

23:48:03 : Test started.
23:53:47 : 01/01  50.0%
23:54:06 : 02/02  25.0%
23:56:44 : 03/03  12.5%
23:59:30 : 04/04  6.3%
00:04:43 : 05/05  3.1%
00:06:34 : 06/06  1.6%
00:13:17 : 06/07  6.3%
00:19:35 : 06/08  14.5%
00:26:24 : 07/09  9.0%
00:36:43 : 07/10  17.2%
00:36:47 : Test finished.

----------
Total: 7/10 (17.2%)


I have also a  desktop pc, with a better sound card, and full sized headphone. Unfortunately they are in an other town,
and I can't  use them for this test before a while.

Impact of listening fatigue , during ABX test.

Reply #16
During an AB-X test, you don't necessarily compare a lossy version , with a
lossless version.  Example: two versions encoded with different lossy codec.
But yes, if you just want to listen to the most "faithful" version, you might
just listen at the source, and not bother with any lossy codec.


I'm curious, what useful information do you expect to derive by from the knowledge that two non-identical lossy versions sound different from one another?  I've been thinking for a few minutes and haven't thought of one yet.

Impact of listening fatigue , during ABX test.

Reply #17
Once I found a good spot I didn't find this sample all that terribly difficult to ABX.  I typically run 16 trials when differences are subtle as they are with this sample.  With casual listening I would probably not notice.

Code: [Select]
foo_abx 1.3.1 report
foobar2000 v0.9.4.2
2009/10/14 19:11:43

File A: C:\New Folder (2)\Carpenter_FLAC.flac
File B: C:\New Folder (2)\Carpenter_128kps_cbr.mp3

19:11:43 : Test started.
19:16:07 : 00/01  100.0%
19:16:19 : 01/02  75.0%
19:16:25 : 02/03  50.0%
19:16:29 : 03/04  31.3%
19:16:41 : 03/05  50.0%
19:18:15 : 04/06  34.4%
19:18:24 : 05/07  22.7%
19:18:45 : 06/08  14.5%
19:19:11 : 07/09  9.0%
19:19:21 : 08/10  5.5%
19:19:27 : 09/11  3.3%
19:19:33 : 10/12  1.9%
19:19:46 : 11/13  1.1%
19:19:53 : 12/14  0.6%
19:20:01 : 13/15  0.4%
19:20:09 : 14/16  0.2%
19:20:16 : Test finished.

 ----------
Total: 14/16 (0.2%)
Fatigue was not an issue.

I find it curious that you'd choose 128kbit CBR with a sample like this.  I'll be the first to admit that I don't have golden ears and with my music (hard rock) I use -V3 and have been very satisfied.  I can often ABX -V5 and sometimes -V4 as well.  Because these settings typically produce bitrates that are higher than 128kbits with my music and I have the room on my DAP, I wouldn't even consider 128kbit CBR.  This isn't to say that there aren't plenty of samples out there that would fool me at 128kbits (VBR or even CBR).  This just isn't one of them, not when my interest is to achieve transparent results with attentive listening.

Another point worth mentioning is that despite common belief, good equipment is generally not necessary in being able to more easily identify lossy artifacts.  In fact it's often the other way around since psychoacoustic models typically assume playback with a flat frequency response.

Based on the position you've taken on this forum to question the validity of ABX testing, I have to say that I'm a bit disappointed that you've decided to set the bar this low.

Oh, and the testing was done at a normal listening levels which isn't particularly loud (I can hear up to 17kHz and would like to keep it this way).

Impact of listening fatigue , during ABX test.

Reply #18
@Tahnru,
Quote
I'm curious, what useful information do you expect to derive by from the knowledge that two non-identical lossy versions sound different from one another?

Hum ok, I won't tell you stories .
Once I compared a cd-xa version vs microsoft adpcm version  via ABX  test ,
and I found there was more background  noise in the microsoft adpcm version.
http://www.hydrogenaudio.org/forums/index....c=74096&hl=

@graynol
Quote
I find it curious that you'd choose 128kbit CBR with a sample like this.

This sample was  enough harsh to my ears, to the point that I  was wondering "how you could you manage
to abx that ?".    I  used an enough low bitrate, to see if listening fatigue is a true issue here.
If people are having difficulties on an obvious example, then this might come from listening fatigue.

Quote
Fatigue was not an issue.

Well, perhaps it's more an issue if you raise too much the volume .
Also if you already know to which section to look at, this makes things easier.

Quote
Another point worth mentioning is that despite common belief, good equipment is generally not necessary in being able to more easily identify lossy artifacts. In fact it's often the other way around since psychoacoustic models typically assume playback with a flat frequency response.

Ok, will try with very cheap headphones an other time.

Quote
I have to say that I'm a bit disappointed that you've decided to set the bar this low.

Sorry I  disappointed you. I  thought this sample was valid for testing listening fatigue.
My reasoning is that when you ears are getting tired, the bitrate do not matter much (under a certain limit).
But I'd like to tell that I'm not  experimented in ABX tests, and that I don't have a big knowledge of mp3 artifacts.
People that are familiar with mp3 artifacts are probably expecting a harder comparison.

Quote
I can hear up to 17kHz and would like to keep it this way.

I  don't know what is my limit.  Perhaps, the listening fatigue  with high pitched sounds depends of this limit.


Impact of listening fatigue , during ABX test.

Reply #20
@Tahnru,
Quote
I'm curious, what useful information do you expect to derive by from the knowledge that two non-identical lossy versions sound different from one another?

Hum ok, I won't tell you stories .
Once I compared a cd-xa version vs microsoft adpcm version  via ABX  test ,
and I found there was more background  noise in the microsoft adpcm version.
http://www.hydrogenaudio.org/forums/index....c=74096&hl=

The point is that ABX is the wrong tool for this kind of test, not that this is not something worthy of testing. Where you are testing preferences among two or more imperfect encodes you want to use something like ABC/HR.

Impact of listening fatigue , during ABX test.

Reply #21
Hum, ok,
I've redone the test with the speakers of my laptop.
It was more comfortable than my IEMs , and then I  was more relaxed/concentrated while doing the ABX  test.

Also I  didn't find that the section  22.4 sec to 23.0 sec was the best to spot differences,
so I  changed it to 15.5 sec to 16 sec , following shadowking suggestion that it's very bad at 16 sec.

Here is the log, you can ignore all the attempts before last trial reset, or not.
But it's only in the last trial that I'm using the  15.5 sec to 16 sec, section.

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.8
2009/10/15 18:19:47

File A: C:\Users\mehdi\Desktop\carpenter\Carpenter_128kps_cbr.mp3
File B: C:\Users\mehdi\Desktop\carpenter\Carpenter_FLAC.flac

18:19:47 : Test started.
18:38:54 : 00/01  100.0%
18:39:06 : Trial reset.
18:39:48 : 01/01  50.0%
18:40:44 : 01/02  75.0%
18:43:37 : Trial reset.
18:55:29 : 01/01  50.0%
18:56:41 : 02/02  25.0%
18:58:29 : 03/03  12.5%
19:00:07 : 03/04  31.3%
19:06:17 : Trial reset.
19:11:58 : 01/01  50.0%
19:13:08 : 02/02  25.0%
19:14:10 : 03/03  12.5%
19:14:46 : 04/04  6.3%
19:16:17 : 05/05  3.1%
19:17:16 : 05/06  10.9%
19:21:55 : 06/07  6.3%
19:22:29 : 07/08  3.5%
19:23:15 : 08/09  2.0%
19:24:21 : 09/10  1.1%
19:26:07 : 10/11  0.6%
19:27:00 : Test finished.

----------
Total: 14/18 (1.5%)



Also, I'd like to add that it's a pity that the ABX  logs do not show  which sections are exactly used  at each steps.
Not sure if it matters, but I  used  Replaygain for the test. This is not shown in the log too.

@greynol
Quote
I sincerely hope you aren't boosting the level solely in an attempt to help you make out differences between the two samples.

I realized that raising too much the volume , makes actually  this ABX  test harder. Especially because of the high pitched sounds.

In my other mentioned test (ADPCM  vs CD-XA) I  got to raise the volume a lot to  find a difference .

@pdq
Quote
Where you are testing preferences among two or more imperfect encodes you want to use something like ABC/HR.

Will study that thing.

Impact of listening fatigue , during ABX test.

Reply #22
Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.8
2009/10/15 18:19:47

File A: C:\Users\mehdi\Desktop\carpenter\Carpenter_128kps_cbr.mp3
File B: C:\Users\mehdi\Desktop\carpenter\Carpenter_FLAC.flac

18:19:47 : Test started.
18:38:54 : 00/01  100.0%
18:39:06 : Trial reset.
18:39:48 : 01/01  50.0%
18:40:44 : 01/02  75.0%
18:43:37 : Trial reset.
18:55:29 : 01/01  50.0%
18:56:41 : 02/02  25.0%
18:58:29 : 03/03  12.5%
19:00:07 : 03/04  31.3%
19:06:17 : Trial reset.
19:11:58 : 01/01  50.0%
19:13:08 : 02/02  25.0%
19:14:10 : 03/03  12.5%
19:14:46 : 04/04  6.3%
19:16:17 : 05/05  3.1%
19:17:16 : 05/06  10.9%
19:21:55 : 06/07  6.3%
19:22:29 : 07/08  3.5%
19:23:15 : 08/09  2.0%
19:24:21 : 09/10  1.1%
19:26:07 : 10/11  0.6%
19:27:00 : Test finished.

----------
Total: 14/18 (1.5%)



And this EXACT THING is why I cannot stand that the Foobar ABX component shows the progress of the test while it is occurring.  It seems that it only serves to trap newbies who think they have to "pass" an ABX test, retesting over and over again until they get their desired result.

Don't take me wrong, EBM, I appreciate your honesty here.  Thank you.

I see that pdq has already mentioned ABC/HR.

Impact of listening fatigue , during ABX test.

Reply #23
And this EXACT THING is why I cannot stand that the Foobar ABX component shows the progress of the test while it is occurring.  It seems that it only serves to trap newbies who think they have to "pass" an ABX test, retesting over and over again until they get their desired result.



But there is a 'hide results' option that can keep the subject's scores secret until the test is stopped.  It's a checkbox in the ABX comparator popup window.


Impact of listening fatigue , during ABX test.

Reply #24
I've seen the hide results checkbox.  Judging by the various new user posts that have been cropping up, I expect that said checkbox is ignored by the vast majority of users.

I've pointed out my annoyance at a lousy checkbox as a personal shortcoming of mine in the past.  That hasn't changed.  In my mind, its a key facilitating factor for the random drop-in users producing bad tests and erroneously interpreted test results.  That and a complete lack of understanding of blind testing.

Really, the utility of the "hide results" box is a topic for a separate discussion.  I'll be done with it here.