So here is my plan:

- He has chosen two cds he owns and loves, and from them, two songs he knows very well. Jazz music, by the way.

- I have taken these 2 songs, ripped them using EAC Test&Copy, absolutely no ripping problems.

- Then, I have saved 30 seconds out of each of them in separate files.

- I have encoded both songs and both 30 second clips with Lame 3.90.3 --alt-preset standard, and with Nero 6.3 96Kbps CBR, Highest quality setting (sort of an anchor).

- Decoded them with Foobar2000, Dither disabled.

- Then using Wavegain, made sure they all had the same volume as the original, with Light dither.

- Now I will burn them in a random order on a CD, and give the CD to him. He will try to tell me, out of them, which ones are coming from MP3, and which ones aren't. No time limit, no hardware influence (or same influence on all the tracks, that is).

He is not a computer expert, but a musician and composer. He also has studio experience in recording and editing with professional equipment.

Now, this doesn't try to be a scientific test of any kind. Just some amusement for both of us to see how much of the difference he believes to hear is really there. I am betting he will not be able to tell the aps mp3 tracks from the originals... and I have my doubts about the Nero ones, but well, that would be too much, I think.

So, have I screwed up at any point?

Yes, there is a problem with wavegain. MP3 encoding, as long as you don't use replaygain, or mp3gain, and no preset using scaling, don't change the volume.
Using wavegain will, at best not do anything, and at worst make the different version sound different because of minor adjustments. You should avoid this step.
Applying a volume correction must be done only with sources that have different volumes (Soundcards, CD Players, Speakers ...)

You should also scan the MP3 so as to be sure that they do not clip, which can be audible.
If they clip, reduce the volume of the original wav before the encoding, until it doesn't clip.

Also, why taking only 30 second samples ? You can burn the whole tracks and let him listen to the parts he wants to.

The number of samples is important. If you burn each of them once, the test won't tell anything. There should be at least 8 samples in total, plus 3 references included. But I don't know if more are needed, since three samples are used instead of two, like in an ABX test. I'd go with a total of 16 samples + 3 references. Just to be sure.

Last, if you want to prove MP3 to be transparent, it is essential to tell him only to give answers about which he is absolutely sure.
Otherwise, he will say that he was not concentrated enough, or that you CDR copy sounds so bad (because of high speed computer burning, you know ) that all details have been removed from the three versions anyway, and that they all sound much worse than his original...
Then no further testing will be possible, since you start a "sequencial ABCX" test, for which we don't know the probabilities of guessing.

If he is not sure of his answers, then the samples are almost transparent to him.
If he is sure of his answers, whatever he can say about possible flaws when you prove him wrong, you can always answer "but one minute ago, you pretended to have easily heard the difference... Thus you were wrong ?"

Why not let him have fun with the foobar abx comparator?

Mmmm...
Well, Wavegain was reporting a minor volume difference in the Nero-encoded tracks, but quite small. Now I am at my work, so I can't provide the actual numbers, but I will when I get home. The APS encodes, as far as I remember, were all very close to the original volume. Also, only the Nero-encoded tracks clipped very slightly after decoding. Not a big deal, they were meant to be easier to spot anyway... I think

That said, my intention was to do something like this:

Track 1. Song1 complete, from original wav.
Track 2. Song1 complete, from APS
Track 3. Song1 complete, from Nero, 96kbps CBR

Track 4. Song2 complete, from Nero, 96kbps CBR
Track 5. Song2 complete, from original wav.
Track 6. Song2 complete, from APS

Track 7. Song1 30sec segment, from APS
Track 8. Song1 30sec segment, from original wav.
Track 9. Song1 30sec segment, from Nero, 96kbps CBR

Track 10. Song2 30sec segment, from APS
Track 11. Song2 30sec segment, from Nero, 96kbps CBR
Track 12. Song2 30sec segment, from original wav.

This way, he can have easily accessible shorter clips to try, but also the full songs.
Now, if I understand what you said correctly, I should provide him also with the originals, telling him they are the originals? "Here is the original, here are three other versions of it: one comes from good mp3, one from bad mp3, and one is again the original, which is which?"

This is not enough. If the 96 kbps file is easy to spot, the challenge will be to recognize APS from original.
Since song 1, 2, complete and shortened can be obviously distinguished, he's left with four A/B choices.

If he get them all correct, the probability of guessing is 6 %
It can happen by chance even if he doesn't hear anything.

There should be at least three more APS / Original challenges, in order to get a probability inferior to 1%.

My idea about the original was just an ABX adaptation (ABCX here), so that it is easier for him to find which is which, and the p value is lowered using less samples.

First, 6 reference tracks, tell him what they all are :

Track 1. Song1, from original wav.
Track 2. Song1, from APS
Track 3. Song1, from Nero, 96kbps CBR

Track 4. Song2, from original wav.
Track 5. Song2, from APS
Track 6. Song2, from Nero, 96kbps CBR

Then the mystery tracks. Tell nothing. He must guess everything. Maybe they are all originals, maybe they are all MP3s, maybe some are originals... They are chosen completely randomly.

Track 7. Song2, from APS
Track 8. Song1, from Nero, 96kbps CBR
Track 9. Song1, from original wav
Track 10. Song2, from APS
Track 11. Song1, from APS
Track 12. Song1, from Nero, 96kbps CBR
Track 13. Song1, from APS
Track 14. Song2, from original wav
Track 15. Song1, from Nero, 96kbps CBR
Track 16. Song1, from APS

In order to generate this list, I tossed a coin.
I first made 7 choices between original and APS, as it will be the main challenge, and the probability of guessing must be <1%. Then for each of them, I tossed again a coin for song 1 or song 2 (this has no influence on the p value). Then I decided to include 3 96 kbps tracks, and I tossed a coin to choose where. Then I tossed again for song1/2 on these tracks.

If he guesses tracks 7 to 16 all right, I proves that he can hear a difference between original and APS with more than 99% confidence.
If you add wave gain, it won't be as clear, because he may hear the wav gain process and not the MP3 encoding.

Wavgain relies on an imperfect psychoacoustic model in order to evaluate the perceptual loudness. Comparing the gain value for an original and an MP3 can lead to slightly different values, even if there are no audible differences between them.
Applying these gain values can make the tracks sound different and measure the same, while they originally sounded the same and measured different.

I have just discovered that there is no "adoration" smiley in this forum...

I will follow your suggestion totally. I will discard WaveGain then, and just put the unprocessed Wav files in the Cd. Much easier, actually. (and I was thinking I was so cool and wise to be doing it so perfectly... )

In any case, this all leaves me wondering... What would then be the proper way to assess that two Wav files are of the same loudness? (for example, I decrease the volume of a wav file, encode it, decode it, and then want it to sound "as loud as" the original used to sound)

Thanks!!!

I agree with Pio2001's point about wavgain - I've said it repeatedly myself in respect to foobar ABX.


What is the point of the 96kbps file? You don't really need an "anchor" for a "can you hear the difference?" type test - only for a "grade these samples" test. You're just wasting his time and confusing the issue.

It's better just to give him the reference samples (i.e. original and coded for each song), and then 8 tracks for each song. Each track can be original or mp3, decided by tossing a coin (or similar random number generator) and he must say which is which.

I'd make two discs.

CDR 1:
1. original
2. coded
3-10 random choice from the above two!

CDR 2:
same, but for the other song


btw, I wouldn't be so sure that ABXing preset standard (which, in effect, is what he will be doing) is so impossible. Most people can do it with one or two problem samples, some people can do it with ten or more samples. Exceptional people can do it with much of their music collection.

It depends on his ears, and what tracks he chooses. That said, many "audiophiles" think they are more critical than they really are - if they haven't experienced or learnt to hear coding artefacts, even "experienced" listeners can miss quite obvious problems.

Be sure to let us know how you get on!

Cheers,
David.

The trivial answer is that you reverse the change that you know you made!

For possible unknown changes (which is what I think you meant), see my post in this thread:

http://www.hydrogenaudio.org/forums/index....howtopic=20694&

Cheers,
David.

Well, the point of the 96kbps file comes from the point of the whole thing actually...

I am not trying to fully prove the transparency of --alt-preset standard, or the transparency of mp3 when done properly (I would have gone for --alt-preset insane in that case). I am actually quite sure he will not detect the APS mp3 tracks. He knows his stuff regarding music, no doubt about it, but he is close to 50, his favourite instrument is drums, and he is also not trained in detecting mp3 artifacts. Very unlikely that he is a "Golden Ear".

The goal I am looking for is more "educational". When this whole idea started, we were talking about digital vs. analog, possibilites of either of them, blabla, yaddayadda... Then he mentioned he had tried some mp3 encoding with the software that came with his new SB Audigy2 (one of the external USB models), and found that it sounded less "warm", less "dynamic", "worse sound staging", and other similar totally subjective words...
So I want him to learn that mp3, when done properly, can sound very fine indeed. I think this will succeed even if he manages to score 100%. And also that, when done carelessly, can sound pretty crappy. And I think he will feel better with himself knowing that he can at least detect something... (if he doesn't detect the 96kbps files, well...)

Here is me, Paladin of Lame-ness!

Also, time-alignment, after reading the post in that thread 2Bdecided mentioned, might be an issue with the Nero files... I have noticed that FB nicely decoded the APS files to exactly the same size as the original, so I think that one is safe (I had heard often that it would be the case, but verifying it is always nice ). Should I worry about it?

Not much of an issue in files meant to be "not-so-nice" anyway... And I doubt he will check that anyway, because the plan is to use standalone players, and not a computer.

(I think part of the cause for the differences he heard between cd-audio and mp3 when he tested was that he was comparing the cd-audio going from a standalone through the sound card, with the wave-output mp3 sound... so using a standalone and decoded mp3s avoids this issue)

I intend setting up something similar to that. However, I still don't know what to do about the volume issue (I read all the posts here + the linked ones carefully). How do I find out whether a song from the CD and mp3 has a different volume? Couldn't I just rip some songs as wave and 'normalize' them (ie. CDex allows that). I would use a volume, let's say 90 to make sure that neither the wav file nor the resulting mp3 file will clip. Both files would have the same volume, right?

I'm asking this, because I always thought that LAME automatically used 89db as a 'reference' volume and therefore would lower the volume automatically...

thanks for your help

Apparently, if you use a LAME preset that doesn't include internal "scaling", then it is not an issue, unless the resulting MP3 file clips.

You should be able to check for clipping with mp3gain, and then, what you should do, is add an extra scale parameter to the encoding command line in LAME, and re-encode the original wav. Check again the resulting MP3, adjust the scale again, and so on, until the resulting MP3 file doesn't clip.

But, if the resulting MP3 file doesn't clip right from the start, and the LAME preset doesn't include scale, then the resulting MP3 file should have the same volume level as the original wav file.

Now, regarding my test, I will update the thread as soon as I get some results.

Unfortunately, I could not include all the tracks in 1 CD, and I had to discard some. I hope I did it correctly, but I fear that giving him 2 CDs would have scared off my friend...

So well, I'll let you know how it goes, even if it is not statistically perfect...

all right, so what is the setting for --preset standard (LAME 3.96) that doesn't change the volume?



why would I wanna add a scale to the LAME command line? I mean, the volume change has to be applied to the wav file as well. So I think it's better to normalize the wav file (while ripping). After that, I use --preset standard that doesn't alter the volume. If mp3 gain ´doesn't report any clipping I take the file. Ottherwise I ripp it again and choose a lower volume for the wav file.

Anything wrong with that? Also, what is the command line for --preset standard that doesn't alter the volume?

Sorry, true, my mistake: you must apply the volume change to the original Wav file...

I had a temporary brain shortcut...

As far as I know, --preset standard doesn't do anything to the volume.

In any case, if you want to be sure, just add --scale 1, which will override the scale setting that the preset may have (if any). I believe, but can't verify at the moment, that the output of --preset standard will be the same with and without that extra parameter.

Ok, I read this whole thread.........

So How did the test go?????

Sorry, I still don't have results, but don't worry, this kind of test takes time.

I know I will get them, and you can be sure I will post them! In the mean time, patience is the only thing I can advice

my friend was able to ABX every song a 100%! I had 4 songs in total with 8 samples each, I told him the source of two of them. He is not a professional nor does he have professional equipment. He messed around a bit with the treble and said that the voices sounded kind of disorted...the tracks were chosen randomly but were all on the same album...

That will violate all assumptions the psychoacoustic codecs made. (Especially if he turned it up a lot)

btw. I meant "treble"...anyway, this is what I told him too. However, I think music sounds better with a bit more treble...so what assumptions will the encoder make? Some players may add more treble than others...wouldn't a mp3 file sound worse on those systems then since the origianl CD allows a higher treble?

It depends on what you're doing. If you're putting the treble up because your playback system doesn't have a lot of it, effectively going more towards a balanced playback, you're fine (and the codecs wont have problems). If you excessively add it to skew the sound, the mp3 will become more easily distinguishable. I don't know if 'worse' is the right word since you are already moving away from the original soundscape...what defines whats good and bad then?