I've been using Musepack for a few months now. Today I heard my first clearly audible artifact (at least with the setup I was using) without trying to use some of the well known hard to encode tracks.

Track: Madonna: Frozen (from album 'Ray of Light')
Encoded with: Klemm's latest fast encoder using extreme profile
Ripped with: latest EAC (read offset calibrated) in secure mode off a pristine original disc
Decoded with: Latest official decode from Andree's home page
Burned to: TDK Reflex Ultra 24x certified low BLER media
Burned with: EAC (write offset calibrated) with Plextor PX-W1210S
Compared to: Original track from original CD (Madonna: Frozen)
Played with: Acapella Campanile horn speakers with plasma tweeters, AudioNet Art V2 CD-player with 'fast' emphasis, Audionet Amp I + Audionet Pre 1 G2 + Siltech SQ-88 G3 cables

Now, before you say anything, I listed the equipment not to boast (hell, I don't even own it), but because it is highly accurate - bringing out even the most smallest details. In my opinion the speakers best even electrostatics in some respect (especially in transient and high frequency response).

So, what is the artifact?

Probably not anything new to you, but new to me personally.

When Madonna sings a sibilant (a hissing sound) the hiss ATTACKS you forward from the speakers in a way that *almost* hurts the ear. It is very unbalanced - it is impossible NOT to hear it. Original track does not have this forward sibilant. All three listeners reported the same thing.

I can't hear it on my Grado SR-60 headphones nor on my Sennheiser SD-590 (well, maybe if I imagine hard enough) when connected to my lowly soundcard (Gametheater XP). But on the above high end setup, the artifact is very much real.

The other people on the listening sofa (dedicated hard core high end listeners) claimed to hear phase anomalies and "pumping", although they couldn't desribe it better with the very short 2 minute listening we had.

Now, I'm not a "golden ear" nor a high end person myself. I'm just interested in audio encoders as a lay person.

What I would like to know is that if I keep harassing my audiophile friends with more hard to encode samples and make them listen to mpc encoded tracks with their high end gear, is anybody interested in the results?

If we find consistent artifacts that are hard to pick up on ordinary gear, but easy to produce on high end equipment, is this of any use to the encoder developers?

All sane comments welcome.

Before anyone starts: I don't have the skill nor the inclination to start random controlled, statistically validated and methodologically sound double blind listening tests with over 1000+ participants in it

All I can offer is a couple of highly trained audiophiles with 15+ years of dedicated listening experience and a very transparent audio reproduction gear.

If that isn't useful to anyone, then I won't offer it


Cheers,
Halcyon

PS Just as a comparison, mp3 with dibrom's latest tweaks (for c. 192 abr) was in my opinion worse. Analog mixing table noise was distorted, transients had a slight attack before them and there was some sort of slight "swish-swash" type of repeating thing going on. All this on the same "Frozen" track. I'm not very adept at describing these things yet, my apologies.

It's not complicated to perform a blind test. You can get your friends to help you by having them repeatedly play either the original or the encoded file without your knowledge, while you write down what you think the file is. Repeat until you have performed 16 trials. If you chose properly 12 or more times out of 16, the probability that your results were due to random guessing is less than 5%.

You can also perform such a test by using the randomize function from within Winamp. However, the easiest way to perform blind tests is to download an ABX utility, such as PC-ABX (http://www.pcabx.com/).

Reporting reliable results is the first step toward improving a codec. The next step is making the test sample available.

ff123

i think tests would be interesting if not helpful....

i always thought mpc provided bes quality compared to others... especially in the higher ranges and other stuff... phase info as well..

whatever just some random thoughts....

ff123,

thanks for the good comments. A couple of points:

- 16 tests is by no means a statistically useful number (even if I would get 16/16 same results) in this context

- Repeatedly hearing the same sample over and over again does not take into account the accomodation of hearing (i.e. changing of the action potential of your synapses). This results in skewing of the test data due to listener accomodation to the new signal. This is perfectly documented in neurological research on sensory excitation.

- I don't consider basic (single) blind tests to be solve all be all. The test would have to be (like I said earlier):

1) Random controlled (not just the two of my friends, but a selection of testers)

2) Methodologically useful (i.e. the same test runs repeatedly on different days when people's hearing is different, measuring most test condition context variables, etc.)

3) Statistically validly analyzed (this is way out of my league)

4) DOUBLEblind (makes it much more difficult to perform the test, especially without introducing what are often second rate audio components - like computers - into the audio reproduction chain)

This is way too hard to set up. I know - I've read a few Lancet, New England Journal of Medicine and Journal of experimental Psychology articles. It is HARD work to do properly.

Thus, I'm not willing to go along with that.

Still, if anyone is interested in *potential* clues on the basis of which to start setting up their *own* tests, then I believe that I may be able to offer a reasonable ratio of good clues vs red herrings through my own proposed setup. A Reasonable ration here means that it is statistically significantly better than giving clues on what to test by simply guessing.

I totally agree on what you say later on:

"Reporting reliable results is the first step toward improving a codec. The next step is making the test sample available."

Personally I'm not willing/skilled enough to do the test properly (as outlined above), but I can provide clues for other people on what to test. I can also provide the samples.

Would this be useful?

Please, I don't want to start a flame war - even though I don't necessarily agree with all points brought forward. I come in peace

I think even basic single blind tests are useful (to some extent), but much too hard for me to do and too cumbersome to impose on my friends.

Best regards,
Halcyon

Incorrect. With an ABX test, 12 out of 16 correct responses yields less than a 0.05 probability that the results came from random guessing. At 15 out of 16 correct, the probability of getting those results from guessing are less than 1 in 1000. You can verify this for yourself by writing a short simulation program.



The ABX test is protecting against type 1 errors (false identification of differences). That is, if you get 15 out of 16 correct, you can be sure (with 99.9% confidence) that the results were not from guessing. So even if the ear gets less sensitive with repeated listenings (which seems to be true in my own experience at least), the ABX results are still valid.



It depends on what you are trying to test for. If you want to find out whether artifacts from an encoded sample are audible to the general population, then yes, you'd have to sample randomly. However, if you want to determine whether you, personally, are really hearing artifacts, then ABX will do the trick.



Again, consider what is trying to be measured: are you personally hearing a difference, and how can you be sure that guessing is not a factor?



ABX can be analyzed by simulation. However, it is easier to just look at the (approximate) p-values given from the PC-ABX program.



I presume then that you are listening to MPC files which have first been decoded and burned to CD, so you can listen to them through your preferred CD player? If so, yes, that does make it harder to perform a double-blind test (ABX tests are the equivalent of double-blind). But even single-blind test results would be useful. We are not talking about bringing a new drug to market. We are talking about making sure that you are reliably hearing audio differences.



It is not hard. For example, if your CD player had a remote and a randomize feature, you could design your own "methodologically sound, statistically valid" experiment to find out if you are reliably hearing differences from the original on your preferred audio setup. Of course, it's much easier if you had a computer setup. There are a number of professional sound cards which you could use, as well as an outboard D/A converter, and you could place the PC in another room. In principle, then, it's not difficult to perform these tests. It's a matter of logistics.



If this means that you're going to offer your sighted (i.e., non-blind) opinions, you can't say that will be statistically significant. It means that skeptical people like me will have to slowly develop trust in your opinions over time, after I read what types of artifacts you claim to hear in different types of music, and correlate those opinions with those of other listeners who I already trust.

ff123

And here we have yet another MPC artifact reported without a sample or reliable test.

Anyway, this is getting too boring to rant about it again.

The album is wellknown and I'm certain someone here has it. Please list the timeindex where you hear this 'artifact'. Thank you.

--
GCP

Halcyon, try the sample with disabled ans. Download:
http://sivut.koti.soon.fi/julaak/mppenc33_10.exe

This is the latest beta version before 1.7.9 and still has the ans switchable.
To disable adaptive noise shaping, use -ans 0.

If that doesn't help, make sure it's not mid/side coding problem, by disabling it and using stereo instead.

Ah, you might be right. I was also including the other side of burden of proof (i.e. that the effect must be heard by more than one person, which would have made it much harder). Thus, I was speaking about the number of people not the number of tests (e.g. not 1 person x16 tests, but say, 100 people x16 tests).



Just a minor correction: it's not (AFAIK) the 'ear' that gets less sensitive (unless you destroy your hearing), but it is the "neurological connections pool" (for the lack of a better word) that gets less sensitive within your hearing center of your cortex.

But with audio accomodation you will not necessareily get those 15/16 right, even if the difference is there. This is the problem. The faster the accomodation happens (usually when differences in action potential are smaller) the faster the test becomes invalid (i.e. it doesn't accurately measure what it is supposed to measure anymore).



Indeed. My follow-up question would be. Let's assume (for the sake of discussion) that a friend of mine would be able to distinguish an artifact in ABX testing 12/16, but only that one person. None of you people here with your own gear would be able to reproduce it/hear it. Would it still be useful?

Or would you just assume that I'm making things up and your hearing/gear/setup is as good as the one I'm offering? How many people and what kind of statistical confidence would I have to achieve, before you'd trust the results, even if you couldn't hear it yourself?

This is a theoretical question and doesn't imply/assume inferiority of anyone's hearing/gear, so please do not get offended. It's not my aim. I'm just playing the devil's advocate here.

This is also one of the reasons why I'm not willing to play the ABX game. I know I'd be stuck neck deep in ABX tests with a few dozen of people for the next umpteenth months just to make people here happy. On a artifact that I can so obviously distinguish myself that I don't for a second doubt that I need an ABX test on myself to prove it to myself. Nothing wrong with that though. I'd be quite as skeptical myself



Yes.



Well, with this I don't agree with. I think that any number of listening tests I've seen or read about fail miserably in the methodology department. They measure "something" for sure and some of them may even somewhat accurately measure that "something", but most testers don't have a faintest clue of what that "something" is. Even though I understand this, I'm by no means better and I don't think I could pull off any better a test (although I could imagine parts of it



Indeed it is. Unfortunately I have neither the golden hearing myself nor the access to precious equipment (be that high end audio or professional sound card gear). On the other hand, I can't impose too much of a testing procedure on my friends (who *might* have a better accomodated hearing and certainly better set up

This was my original problem and why I asked whether my anecdotal findings would be of interest to anyone.

If not, I fully understand that and I thank you for the academically interesting discussion and correcting my mistakes



I know. I should have written "I believe to be statistically..."

I know anecdotal evidence would require huge amounts of trust and that my friends is hard to achieve with methodologically correct testing (at least in my book it is

Thanks to JohnV as well, I'll probably try that if I get to hear the same setup again.

Best regards,
Halcyon

PS Call me delusional if you will, but I don't doubt hearing the artifact I originally mentioned with sibilants (I could only hear this one, the "golden ears" *claimed* to hear several artifacts). Personally I would like to know that where do people themselves draw the line on proving the artifact's existence to yourself and to others? Do you organize an ABX test for yourself on a test with orig pcm vs 32kbps mp3? Hell no, the artifacts are so obvious that you don't need to. For me the level of phenomenological obviousness was personally at such a high level that I don't think I need to prove it to *myself* under ABX. Then again, you have absolutely no reason to believe me either, especially if you can't hear it yourself...

Halcyon

Why are you ranting against those that simply want you to apply scientific rigour?



Precisely the reason why you indeed need to apply ABX. The mind of the 'believer' can see and hear anything that is in the mind's eye!

First of all, if that is considered ranting, I apologize. Second, if the answer is not evident from my writing, then I have miserably failed at my attempts at communicate my POV. I don't think I can further influence anyone to understand my position. My skills in English are lacking.

And yes, I'm patently aware of the fact why ABX needs to be applied for data that is to be considered scientifically useful in this context (although it alone is not a sole guarantee of the trustworthiness of the results).

Still, you fail to understand the signifigance of my Post Scriptum. Where each one draws a aline is a personal decision and I've already drawn mine due to reasons given

Again, no offence meant to anyone. I appreciate all comments so far.

Best regards,
Halcyon

I can't speak for developers whether your remarks on artifacts are useful and where to send them.
Just a suggestion: burn also the wave file to the same (or similar) CD-R to rule out that the ripping/burning process and/or playing from CD-R might influence playback quality.

If you like to experiment with encoder settings try: -xtreme -nmt xx (start with xx=10 and maybe 12, 14 to see if it changes things)

Edit: An other thing that comes to mind: did you use Dithering while decoding? If so maybe try without.
Edit2: Have you checked if the decoded file clips? This is a common problem in decoding lossy compressed files, not MPC specific.


BTW. Let's not forget that MPC is also a lossy compression, it's not perfect, just very darn good.

Halcyon,

In the amount of time you've spent typing on these message boards, you could have done an ABX test!

If you're worried about subtle differences vanishing as you listen to the track again and again and again to the point of tedium, then simply do one trial per day. Listen to A, listen to B, listen to X, make your choice, and leave the next test for another day.


A good (curious) audio engineer is always keen to check anything that they think they hear via a blind test. Those audio engineers that aren't either haven't learnt the power of placeabo yet, or have learnt the power of the placeabo effect and are using it to try and sell something!


Try this with your friends: when they say "that sounds worse", play them the original, the coded version, and then the original again (without announcing which is which). Ask them to pick the odd one out. You must randomise the order of the tracks (original orginal coded, coded original original), and ask them whether A, B, or C was the odd one out.

I'd be interested in the results.

David.
http://www.David.Robinson.org/

P.S. if you read through other comments of mine, you'll find that I'm not a total cynic. I believe there are people who can ABX very very good coded audio - but I haven't met them yet. However, I have identified a change in phase distortion in a blind listening test myself, so I'm quite willing to believe that people can hear what everyone says is "inaudible". I'll be very interested to hear how you get on.

Yes, this is indeed what I already did. I had the original PCM, MP3 with latest lame 3.90a compile with dibrom c. 192 abr flags and mpegplus/musepack at xtreme profile.

If I get to organize another listening sesssion with the same friends, I'll try and improve the system with randomization. I was thinking that I could hide the CD-player front panel, take away remote control, put on RANDOM PLAY and then direct a video camera towards the cd player panel to record the order of tracks played. Then even I wouldn't know the order.

This would be a very poor mans blind test and I'm not sure they are willing to along with this

A lot of people I know would take it as an insult if I asked them to prove their listening ability under blind testing conditions. This means the the ethical considerations of blind testing also step into the equation. I can ask nicely, but I can't insist



Do you think it might be worthwhile to combine the switches you suggest with the changes that JohnV suggested above (ANS switch)?



I source was 44.1/16 and so was the target pcm after decoding. I assumed that without any flags the standard decoder doesn't dither under these circumstances, unless specifically told to do so with the -dither flag (?).



No I did not. The effect didn't sound like transient clipping though, but I will check. Thanks.

A simple question: would using the -noclip flag be a good idea just in case? Would this result in compression of the dynamic range?



Indeed! I make no complaints about the codec or the authors. I think they've achieve something really extraordinary and I commend them for their work.



Now, you are getting one step closer to a properly set up listening test that would cut the mustard in the clinical psychology journals.

This is also the reason why I don't do the above: I do it for fun (on the side to other things) and not a as a full time job

Furthermore, I'm not an audio engineer and I'm somewhat aware of the tribulations of various different blind testing methodologies, but I'm not willing to embark on a 6 month research journey. Maybe I can find something quick and dirty that still satisfies the minimum level of requirements for some readers in this forum without not becoming a full time job for myself

Cheers,
Halcyon

PS Again, thanks for all the suggestions. You've already convinced me to at least try and do a quick&dirty blind listening. You can be sure that if you don't hear from me again, then I wasn't able to convince my listeners to go along.

If the test yields positive results (i.e., 15 of 16 results), this means you have positively identified differences, in spite of decreasing sensitivity with repeated listenings. If you get negative results, it is inconclusive. ABX testing does not protect against type 2 errors (false acceptance that there are no differences). The bottom line is that if you get enough correct on an ABX test, you can be sure (within a certain confidence) that you really heard a difference! The "methodology is clinically sound," to use high-falutin words which for some reason really irritate me. Statistics isn't magic or meant to be practiced exclusively by white-coat priests in a laboratory. It's based on common sense.



Yes it would be good enough. I personally prefer to use 14 of 16 correct (p < 0.01) before I claim to hear a difference. ABX is useful precisely because it is possible to generate statistical useful results with just one person.



The most important ABX test you need to run is the first one, to establish your (or your friend's) credibility.

ff123

Unfortunately I don't have time to respond more in depth to some of these comments but I'll just add a few things:

Halcyon:

You are making this far more complex than it has to be. If you hear a problem, perform an abx test (or multiple tests across whatever timespan you like if you feel that is necessary) and a clip.
There's little point in speculating on the worth of abx and the ethical issues (er...) and this and that, IMO.

Sorry if that sounds harsh, but I don't really see the point in going around and around on this topic. If you are sure you can hear a problem, then just provide a clip and some test results. If your friends claim they can hear a problem but they refuse to participate in blind testing, then I would discount their statements. This may seem dismissive, but if they do not see the worth in trying to scientifically prove they can hear a difference, then I don't see the point in trying to spend all kinds of time trying to fix problems that may not even be there, just for them.

As ff123 said, we aren't trying to bring a new drug to market here.. we are simply trying to substantiate some claims.. that's all. Don't make it out to be such a big deal

Whatever happens, if you are worried that people may not believe claims that are made.. I can assure that people will be that much less inclined to take notice if you or your friends do not bother to take part in abx testing.

Halcyon, can you provide a clip, or tell the position so that somebody else can make it available?

ff123 & Dibrom,

This has turned into a pissing match - which was not my original goal. My apologies for that. I meant no harm, but misrepresented my cause quite badly.

I thought it was possible to have a semi-academic discussion here, but I'm afraid that my skill in expressing these things in English is so bad that I only make other people angry without them understanding any of my arguments

I also got the answer to my questiion: anecdotal data is worth almost nothing here. I accept that and I'm even happy about it (no need to do more testing

Thus, I will conclude this discussion on my part. If I have time to do simple non-scientific AB or ABX tests, I'll report back. If you won't hear from me on this issue, I didn't have the time.

Thanks to you all for your comments.

Cheers,
Halcyon

PS JohnV, I'll report the position privately so that this discussion can conclude.

Yeah, well I believe it's quite probable that you heard something wrong. The description of distortion was quite exact, not something like "I think I hear loss of bass". Problem so far has been that no time position information or clip has been provided.
In my opinion IF you are absolutely sure you hear something wrong, even ABX isn't the most important thing for you to do. Most important thing would be to provide time position or test sample, so that others can check how serious it is and maybe can actually abx it. I think some people have been wondering that despite the long messages, you havent provided anything concrete yet.
This just happens too often nowadays. Somebody just says "I think I hear, that format x has loss of bass" for example, and disappears. No time position of track, no test clip, no blind listening tests, no nothing... So forgive us, we don't really mean to be rude but just want something conrete, anything at all...

If you post the position to me privately I'll put it here, but in my opinion you can just as well post it to this thread. I don't have the album, but I'm sure somebody has it.

Hrmm.. well I don't feel that way. My message was not meant to be offensive.



I believe I understand your arguments fine. I just think you are making much more of a big deal out of this than is necessary. What we are trying to do here is a fairly simple thing, it doesn't have to be extraordinarily complex and it doesn't have to be at the level of precision you would see in some very important lab test or something similar.



That's not entirely true. It's worth something as long as you have established yourself somehow. This isn't meant to be elitist, but without abx results, a clip, or something it's hard to go on face value of just "I hear a problem, etc, etc". As JohnV has said, we see this all the time now and probably 70% of the time there's nothing wrong at all. If you can just provide us with something to go on, chances are we'll believe what you have to say and we'll be interested in your opinion.



The thing is that they don't take long at all. As David already pointed out, in the length of time you spent reading and responding to some of these posts you could have easily performed an abx test and provided a clip.



Sure... but please don't take this matter personally because we are asking for some shred of evidence of some kind. It isn't meant to be that way.

I definitely won't

I was more afraid of the fact that others might find this discussion offensive and therefor I concluded it on my part.

And like I stated several times, the plea for more evidence is totally understandable and non-offensive. I should know - I'm usually the first one to demand additinal evidence myself

cheers,
Halcyon

PS I've started another thread, where I hopefully ask a less inflammatory question. See: http://www.hydrogenaudio.org/forums/showth...s=&threadid=427

I think I understood your arguments just fine.



Not quite true. I take anecdotal data with skepticism, meaning that I evaluate it in the context of all the other information that has been supplied. Like Johnv said, the description of what you heard wrong was specific, which is good. However, I don't know you from Adam, so how do I know that you aren't imagining that you're hearing something bad? How do I know that you haven't done something wrong in the encoding or decoding process? That's why we're asking for specific track numbers and time stamps, so that it might be possible to reproduce what you're reporting. A basic step in a scientific process.



Oh, ABX is scientific all right. You seem to be equating the extreme rigor of lab conditions with the word "scientific." Just because not all the conditions are controlled doesn't make it "non-scientific." ABX testing controls the most important variable of interest, and is certainly more scientific than leaving yourself open to the possibility that you may have only thought you were hearing something wrong.

ff123

On this I agree and I phrased my words badly again. Let's give it a rest. I'm not in a mood for a pissing contest. I won't touch your other arguments even though I don't agree with them

Respectfully,
Halcyon

Ok, Halcyon provided the clip and I put it available here
http://sivut.koti.soon.fi/julaak/frozen.zip

The artifact should be in the position of *S* (You only *S*ee,).

As Halcyon said he can't hear it with his computer/headphone system. But it was supposedly audible with the system described in Halcyon's first message.

LOL - the lyrics are perfect!!!
Dripping with irony.

I also can't hear anything wrong with the "s" sound or the sample in general using my computer/headphone system.

ff123

I can't pick any difference between the two samples on PC/headphone. Certainly, the sibilant 's' sound is highly reverbed and sharp with lots of early reflections from the studio reverb processor, but this is apparent on both samples.

I will do some listening in front of loudspeakers next.

"You only see what your eyes want to see." Madonna

Results of loudspeaker audition of two waves provided burnt onto Imation CD-RW using Nero. Equipment as detailed in alternative post started by Halcyon above.

Sorry, couldn't pick the reported artifact. As I said above, the first 'see' is quite sibilant, but this is so in both samples. Studio reverb programs with lots of early reflections and undamped HF do this quite nicely for effect. But as I say, both samples were the same.

2 listeners in room, the other listener to myself tried to pick the one with the reported artifact and scored 3/8. I couln't tell any difference anyway, so there was no point in doing an ABX (tell me if I'm wrong in this assumption).

It's darn subtle, but I believe he is correct:

Reading frozen_pcm_orig_eac_secure_calibrated.wav ...
Reading frozen_mpc_klemm_fast_0-90e_extreme.wav ...
2*16 bit 44100 Hz*4.394 sec
Listening X Vote for X:=B OK 1/1 1.00000
Listening X Vote for X:=B OK 2/2 2.00000 1.00 bit
Listening X Vote for X:=A - 2/3 1.00000 0.00 bit
Listening X Vote for X:=B OK 3/4 1.60000 0.22 bit
Listening X Vote for X:=A - 3/5 1.00000 0.00 bit
Listening X Vote for X:=B - 3/6 1.00000 0.00 bit
Listening X Vote for X:=A OK 4/7 1.00000 0.00 bit
Listening X Vote for X:=B OK 5/8 1.37634 0.06 bit
Listening X Vote for X:=A - 5/9 1.00000 0.00 bit
Listening X Vote for X:=B OK 6/10 1.32642 0.04 bit
Listening X Vote for X:=B OK 7/11 1.82206 0.08 bit
Listening X Vote for X:=A OK 8/12 2.57935 0.12 bit
Listening X Vote for X:=A OK 9/13 3.74748 0.15 bit
Listening X Vote for X:=B OK 10/14 5.56900 0.19 bit
Listening X Vote for X:=A OK 11/15 8.44101 0.21 bit
Listening X Vote for X:=B OK 12/16 13.0187 0.24 bit

I will try to get a stronger result after I had more sleep.

--
GCP

Listening to the clip on my headphones I hear no difference, but when I use my speakers and amp at high volumes with some EQ I can tell a difference, and scored 9 out of 12 on ABX.

Originally posted by TrNSZ


Hrmm.. 12/16 and 9/12 are not particularly promising results for clearly stating that there is an artifact here though IMO. I'd like to see a few scores which bring a higher percetange than 75% before it's accepted that there really is a perceivable difference.

Particularly in the case of only performing 12 trials, how many times in a row are you able to do this? Or how many are you able to descern correctly at 16 trials or higher?

No I haven't had a chance to listen to the clip yet myself either. Just thought I'd comment on the results so far.

Now I am back after doing the ABX tests. For the "lifeless" (I'd mentioned in previous threads) or in other words "loss of bass", I believe it's due to the DSP plugin for Winamp which mass the thing up. With the DFX 4.0 (a DSP plugin that I used) enabled, I do hear slight loss of bass (noticable for me and I can pick the MPC everytime I random play the original and the MPC in Winamp and that's the way I do my listening tests) for MPC but not for MP3 and AAC. Then I did the ABX tests as recommended by the experts (you know there are no DSP effects for the ABX software), and I get 7/16 for MPC, which means I couldn't differentiate MPC with the original. I'd done the test many times and I always get 7/16 so I believe MPC is transparent for me.

MPC encoder version I used: 1.7.9c
MPC decoder plugin for Winamp: 1.7.9f with "clipping prevention", "dithering", "eq controlled by winamp" disabled
ABX software version 1.74
Winamp 2.74 with EQ disabled (always disable that lousy EQ)

Ok, I did also some tests with frozen, although I believe my hearing ain't yet totally recovered from a concert from yesterday.

Anyway, couldn't abx it clearly, after some tries I got 12/16.

Well, then I encoded it with xtreme and disabled ANS. Interesting thing here is that it clearly changes the sound of total quantization noise of the sibilant (when listening inverse mixed signal). Other question is, whether the mpc then is audibly better. I'll try it tomorrow.

For VeryBlur: Yes, DSP plugins should never be used when testing lossy codecs because they can destroy the masking effect. However, I would be interested if you could do some tests using DSP plugin and ANS-disabled mpc. Reason for this is, that when disabling ANS, there's sometimes considerably less low frequency total quantization noise. Disabling ANS could help the sound with DSP-plugins (just a guess).

As far as I've understood disabling ANS can't hurt the quality in anyway. Only thing it can do when enabled, is that it allows a bit higher efficiency (leads to a bit lower bitrate) by "pushing and shaping" more quantization noise under the masking threshold. In principle this should be inaudible, but Beatles loved the sound of ANS-disabled MPC at ultra high bitrates, and obviously is able to abx between ANS and no-ANS.

Fresh in the morning:

Reading frozen_pcm_orig_eac_secure_calibrated.wav ...
Reading frozen_mpc_klemm_fast_0-90e_extreme.wav ...
2*16 bit 44100 Hz*4.394 sec
Listening X Vote for X:=A OK 1/1 1.00000
Listening X Vote for X:=B OK 2/2 2.00000 1.00 bit
Listening X Vote for X:=A - 2/3 1.00000 0.00 bit
Listening X Vote for X:=A OK 3/4 1.60000 0.22 bit
Listening X Vote for X:=B OK 4/5 2.66667 0.35 bit
Listening X Vote for X:=A - 4/6 1.45455 0.10 bit
Listening X Vote for X:=B OK 5/7 2.20690 0.19 bit
Listening X Vote for X:=A OK 6/8 3.45946 0.25 bit
Listening X Vote for X:=B - 6/9 1.96923 0.12 bit
Listening X Vote for X:=A OK 7/10 2.90909 0.17 bit
Listening X Vote for X:=B OK 8/11 4.41379 0.21 bit
Listening X Vote for X:=B OK 9/12 6.84950 0.25 bit
Listening X Vote for X:=A - 9/13 3.74748 0.15 bit
Listening X Vote for X:=A OK 10/14 5.56900 0.19 bit
Listening X Vote for X:=B OK 11/15 8.44101 0.21 bit
Listening X Vote for X:=B OK 12/16 13.0187 0.24 bit
Listening X Vote for X:=A OK 13/17 20.3908 0.27 bit
Listening X Vote for X:=B OK 14/18 32.3794 0.29 bit
Listening X Vote for X:=A - 14/19 15.7312 0.22 bit
Listening X Vote for X:=B OK 15/20 24.1607 0.24 bit

I hope this is 'beyond reasonable doubt' for everyone? (it's <= 2%)

MPC extreme is not transparent on this clip.

(I just did another, longer trial. 34/50 (<= 0.7%). It is there, but _hard_ to pick up. I will try on speakers too.)

--
GCP

Basic statistics lesson:

Ok's / Trials ?
9 12
1000000 iters, 73229 success, 7.322900% luck probability
(9 out of 12 is 92.7% confidence)

Ok's / Trials ?
12 16
1000000 iters, 38662 success, 3.866200% luck probability

(12 out of 16 is 96.2% confidence)

Ok's / Trials ?
15 20
1000000 iters, 20823 success, 2.082300% luck probability

(15 out of 20 is 97.8% confidence)

Ok's / Trials ?
34 50
1000000 iters, 7237 success, 0.723700% luck probability

(34 out of 50 is 99.3% confidence)

Note that if you try to ABX it 10 times it is quite possible to get one 9 out of 12 result even if there is no artifact there.

That's why there is no point in doing multiple tests and posting the 'best' result, unless it's with a very high confidence. (and that's why I wanted a 'stronger' result than 12/16 myself, as I did 3 trials yesterday)

The actual 'percentage' of correct answers is irrelevant. 60 out of 100 is [b]more significant than 7 out of 8

--
GCP

The question should be now, does disabling of ANS make it better.

Can you put up a clip with ANS disabled?

(I downloaded and used yours)

--
GCP

I'm not at home, but just download this:
http://sivut.koti.soon.fi/julaak/mppenc33_10.exe

You gotta use .mp+ extension with this, since this is the latest beta before 1.7.9, but should be very similar quality wise compared to latest encoder.
So, just use -xtreme -ans 0

I get 23/32 (<1%) with xtreme -ans 0 on that version on my first try.

I doesn't help at all. Actually, my initial impression was that it was worse.

Edit: Hmm, that's probably because I'm getting trained to the artifact. Still, I do not think there is a noticeable improvement.

--
GCP

Hmm, well since you are getting trained to it, could you try once more, this time ms-coding disabled.

If this still doesn't help, then it clearly needs higher nmt value.

I'd be interested in seeing this with a slightly higher nmt value also. I've run across a case or two before where there was a very slightly artifact right on the edge of perception, but that using -nmt 10 or so was enough to where I could no longer discern a difference. Granted this was on more difficult and transient heavy clips, but who knows...

Well, I at least commend him (Halcyon) for getting off his duff to bring this to our attention. Unfortunately, the mind can play some tricks on our sensations, so ABX tests are a requirement. Like JohnV said, I'd like to see Halcyon reencode the material with both ANS and joint stereo turned off. At that point, ABX tests of that vs. the original encode would be very helpful in determining if either ANS or js is the issue.

John

PS: A cursory listen w/ my Sennheiser HD 600 + Headroom Home amp revealed no artifact. I had the delayed cross-feed on my amp enabled to mimick more fully the effect of stereo speakers.

This certainly has to be a very subtle artifact: I've never seen a case of a trouble sample where so many trials were necessary by a sensitive listener to verify the existence of a problem.

For reporting best results of 16 trials, I like to use 14 correct (p < 0.01), although it is simpler to keep track of the total results like Garf did. However, if I can't get 14 correct on a 16 trial run, I've usually found that my results fluctuate quite wildly.

ff123

This is mainly because of how I do ABX tests: I tend to listen to each sample only 1 or 2 times and then follow my 'feeling'. I do not spend a great deal of time making sure I am listening to A or B. I make more mistakes, but I can also do more trials, so it ends up being the same.

I do it this way because I find it to be less stressfull. You will notice that in your listening tests I tend to make mistakes even when ABX'ing Xing files for example.

The artifact is indeed quite subtle though. I would not pick it up during casual listening.

In the S sound, both samples have reverbration in the hissing sound, but in the encoded sample it comes out stronger and the noise is more spread out. (I hope someone gets what I mean)

--
GCP

I would have described what I thought I heard in a similar way, although I might have have said that I thought the encoded S sounded heavier. I tried maybe half a dozen runs of 16, and got 13 correct on my highest run, but I didn't keep track of the total. I also got a run of 9 correct, I think. It's not something I would usually report as a definite artifact.

ff123

So what are we saying here? That there is the probability that there is an artifact introduced by mpc xtreme that is 'damn subtle' by the best ears on the board.

Well the probability is that this is true given that you are cutting out 7/8ths of the data in the original file, but many listeners might miss the problems. I guess this demonstrates just how good MusePack is. To put it another way: Of all the codecs discussed here, there is more anguish in trying to prove any artefacts with MusePack than with the others.

Is this another case similar to the "crap mpc" thread with the weird sounding hi-hats on Nirvana’s “Lounge Act”; in which Garf wrote:

and Dibrom wrote:


It is then probable that mpc is not quite transpparent at xtreme setting. Will a few tweaks as have been suggested help with this? I encoded a track (irrelevant which one) with xtreme and with -xtreme -nmt 10 and the bitrate rose from 186 to 201, which is not huge. I would be interested if Garf would try -xtreme - nmt 10 suggested by Dibrom.

Is the aim of this type of process to improve the encoder, and if so, who does it? Dibrom, do you tune mpc in a similar manner to mp3?

In this case the only person who could do something is Andree Buschmann. I believe this situation would need a somewhat modificated tonality estimation, so that enough resolution was used by the encoder.

There are problems for MPC when played with DSP plugins but there are no problems for MP3 or AAC (comparing to the original WAV, at least no loss of bass) when played with DSP plugins. I wonder why MPC does not do well with the DSP plugins.

And you tested with ANS disabled? There are problems with mp3 if played with DSP-plugins, maybe not the same that MPC has, but these are different codecs.

This is not true. Any lossy format will possibly have problems when post processing is used. This kind of thing "breaks" the idea behind the psychoacoustics in the way masking works and such. MP3 and AAC are not immune to this. In fact, I would probably say that more often than not they would probably show more problems with post processing than a codec like MPC due to the nature of the way the different encoders work. For the same reason that I believe MPC would be more forgiving of post processing, it also lends itself fairly well to transcoding if necessary.

Sorry to say, but the "crap mpc" thread was a load of BS... heh. All the parties involved in that charade simply vanished when we pressed for some evidence or something to back up the claims that were made. There was not a shred of credibility shown there. What Garf was talking about was an entirely other issue which was not related.

QUOTE

It is then probable that mpc is not quite transpparent at xtreme setting.

Well of course it's not [b]absolutely transparent. I don't think any lossy encoder at any mode is period. There will always be some sample out there that someone will be able to hear a difference with no matter what. This is just a given. The point is that the probability of MPC not being transparent at -xtreme is much less likely than with a codec such as MP3 or even some others.



I don't think simply raising the nmt would really be the best way to address this overall. IMO the bitrate is already where it should be, if anything the psymodel should just be tuned to be a little more adaptive still. However.. do not take that statement out of context. MPC's psychoacoustics are already extraordinarily well tuned. Even if this sample is not transparent, I'm not sure I'd say it was a "problem". As noted, this apparently is a very hard difference to distinguish, and from what I gather it's certainly not of the offensive kind. Garf even said if he wasn't listening critically he would have missed it. So while for academic purposes it would be great to improve upon the psymodel to "fix" this clip, I don't think it implies some great problem or anything like that.



As JohnV pointed out, Andree would have to address this. From what I gather, a larger scale reimplementation of something like the tonality estimation may be in the works which may address issues like this a little better. Again, don't take that out of context though.. MPC already does an excellent job in probably 99% of the cases I've seen so far.



Nope.