Well, the title says it all I guess

Let's take a RME Fireface and a Apogee MiniDAC. Both are using a multibit delta sigma-design. And from the listening tests I've done (ABX) some of the participants (like 7 out of 18), have left commentaries on that there's a bit of difference between the two, pointing towards that the Fireface has a bit more bass and a bit more treble. (I guess it's worth to mention that the MiniDAC was connected via SPDIF from the Fireface, and therefore got it's clock from it. (unless I'm mistaken)

What could be the causes to this difference?
I'm sure that the MiniDAC has got a lot more pricey components in it, and that's why, but what components?

(I'm not asking for anyone to do my homework here - however, finding information about probable causes to this seems difficult. Oh, and I've googled, as well as searched this forum for similar posts =)

Did the participants really pass the ABX tests? Did you level match and time align the audio when doing the tests? Were they blind, and double-blind? How many trials did you do? What were the scores?

I ask all this because in practice any half decent DAC is very difficult, if not impossible, to tell apart in an ABX test from any other half decent DAC in a properly executed ABX test. As to "bass" and "treble", only poor DACs would have problems in this respect.

Edit: and your DACs at tests are much better than "half decent".

Well, the short answer is, I think, yes.



The long answer =) :

I tried 3 different audio clips, repeated 4 times in one test, making for one particiapant to answer 12 questions. The clips were in 16 44. Time alignment as well as level matching was done.

The question order for each participant was randomized (with the generator over at www.random.org).
For each question there were 3 buttons; one button went to converter a, one to b, and one to either a or b, and this was randomized for each question as well as for each test. (Does this make it blind or double blind btw?)

The participants had no idea of what the test was all about, only that it wasn't their hearing that was being investigated, and I had a total of 18 participants. (All of which were either second year audio engineering students, or teachers, or had worked professionally for more than 3 years.)

I've just finished the chi-square testing, and the results are as follows (and bear with me, I'm not used to writing in a correct way, but hey, that's partly why I'm writing this essay ):

- For the whole test, under the 10% level of significance
- For audio clip 1, under the 10% level of significance
- For audio clip 2, under the 5% level of significance
- For audio clip 3, under the 10% level of significance

So, scientifically speaking, as the result is not below 1% or 5% on the total, we can only speak of a marginally significant difference between the two. Which is good, because that goes very well with my own perception that there's a difference, but that it's a very very discrete and small one.

Since you mentioned ABX tests, I will give you the benefit of the doubt and assume that they were conducted properly (level matched, etc).
Clocking issues related to SPDIF which aren't present on other interfaces can introduce jitter and a measureable deterioration in the analog signal. I don't know whether it will be audible, but it's a possibility.

Another thing that could differ is the buffering on the output - the apogee for example advertises "ultra low impedance, high current drive balanced outputs", which are not supplied by the DAC but by some sort of buffer. Such a buffer could introduce a variety of problems, including bass or treble rolloff. Both are good cards though, so I would guess that the output buffer is well designed and doesn't suffer from these problems.

Other issues could be power supply quality (which has an effect on SNR), board design (which can effect frequency response), and some other factors. I would doubt they would be audible, but quality differences are possible.

Ok, but how and to what level was the level matching performed?

This is great - thanks. Do you think you could be a bit more specific about the "other factors"?

It's single blind, at least. Now if you didn't know which DAC was playing the randomized button, or you were not in any kind of contact with the testers during the test, then it would be double-blind.



But you played 3 clips 4 times each (12 times total). With 4 trials on 1 clip you can't achieve a 5% significance: 4 out of 4 is p=6,2%. See http://www.kikeg.arrakis.es/winabx/bino_dist.zip.

Also, p<10% can't be lightly interpreted as the difference being small. It could be the case, but there's no way to know it. Strictly, it means the tester didn't pass the test. ABX tests are for determining if there is a difference or not, but in order to determine it, you must pass (<5%) the test. In case of doubt, repeat the test, account for all the results, and see if p<5% is achieved.

Also, I assume either the testers didn't know if they were right or wrong in each trial, or that the nº of trials was fixed before the test. If none of these were true, p<5% would not be enough to pass the test.

The level matching was performed by running a test tone through each converter, and adjusting so that they were less than 0.1 dB different (checked with an RTW PPM (can't remember what model though).

Ok, that's fine.

Ok, no, I was not in any contact with the testers during the test, so I guess that makes it a double blind.



Hm, this is weird - I'll have to check that out.



Hmm.. ok. But (and let's assume that the chi-square testing is correctly done) since I'm actually close to breaking the 5% barrier, does it actually mean that I have to interpret the result as there being no difference between the two? Is there no "wiggle room" so to speak?



No, they didn't know whether they were right or wrong, however they did know the number of questions.

Thanks for taking your time!

It doesn't mean there is no difference, strictly talking that's impossible to prove. It means you couldn't prove there is a difference.


It really doesn't matter if they knew. What matters is if the nº of trials of the test is not fixed beforehand, the testers know the result of each trial, and the test is stopped when there are "favorable results". But this was not your case, so the test was statistically OK in this sense.

Ah, of course =)
No, I did as many tests as time made it possible to.

Edit: fixed the quote.

But back to the topic at hand - what would make different DAC's sound different, if they both are using a multibit delta sigma design?

There are a couple of issues that need to be examined.

First, and foremost you are not only getting the signal out of the DAC - what you are hearing is a combination of a few effects:

• The digital transport to the DAC. Jitter can have a measureable effect and could possibly have an audible effect. Most other sorts of digitals errors would be clearly audible (clicks, etc)
• The DAC itself. DACs have different frequency response, passband ripple, reaction to jitter, SNR, THD and other factors
• The power supplies in the DACs - one might be dirty and the other might be cleaner - leading to a difference in SNR or an audible hum or buzz on one
• The effects of the output amplifiers/buffer which could introduce noise, frequency limitation, frequency response ripple, slew rate limitation and all manner of other mangling (distortion)

The most likely reason is that you get good DACs and bad DACs. Just because two use the same topology doesn't mean that they will have identical performance. As it always goes with HiFi, more expensive doesn't mean better.

Jitter is another factor which might be audible.
Some interesting reading (take pinches of salt where appropriate):
On Jitter and SPDIF
Digital Domain - Jitter

Perhaps you would interested in the only half-credible perceptual test I have read concerning jitter. It was produced by [1]Dolby Laboratories. Unfortunately, it was more of a preliminary test than one determined to find precise average thresholds. But this test [sadly] is by far the best one performed so far. Much better than the endless speculations and purely sighted evaluations of which make up virtually every other jitter 'threshold' article. Because of the large amount of jitter required to create audible effects in this test, no further testing was ever conducted, since the values established to become audible on music program, even if not very accurate, were well exceeding that of a properly operating/designed DAC. I can only speculate that they saw no reason to do further research due to the very high values found to be required in the preliminary. Since the objective of this test was to determine if jitter was an issue of sound quality in average equipment, no reason existed to perform a test establishing precise values.

I posted some graphs relating to the test results near the end of this post:
http://forums.audioholics.com/forums/showthread.php?t=4547

PM me if you want a copy of the full article.

-Chris

[1]Theoretical and Audible Effects of Jitter on Digital Audio Quality
Benjamin, Eric; Gannon, Benjamin
AES Preprint: 4826

What is the case with the level and frequency-response match? Did you test both to determine how close in terms of anti-aliasing filters and gain were matched?

Those graphs you posted were extremely interesting. The threshold for even the best listeners was in the order of nanoseconds, and tens and hundreds of nanoseconds in some cases. This is interesting because, as you say, other studies have claimed audibility of jitter in the order of a few picoseconds. The method you described indicates that the test was less than perfect, but the results are still interesting.

More interesting reading (which places the threshold of jitter audibility much lower than the dolby article):
Jitter: Specification and Assesment in Digital Audio Equipment by Julian Dunn
Stereophile: Bits is Bits

It really seems as though nobody has done a double blind test with decent methodology. Every test I have seen is flawed in some major way. It's a pity that these tests are so hard to do or we could organise one here and get a good answer to the problem. It seems the best answer you can give at the moment is that the audibility threshold of jitter reduces with frequency and lies somewhere between 10ps and 100ns.

Exactly. Dolby Labs research is by far the best in this particular area. The other jitter audibility threshold reseach was just so bad, as to make Dolby's[which is far from perfect] look like gold in comparison.

I have read all of the links[though I'm not sure about the Julan Dunn one; the link would not work you provided, so I could not review it to see if I had already read that paper] you have posted at one time or another, and the more I read, the less respect I have for most of the authors, due to their lack of competance regarding proper audibility testing methdologies.

-Chris

If your p-values were calculated adding results from all testers, then it would be perfectly possible to achieve low p-values, since the total nº of trials would be high enough. p<5% would mean that there was an audible difference on that clip. Were they calculated that way?

As to level alignment, what frequency and level did you use for the test tone? As to time synchronization, were the clips synchronized while you switched the playback, or did the clips start every time a button was pressed? In the first case, a proper time synchonization is difficult to do, what procedure did you use?

It depends if the listeners were listening at once in the same room. If uncontrolled communication can occur between listeners, it could lead in the worst case as everyone answering the same as a given listener, who could get 4/4 by chance.
To check this parameter, all we can do is getting from one of the listeners a description of how the test went.

Ah yes, that's the way I calculated. Whew .

I used a 1 kHz sinus tone, and calibrated it to -20 dBFS. Then I measured the sweet spot SPL to 80 dB.
Now, I read in Principles of Digital Audio (Ken Pohlmann) that in order to utilize the maximum number of bits used, one should use other test tones, like 993 Hz for example. Could this mean that the test is executed in an incorrect way?
(it is true that I did not fully understand that paragraph (in the book), however in my opinion I think I'm in the clear on that one, or?)

For timematching and switching, I used a program called SFX by a company called Stage Research, which I programmed four keys for each question;

- one for starting each clip
- the other three for selecting an output

The program made direct (inaudible) x-fades between the different DAC:s.



Every participant made the test on their own, with only themselves in the room, i.e. one person at a time.

Then everything seems fine, save for the weak statistical result.

You can look for the listener and sample combination that lead to the best results, and try to train these listeners so that they get a better result.

To answer your question, you have to measure the performances of the DACs yourself. Since some listeners talked about bass and treble, pay attention to the frequency response. Also, look if these 7 listeners have got some good individual results.

In addition to the classical RMAA tests, you can perform DAC specific measurments : spectrum analysis of a 11 kHz sinewave (shows jitter + amplitude modulation + noise), spectrum analysis of a DC signal (shows amplitude modulation + noise), noise (shows noise only), udial test, +3 dB fs/4 sinewave.

That's a great idea, I'll be sure to do that (test those 7 ppl separately)!

I'd love to have run those tests, unfortunately that's not possible though - the MiniDAC was rented for the duration of the test period. However, they sound very interesting, and wonder if you could give me your source of information about them - what I mean is, where can I find information that backs "spectrum analysis of a 11 kHz sinewave (shows jitter + amplitude modulation + noise)" up?

(although it is tempting to write "this guy on the internet told me so" in my report =)

While were on the subject - since the MiniDAC was connected to the Fireface via SPDIF, the Fireface was the clock master with both converters. And if I've got it right, jitter is a cause of a faulty clock. So measuring the jitter on the MiniDAC in this scenario should give the same results as on the Fireface, right?

(Edit: added the last paragraph)

No. The external DAC recovers the clock from the SPDIF stream, which means that the clock that the external DAC uses and the master clock aren't necessarily perfectly synchronised - which introduces jitter.

This point is discussed here by Julian Dunn, Ian Dennis and Doug Carson : http://dspace.dial.pipex.com/town/pipexdsl...ds/cdinvest.pdf

It is interesting to compare this study with Bruno Putzeys' posts, where he states that in the 11.025 kHz spectrum, "anything else [than 11.025 kHz] is jitter" : http://www.pgm.com/pipermail/proaudio/2005-April/000639.html

That's what Dennis et al first assumed too. But they were surprised by the unusual look of that "jitter". So they measured the spectrum of a DC signal, which by definition cannot be affected by jitter. And they found exactly the same perturbations. They concluded that this was an amplitude modulation, not jitter, and that it was likely that other authors also had made the same mistake : taking amplitude modulation for jitter.



No. A good DAC has the ability to reclock the incoming stream. You should measure mostly the DAC's own jitter. Most of the Fireface jitter should be filtered out.



The fact that the clock is embedded into the stream introduces jitter, right (interface jitter, that depends on the cable), but this jitter is then filtered out together with the source jitter by the DAC. What remains is the DAC's clock own jitter, plus maybe the lowest part of the source + interface jitter spectrum.

In practice, Julian Dunn et al have been able to see the interface jitter at the analog output of a bad DAC, chosen on purpose, but not at the output of the Prism DA-1.

I don't think so. 993 Hz and similar are theorically better but only for tasks such as distortion measurement. For level matching it doesn't matter. -20 dB should be OK too.


So you used a software solution for the audio switching. When switching between A and X(A) or B and X(B), was the same fade performed that when switching between B and X(A) or A and X(B)?

I also assume you fed a mixer with the output of the two DACs, and used its mix output to feed an amplifier which fed the speakers/headphones. In which part of the chain was the level-matching performed? If you employed a mixer, did it have tone controls in each input channel/pair assigned to a DAC? Those could be a cause for a frequency response inbalance between the channel(s) assigned to each DAC.

Edit: also, when doing the level matching, was it made "in paralel" with the whole audio setup, or some devices were disconnected, etc, when doing it?

That's an extremely interesting paper. The most interesting conclusion for me was how badly one box CD players performed and the mechanisms behind the deterioration of sound quality. Even in well known name-brand one box players it's clear that there is inadequate attention being paid to isolating the DAC from the CD reading and processing electronics.

From the paper:

That sums the situation up very nicely - when the paper was written many (high-end) integrated CD players were badly designed and corners were cut on power filtering at the expense of sound quality. As they authors state, it would not be an expensive task to include a completely seperate and well filtered power supply and voltage reference for the DAC, which would improve quality significantly.

Their concerns about DVD players are also well placed. I would guess that the vast majority of DVD players (even high end models) will suffer from distortion caused by similar mechanisms. It is likely to be different due to the difference in sampling rates and disc reading technique, but there is little doubt that some distortion would be introduced.

I would guess that the majority of electronics inside a PC run at a sufficiently high frequency that similar distortion mechanisms on a sound card would be inaudible, but there are some things (like hard drive head seeking) which might have an effect.

The noise of a 3 GHz CPU is very audible in the output of an onboard audio chipset. The high frequencies have either low frequency fundamentals, or produce low frequency distortions one way or another.

The 1000hz context switch rate of Windows is audible on some onboard sound chipsets, depending on the system load. This is independent of CPU speed - actually it's more dependent on power supply quality.

This is very interesting. Can I ask you for tips on books for reading up on this?

Here is an essay about jitter http://www.digido.com/portal/pmodule_id=11...der_page_id=28/
It describes the theoretical effets of jitter in S/PDIF lines, and insists on the jitter elimination everytime the data are reclocked. For example when it is recorded on hard drive, or burned on CD.
But keep in mind that the comments on sound quality are not backed up by rigorous blind tests.

The presence of measurable interface jitter depending on the cable in the analog output of a DAC has been reported by Julian Dunn et.al. in the above paper, though the paper does not give details about the measurments.
The ability for a DAC to reject most of the source jitter is part of its basic operation. This point was discussed here : http://www.hydrogenaudio.org/forums/index....topic=11909&hl=

Here is 2BDecided's input on the subject :

Sweet, thx!
And btw, big thanks to everyone participating in this thread - I really appreciate it!

I remember also having seen jitter rejection discussed in webpages about asynchronous sample rate converters (SRC), that are sometimes used as antijitter (I find it to be a strange idea : to use a device that introduces jitter permanently as an anti-jitter )

Yes, exactly.




The mixer I used did have EQ's for the channels used, however they were bypassed with the bypass button on the desk (if this means the signal actually never went through those circuits I don't know, however I should be able to find that out).

The level matching was done like so:
The Fireface was set to use it's +4 dBu output mode, and was then from the channel on the mixing desk fed into the PPM. The MiniDAC was then adjusted with it's volume knob to reach the same level (during which the Fireface channel was switched out).
And then they were individually adjusted with their respective channel faders to -20 dBFS (which meant that both of them were at the same position)

(however I don't understand or remember my reason for doing so - it would've been smarter to just let the faders rest at 0 dB right, in order to let the circuitry in the mixing desk work and therefore color the program material as little as possible?).

A third placebo fader was placed at the same position as the other two.

Also, the level matching was done before each trial (when consecutive trials were done, it was a question of checking that the faders had not been moved by running the test tone sequence once again. They never were moved, had they been I would've disqualified that trial.).


(Also, the listening room used is a control room meant for radio broadcasting, built with bass traps, and uses a pair of Genelec 1022 speakers. I measured the background noise in the listening room to around 29-30 dB SPL.)

Edit: added the part of the Fireface being switched out when adjusting the level on the MiniDAC.

So, where in the signal chain did you perform the level matching? At the output of the mixer?

No, with the volume knob on the MiniDAC.

The level matching should have been performed measuring the signal present at the output of the mixer or at the output of the amplifier.

Not doing so allows for a possibility that a small difference in the amplification/attenuation of each mixer channel (due to electronic component tolerances, for example) ruins the level match between DAC outputs.

Ah, my bad for not clarifying; the measurement was done on the output of the mixer.

Maybe I missed it, but I didn't see where the actual frequency response of each of the units was checked. That tends to be the first thing after level mismatch that I suspect.

One of the early discussions I remember about jitter was that initially it was assumed the jitter would be unrelated to the signal, and in practice it was highly signal related which made it much more audible at lower levels than random noise.

Things got so silly I stopped paying much attention around that point.

reading up, and this caught my attention:
Pio, what did you mean by this "introduces jitter permanently"? ASRC is a very stong anti-jitter technology..

Here, I am talking about the audiophile anti-jitter device that takes place between the CD player digital drive S/PDIF output and the S/PDIF input of the DAC, and that is supposed to eliminate the jitter of the drive.
The ASRC anti-jitter works this way :

Input -> Jitter rejection -> Sample rate conversion -> output.

The asynchronous sample rate conversion records any jitter left after rejection in the digital data. The (extremely small) distortion becomes part of the signal. There are actual informations stored about it in the new digital stream.

I find this concept very strange because DACs have exactly the same jitter rejection devices at their input. So I don't see the use of an additional one. The output suffers from exactly the same jitter as the output of a CD player, that comes from its master clock. And this jitter is negligible anyway compared to the jitter introduced by the digital cable. So at the end, we get the jitter of the cable that is between the anti-jitter device and the DAC. It is exactly the same as if the cable was just between the drive and the DAC.

The only difference is that we have introduced two additional sources of distortions : the sample rate conversion that is lossy, and the fact that it is asynchronous, which translates what's left of jitter after the rejection into the digital data. At the end, it adds up with the new jitter generated by the clock of the ASRC device and by the next cable.

All this mess should be inaudible, but making an ASRC that is inaudible is not trivial. This kind of device should always be tested with udial.wav in order to check for added distortion.

Oh, I see. It wasn't obvious from your post and left impression that you object ASRC generally. Putting ASRC between 2 SPDIF cables is strange indeed.

ASRC is meant to replace PLL jitter rejection inside a DAC. It gives you a freedom to use most stable single-frequency clock you can buy for DA, and avoid the need to sync to the input by tuning DA clock to it. By that you can in theory make jitter rejection arbitrarily stong. In any case, jitter rejection of ASRC is way stronger than in any PLL based DAC.

DACs generally do NOT have "exactly the same jitter rejection at their input". Any DAC that doesn't limit you to single sample rate and provide stable master clock for slaving, either uses PLL (most) or uses ASRC. In latter case additional ASRC in the SPDIF chain is pointless, in PLL case it indeed has capacity to remove most of the CD drive jitter, and leave only last cable induced jitter. Probably good for DACs that have weak jitter rejection.

Regarding residual jitter induced (extremely small) distortion becoming part of the signal, this is a problem only for D-D (dubbing) transfers, where input clock dependent ASRC is never to be used. For D-A, residual error is no better or worse than jitter induced DA conversion error. Its actually analog error.

No, it works like this:
Input -> 1M times oversampling to ~40GHz with filtering to digitally represent analog waveform -> Sample correct point for output sampling rate -> output. It is equivalent to analog DA -> AD, but computed digitally. Insane oversampling is reduced to practical implementations by smart tricks.
The residual jitter is in uncertainty of input and output sampling rate ratio, that could be made as low as needed over time.

There a nice thread going deeper into ASRC details here:
http://www.diyaudio.com/forums/showthread....&threadid=28814

Sorry for the late reply.


So, it seems there was nothing wrong in the procedure and people did hear a difference. However it's possible those could have been also from some inbalance in the mixer channels, since the DACs were in theory of good quality.

To find out the causes of these differences some standard audio measurements should have been taken at the output of the mixer, specially a frequency response plot, in order to know if this could be the cause of the audible differences, and if these differences had its origins in the DACs or in the mixer.