CUETools versions 1.9.5 through 2.1.4 (current), AccurateRip support & more Options

[Gregory S. Chudo...]

Just to clarify my position: i think AR does it's job just fine. The exact properties of it's CRC are almost irrelevant, it doesn't need a very strong or very fast CRC to function. My objections to V2 are mostly theoretical. I even could probably implement in some form cross-pressing verification, it's just very annoying that i will have to waste time on it when it could have been much easier with a normal CRC, and the verification could be much faster. I have no enmity towards Spoon or AccurateRip.

This post has been edited by Gregory S. Chudov: May 18 2011, 13:15

[Gregory S. Chudo...]

And just to demonstrate the point about collisions in ARv2.

ARv2 takes the 32-bit sample value, multiplies it by the sample number, resulting in 64-bit number, and then adds the higher 32 bits to lower bits.

Now consider for example the 15th sample in a track, and it's possible values.

CODE

ARv2(15,0x11111111) = 0xFFFFFFFF
ARv2(15,0x22222222) = hi32(15 * 0x22222222) + lo32(15 * 0x22222222) = hi32(0x1FFFFFFFE) + lo32(0x1FFFFFFFE) = 0x1 + 0xFFFFFFFE = 0xFFFFFFFF
ARv2(15,0x33333333) = hi32(0x2FFFFFFFD) + lo32(0x2FFFFFFFD) = 0xFFFFFFFF
...
ARv2(15,0xFFFFFFFF) = hi32(0xEFFFFFFF1) + lo32(0xEFFFFFFF1) = 0xFFFFFFFF

ARv2(15,0x00000001) = hi32(0x00000000F) + lo32(0x00000000F) = 0x0000000F
ARv2(15,0x11111112) = hi32(0x10000000E) + lo32(0x10000000E) = 0x0000000F
...
ARv2(15,0xEEEEEEEF) = hi32(0xE00000001) + lo32(0xE00000001) = 0x0000000F

That is, at least 15 sample values in this location produce the same CRC. At least 14 sample values produce another CRC. I'm too lazy to explore further, but this might be equivalent to loosing 3-4 bits of of this sample.

UPD: Not so lazy:) It gets worse for sample 255...

CODE

ARv2(255,0x01010101) = hi32(0x0FFFFFFFF) + lo32(0x0FFFFFFFF) = 0xFFFFFFFF
ARv2(255,0x02020202) = hi32(0x1FFFFFFFE) + lo32(0x1FFFFFFFE) = 0xFFFFFFFF
...
ARv2(255,0x0F0F0F0F) = hi32(0xEFFFFFFF1) + lo32(0xEFFFFFFF1) = 0xFFFFFFFF
ARv2(255,0x10101010) = hi32(0xFFFFFFFF0) + lo32(0xFFFFFFFF0) = 0xFFFFFFFF
ARv2(255,0x11111111) = hi32(0x10FFFFFFEF) + lo32(0x10FFFFFFEF) = 0xFFFFFFFF
...
ARv2(255,0x1F1F1F1F) = hi32(0x1EFFFFFFE1) + lo32(0x1EFFFFFFE1) = 0xFFFFFFFF
...
ARv2(255,0xFFFFFFFF) = hi32(0xFEFFFFFF01) + lo32(0xFEFFFFFF01) = 0xFFFFFFFF

That's 255 samples having the same CRC... Basically, the same problem as with ARv1.

This post has been edited by Gregory S. Chudov: May 18 2011, 23:21

[greynol]

So you are saying I have an obligation to do this?

No, I believe I was quite clear in what I said. Anyhow, seeing that neither of us has changed our stance since the last time we butted heads, I think I'll just go with the "if you don't have anything nice to say, then don't say anything at all" approach.

[HotShotFR]

Hello there,
a rather simple question for a complicated case:

is it possible to 'verify' against AR/CTDB a Playstation-style cuesheet referencing TWO pre-audio data tracks ? i.e. the first playable audio CD track starts at #3.

All my attempts at it have failed so far, since only the first referenced data track is detected and thus the generated AR record has no match.
Out of despair I'm starting to experiment with alternative cuesheet designs, file based vs. track based, compliant or not, messing with pregaps and stuff...

Overview of the CD TOC, first few lines:

CODE

Track |   Start  |  Length  | Start sector | End sector
    ---------------------------------------------------------
        1  |  0:00.00 |  7:06.02 |         0    |    31951    <= that's data
        2  |  7:06.02 | 21:19.06 |     31952    |   127882   <= still data
        3  | 28:25.08 |  2:12.33 |    127883    |   137815   <= here we go with audio!

(This 2 pre-audio track thing is such a glorious idea...)

This post has been edited by HotShotFR: May 20 2011, 10:40

[drfr]

OK, so now you are getting the same errors I reported earlier on the thread.
I dont know the reason for those errors, they are probably related to .NET 4.0.
Try this code instead, it works for me:

CODE

if (processor.PreGapLength != 0)
return processor.WriteReport();;
foreach (uint gap in new uint[3] {32, 33, 37})
{
processor.PreGapLength = gap;
processor.ArVerify.ContactAccurateRip(AccurateRipVerify.CalculateAccurateRipId(processor.TOC));
}
return processor.WriteReport();

Does this really work for you?
When testing on a rip with known 37 pregap and present in AR database, first it shows greyed AR icon for a while, then for just a fraction of second
I can see it has found match but then stops with batch log result: AR: disk not present in database.
So it seems to be close to doing what it should but something is missing.
Any idea?

This post has been edited by drfr: May 22 2011, 08:43

[korth]

So it seems to be close to doing what it should but something is missing.
Any idea?

See this message

Perhaps the missing parts?

[korth]

I saw some discussion about this somewhere else. How can all the tracks return "(0/0) No match but offset" ?

CODE

[CUETools log; Date: 4/29/2011 1:59:20 PM; Version: 2.1.1]
[CTDB TOCID: gFCqM0DjKCCUX0sr_Hee.snk.mE-] found.
[ CTDBID ] Status
[1998766c] (4/4) Accurately ripped
[AccurateRip ID: 001c6798-011d7713-ac0d620d] found.
Track [ CRC ] Status
01 [68f1f565] (0/0) No match but offset
02 [3dc5d93d] (0/0) No match but offset
03 [559a9dcc] (0/0) No match but offset
04 [a4a4ba22] (0/0) No match but offset
05 [83f4349b] (0/0) No match but offset
06 [b5a142ff] (0/0) No match but offset
07 [74e00357] (0/0) No match but offset
08 [54efd8f2] (0/0) No match but offset
09 [eb152ab6] (0/0) No match but offset
10 [7372a99c] (0/0) No match but offset
11 [8c120f5b] (0/0) No match but offset
12 [aee4e5a8] (0/0) No match but offset
13 [65b0e952] (0/0) No match but offset

Track Peak [ CRC32 ] [W/O NULL]
-- 99.8 [C01A9145] [49C4FCD1]
01 99.8 [CBD678A6] [A9EEAD84]
02 99.8 [3545F77C] [17F572CA]
03 93.3 [45584278] [E5EC2DFD]
04 98.5 [36AB01C5] [477BC49D]
05 97.7 [5D4E7C2B] [B783D705]
06 99.2 [CBDD28B0] [6ADC932A]
07 98.8 [67DA1B68] [355E3F04]
08 99.8 [741C13E7] [A0FCA76B]
09 98.1 [28046873] [74213A74]
10 99.8 [BBCB1F02] [D323A234]
11 97.7 [8B74D3EC] [B4F94215]
12 99.4 [AD9FD8B2] [E6175469]
13 99.8 [C9E0F028] [971B67E5]

[drfr]

So it seems to be close to doing what it should but something is missing.
Any idea?

See this message

Perhaps the missing parts?

Thanks. One day I must find the time to read through the whole thread.

[HotShotFR]

I saw some discussion about this somewhere else. How can all the tracks return "(0/0) No match but offset" ?

IIRC that is either due to past aggressive cleanup of the AR database (?) or, more likely, when only two conflicting rips have been submitted to the DB and thus await resolution by a third one.
I have a small number of rips that return such a "(0/0) No match but offset" or "(0/0) Rip not accurate"... which sounds a bit counterintuitive indeed.

This post has been edited by HotShotFR: May 23 2011, 16:40

[Gregory S. Chudo...]

is it possible to 'verify' against AR/CTDB a Playstation-style cuesheet referencing TWO pre-audio data tracks ? i.e. the first playable audio CD track starts at #3.

I never saw such strange discs, so it's untested. How is cuesheet supposed to look like in this case? Can you send me the cuesheet and eac log?

[Goratrix]

when only two conflicting rips have been submitted to the DB and thus await resolution by a third one.

But in that case, it returns just "(0/0) No match" as there is no checksum data available to compare to.

A "(0/0) No match but offset" result indicates there is some data for comparison, but why the 0/0 then? Mind boggles

[Marc27]

I apologize if this is completely nonsense... but I don't know... with all the discussion about AR/CueTools databases. Why can't the internal checksums be used for verification. Since this info is already stored there would be no need to process the audio again (lot faster)
and it should be more accurate (md5 128 bit vs 32 bit).

[Gregory S. Chudo...]

32 bit checksums are quite enough. But you do have a good point about not having to process the audio again. Unfortunately, this won't work, and there is a number of reasons for that.

1) This would require a CD drive to support overread into leadin/leadout, else some leading or trailing samples are lost and hash changes.
2) If the database was track based, you'd have to keep your rip as separate tracks for this to work, and if it was disc image based, you'd have to keep your rip as an image.
3) You'd have to store your files in a format that does have an md5 hash, i.e. no WAV, no ALAC, probably no WMAL... And even in FLAC it's optional.
4) You wouldn't be able to compare your rip with a different pressing.
5) You'd still want to be sure that audio does indeed match the hash.

Frankly, i don't like the fact that FLAC uses md5 hash and i am sometimes tempted to propose to get rid of it (you can, because it's optional) and replace it by some tag containing a good checksum like CRC32. The reason for that is that md5 is the only part of FLAC encoding process that cannot be parallelized. In the age of GPUs and multicore processors that is unacceptable. For example, FLACCL's encoding speed is already limited by md5 calculation speed.

Hash functions should be used for cryptography, not for data integrity checks. They are slow and non-parallelizable. They are meant to protect against deliberate tampering, not hardware/software failures. They have their uses in DRM, but are out of place in open formats like FLAC. md5 as a checksum is not stronger than CRC128, but is slower. And it's an overkill. I would expect FLAC to use Fletcher-16 for frame CRCs and CRC32/64 for file CRC.

This post has been edited by Gregory S. Chudov: May 25 2011, 22:51

[sauvage78]

I would expect FLAC to use Fletcher-16 for frame CRCs and CRC32/64 for file CRC.

Stupid question, (maybe going a little offtopic) would it be noticeably faster on single core CPU too ?

I ask cauz I think I recall that I compared TAK with Flac in the past & I noticed that (IIRW) TAK encoding was mainly much faster than Flac due to the fact that TAK doesn't output MD5 sum by default while Flac does it by default. Once I enabled MD5 in TAK I think I recall the encoding speed difference wasn't so important anymore. (TAK was likely still a little faster, but not as incredibly faster as some test which compare TAK vs. Flac at default setting without caring for MD5. Indeed TAK compression was still 1or1.5% smaller/better.)

I ask cauz as a flac user I would enjoy that the encoding speed gap between Flac & Tak at default [so Flac(With MD5) vs. TAK(Without MD5)] would be reduced ... TAK would still win by a good margin on compression, but the encoding speed comparison would be more fair IMHO.

I mean if it would really work as you say, & more if it's faster on single core too (A slow Sempron 3000+ user like me cares a lot about speed!!!), it would be an interesting improvement to Flac ... well if Josh still knows how to code for Flac & if he reads CT topics which I doubt anyway ;(

In the end, I think MD5 audio sums should still be available in Flac (just not as the default sum if there is a more efficient alternative as you think), because the main other lossless codecs offer it, because it works & just in case your solution wouldn't work

This post has been edited by sauvage78: May 26 2011, 00:32

[Marc27]

Thanks for the answer and all the information. It's very clear now. I wasn't aware as well of the md5 performance issues...

Edit: about the checksum used, should be "proof of" incorrect ripping settings? IIRC I have read in some threads that some errors may go unnoticed due to this (like C2 correction). Personally I plan to re-rip many albums based on this (though they are reported as accurate by AR), but "in a more perfect world" if they pass AR/Cuetools verification it shouldn't be necessary.

This post has been edited by Marc27: May 26 2011, 00:49

[Porcus]

Edit: about the checksum used, should be "proof of" incorrect ripping settings? IIRC I have read in some threads that some errors may go unnoticed due to this (like C2 correction). Personally I plan to re-rip many albums based on this (though they are reported as accurate by AR), but "in a more perfect world" if they pass AR/Cuetools verification it shouldn't be necessary.

I wouldn't worry too much. Yes there are chances for collisions due to weak CRC, but as Gregory points out, there is no deliberate tampering of data -- the errors are drawn random (subject to passing the drive's error corrections). Yes a few samples at the ends are cropped off too. (You might want to notice that e.g. EAC runs through the ripping procedure first, and checks against AccurateRip afterwards. dBpoweramp, on the other hand, invokes no secure reading mode if a burst rip is reported Accurate.)

If you still worry, then run them through the Cuetools database, which is disc based.

If you still want to re-rip a few discs after this, I would assume that you need to use a different drive in order to detect differences. And «different drive» -- you could probably sort this out, but there are many drives that are so similar in actual spinner and firmware, that you could risk picking up just another CD-ROM drive reading the same way. Now you might want to ask «how likely is that?», but bear in mind that we work under the assumption that AR is not accurate enough. You think «drive collision» is less probable than «CRC collision»?
But let us now assume that you then find a bit-different rip where both new and old rip were reported Accurate: How do you tell which one is wrong? Multiple issues will hit you then:
- Lead-in/lead-out. You could probably sort this out from knowledge of the drives.
- Manufactoring errors. It might simply be that the disc has errors which there is no «single true» way of reading. Presumably that is the case I found here: http://forum.dbpoweramp.com/showthread.php?p=88265 .
- Trying a 3rd drive to judge between the two? Bigger chance that this is a rebadge of, or something working identical as, one of the two others.


My suggestion: scrap the re-ripping at least until you want to sell the CDs (edit: or until you find something audibly suspicious)

This post has been edited by Porcus: May 26 2011, 08:40

[Nassoo]

Hi! Thanks for the great software!
I have may be stupid question, but... There is an option "Write extraction log to 'LOG' tag"... so, after encoding completed, i don't see this LOG tag in the files... where do i go wrong?

[korth]

when only two conflicting rips have been submitted to the DB and thus await resolution by a third one.

But in that case, it returns just "(0/0) No match" as there is no checksum data available to compare to.

A "(0/0) No match but offset" result indicates there is some data for comparison, but why the 0/0 then? Mind boggles

OK I rechecked 4 discs that came up "(0/0) No match but offset" in v2.1.1 and they come up "(0/0) No match" in v2.0.9!

[korth]

Hi! Thanks for the great software!
I have may be stupid question, but... There is an option "Write extraction log to 'LOG' tag"... so, after encoding completed, i don't see this LOG tag in the files... where do i go wrong?

I think you'll need to provide more info about what type of file and what you're using to view the tags.
There are two such check boxes, one in 'verify' and one in 'encode and verify'. Having the wrong one checked for the process you're using will make a difference.

[liquefied]

Also, does this support AccurateRip v2?

[korth]

Also, does this support AccurateRip v2?

In the next release. See this.

[Gregory S. Chudo...]

I have may be stupid question, but... There is an option "Write extraction log to 'LOG' tag"... so, after encoding completed, i don't see this LOG tag in the files... where do i go wrong?

This option tries to locate EAC log and place it as a tag into resulting file.
For this to work, EAC log should be in the same folder as the source files, and preferably have the same filename as a CUE sheet.
CUETools will try to locate it even if there are several logs and/or the filenames don't match, but it can do this only if those logs are from a version of EAC that puts CD Table Of Contents into the log file.

[Gregory S. Chudo...]

OK I rechecked 4 discs that came up "(0/0) No match but offset" in v2.1.1 and they come up "(0/0) No match" in v2.0.9!

This could be the result of AR database cleanup. There are offset-detection CRCs in this entry, but no AR CRCs.
But there might be a chance that there are only ARv2 CRCs for this CD. Wait for CUETools 2.1.2 to find out.

This post has been edited by Gregory S. Chudov: May 26 2011, 20:44

[Marc27]

dBpoweramp, on the other hand, invokes no secure reading mode if a burst rip is reported Accurate.)

So it's strategy is that if there is an error, AR would detect it, in the mean time the rip is faster. Unfortunately I'm not entirely familiar with dBpoweramp yet.

Switching drives is an option, but I was a bit less ambitious Same drive but different ripping settings. Bottom line is how likely is that some discs may have undetected errors? IMHO I don't like leaving things to chance. Though if the chance of undetected errors is very* very unlikely I wouldn't worry.

Manufactoring errors. It might simply be that the disc has errors which there is no «single true» way of reading. Presumably that is the case I found here: http://forum.dbpoweramp.com/showthread.php?p=88265 ."

So now I understand those results. I initially thought they were slightly different discs/releases.
Also a from the thread that doesn't apply to the thread case, but more to this one.

It would be interesting to do a series of tests in EAC to see if you could get both AR results on that track from the same disc by tweaking the EAC settings.

So let's say this case scenario

1. A rip with burst mode and other incorrect settings.
2. Same disc rip with secure mode/correct settings.
3. Both rips have different CRC128/64.
4. AR/Cuetools databases report both as accurate.

How likely or unlikely is 4 (as well as 3)?
I wish there was some sort of study on the matter, but considering all the factors (different rip settings, drives, etc) that may be overkill.

[Porcus]

So let's say this case scenario

1. A rip with burst mode and other incorrect settings.
2. Same disc rip with secure mode/correct settings.
3. Both rips have different CRC128/64.
4. AR/Cuetools databases report both as accurate.

How likely or unlikely is 4 (as well as 3)?

I am not completely sure what you intend to do. Let me be a total devil's advocate here, and make the worst-possible interpretation -- not because I think it is correct, but it might explain what can go wrong.


What makes AR/Cuetools verification go round, is basically the assumption that an erroneous rip will hardly ever match anything, because it gives a checksum at random -- if we are lucky, not too many orders of magnitude away from a uniform draw. Now if you tweak a setting (like, channel reversal), this is a error that someone else can plausibly have made before you on that disc.

Now you have two events which might trigger an erroneous "Accurate":
- there is a scratch, but the wrong sample does by chance evaluate to an AR match
or
- the CD is otherwise correct, but by human error, someone else has done the same settings mistake as you have
(or both)

I would guess that the latter event is more likely.

Now if you are even tweaking the settings around, it looks like you are deliberately chasing "has someone else done this mistake before me?". That's not what you want. Besides, that's not what anyone else wants -- you are offering a false "Accurate" to the next person who accidentally makes the same settings mistake.



So ... if you might correct my interpretation ...?