CUETools DB Options

[Eli]

I have only now become aware of Gregory S. Chudov's effort to develop CTDB (CUETools DB). I am very excited about this as I have actually been suggesting this to spoon (dbpoweramp's developer for over 5 years)

http://db.cuetools.net/about.php

for others that have missed it:

What's it for?
You probably heard about AccurateRip, a wonderfull database of CD rip checksums, which helps you make sure your CD rip is an exact copy of original CD. What it can tell you is how many other people got the same data when copying this CD. CUETools Database is an extension of this idea.
What are the advantages?

* The most important feature is the ability not only to detect, but also correct small amounts of errors that occured in the ripping process.
* It's free of the offset problems. You don't even need to set up offset correction for your CD drive to be able to verify and what's more important, submit rips to the database. Different pressings of the same CD are treated as the same disc by the database, it doesn't care.
* Verification results are easier to deal with. There are exactly three possible outcomes: rip is correct, rip contains correctable errors, rip is unknown (or contains errors beyond repair).
* If there's a match, you can be certain it's really a match, because in addition to recovery record database uses a well-known CRC32 checksum of the whole CD image (except for 10*588 offset samples in the first and last seconds of the disc). This checksum is used as a rip ID in CTDB.

What are the downsides and limitations?

* CUETools DB doesn't bother with tracks. Your rip as a whole is either good/correctable, or it isn't. If one of the tracks is damaged beyound repair, CTDB cannot tell which one.
* If your rip contains errors, verification/correction process will involve downloading about 200kb of data, which is much more than it takes for AccurateRp.
* Verification process is slower than with AR.
* Database was just born and at the moment contains much less CDs than AR.

How many errors can a rip contain and still be repairable?

* That depends. The best case scenario is when there's one continuous damaged area up to 30-40 sectors (about half a second) long.
* The worst case scenario is 4 non-continuous damaged sectors in (very) unlucky positions.

What information does the database contain per each submission?

* CD TOC (Table Of Contents), i.e. length of every track.
* Offset-finding checksum, i.e. small (16 byte) recovery record for a set of samples throughout the CD, which allows to detect the offset difference between the rip in database and your rip, even if your rip contains some errors.
* CRC32 of the whole disc (except for some leadin/leadout samples).
* Submission date, artist, title.
* 180kb recovery record, which is stored separately and accessed only when verifying a broken rip or repairing it.

[sauvage78]

Hi Greg,
In a discussion (that took a bad direction because I was making a lot of assumptions about CTDB which ended to be erroneous, sorry if it is painfull to read), it appeared that CTDB seems to accept non accurate submissions as long as it is done with cueripper (I didn't test it myself). The so-called AR(1) entries.

I am now worried about this, as from my un-educated point of view it seems that it means that you possibly admit entries to CDTB that might be damaged. The proportion of bad submissions should be low & even in this case it is up to the user to verify that once "fixed" the rip is more accurate than before trying to fix it. ... still even if I think that I know how to avoid "fixing" damaged rips with damaged CTDB submissions, I don't like idea that such non accurate CTDB entries will appear in my .accurip.

So, without resurecting this thread, I wanted to simply ask you: Why you are doing this & why you think that this is a non-issue ? I ask because most of the reason why the linked thread derailed is that I couldn't believe myself that CTDB was behaving like this.

I don't understand the logic of being AR(2) dependent if the rip comes from EAC or dbpoweramp & in the same time not dependent of the accuracy at all if the rip comes from cueripper.

Is there any rationnal explanation ? Is it just that you wanted to populate the database quickly ? (too quickly ...) Have you stored any additionnal informations that would allow you to re-check the accuracy of these suspicious submissions later ? (& purge them ...) Do you think that this is a minor issue ?

I don't understand why you store submissions that might be damaged & which does take some space on your database/hardrive when I think I recall that you said that you couldn't store a stronger correction file to keep the database small.

As I don't want to make assumptions about CTDB anymore (I get trouble with Greynol when I do so), I thought that I'd rather ask you directly, sorry ... Am i just completely wrong once again ?

Thks for any answers & all apologies if this is a non-issue.

This post has been edited by sauvage78: May 27 2010, 18:47

[Gregory S. Chudo...]

It seems that it means that you possibly admit entries to CDTB that might be damaged.

Yes, of course, and there's no way around it. Same goes for AccurateRip - when the first submission for a CD is made, there's nothing to compare it with, and no way to make sure it's really accurate.

I don't like idea that such non accurate CTDB entries will appear in my .accurip.

I don't like it either. If e.g. AccurateRip log would show all non-matching entries, it would be very long and unreadable. CTDB verification log is much shorter because it doesn't have separate entries for pressings with different offsets, but still at some point we will probably have to ignore entries that don't match and have low confidence (1?). For now it shows all the entries and it's up to user to interpret this information.

I don't understand the logic of being AR(2) dependent if the rip comes from EAC or dbpoweramp & in the same time not dependent of the accuracy at all if the rip comes from cueripper.

Not that it's entirely logical, but there are reasons.

In ideal world CTDB shouldn't depend on AR at all and only accept submissions from rippers. You asked if i consider CUERipper submissions a hack - well no, it's the other way around. I had to add another way to submit data because 1) Only one ripper supports CTDB so far, 2) It doesn't even work on some drives, 3) I needed to jump-start the database, because noone would have taken it seriously if it only had a dozen CDs. So it's CUETools' submit feature that is a hack, although a necessary one.

The absolute minimum requirement for CUETools submissions is some evidence that the submitted files really come from a CD and were not modified since they were ripped. CUETools could in theory submit any rip that passes AR check with confidence of 1, or even as someone suggested, a rip that doesn't pass AR check, but has a log and matches CRCs from that log. I still hesitate whether i should do this or not. On one hand, this would speed up database growth. On the other hand, this would reduce data quality.

CTDB submissions use much more disk space/traffic, which demands higher data quality than in AR. As i said, in the end there's no way around it - there will be bad rips in the database, but we must minimize the number of those bad rips. So CUERipper for example doesn't submit a rip to CTDB if it had problems reading some sectors. EAC submits to AR anyway.

Have you stored any additionnal informations that would allow you to re-check the accuracy of these suspicious submissions later ? (& purge them ...)

I store the drive model and ripper version so that i could purge entries from virtual drives or ripper versions with known bugs. By the way, another reason to trust CUETools submissions less than CUERipper submissions is that there's no way i can find out drive model and ripper version (at least if there's no EAC log). Spoon might purge submissions from that drive later, but the erroneous entry would already be in CTDB.