ARFlac.pl - Check flac files with AccurateRip, Split from ARCue.pl (Topic ID: 53583) Options

[Cerebus]

Here's the script, which I called ARFlac.pl. It requires that you have metaflac in your path. Note that much of this code came from ARCue.pl, and so I can't be expected to know everything about the internals of the script - I just adapted it to my needs.

Note:

• This script does not currently work on rips with other than a 2 second pregap on track 1. All tracks will come back as being incorrect or not in the database.
• This script does not currently work on rips from cds with a datatrack.
• If the AR values that come back are all wrong, it is likely a different pressing or a different track 1 pregap. If only ONE doesn't match, that track is probably bad.

CODE

#!/usr/bin/perl

###############################################################################
# #
# ARFlac.pl #
# #
# KitchenStaff (Kitchen.Staff.Supervisor@gmail.com) #
# #
# Many thanks to Mr. Spoon for his kind permission to use the AccurateRip #
# technology and database. Access to AccurateRip is regulated, see #
# http://www.accuraterip.com/3rdparty-access.htm for details. #
# #
# Original ARCue script by Christopher Key <cjk32@cam.ac.uk> #
# #
# This script allows the usage of AccurateRip's extensive track checksum #
# database to verify the accuracy of directories of flac files. To use, #
# simply run: #
# #
# ARFlac.pl Flacdir1 Flacdir2 etc... #
# Note that this is not a perfect system. If the disc originally had a data #
# track, this routine may not find a match. This is also dependent on the #
# Accurate Rip database having seen the CD a sufficient number of times. #
# This tool should be used to verify that a set of files is good, but not to #
# necessarily indicate that the entire set is bad (different pressings, etc.) #
# #
###############################################################################

use strict;

use LWP;
use Carp;
use POSIX;

my $lwpUserAgent = LWP::UserAgent->new;

foreach my $flacDir (@ARGV) {

# Get the list of flac files from the directory
print $flacDir . ":";

opendir CURDIR, $flacDir;
my @flaclist = readdir CURDIR;
closedir CURDIR;

my @filelist = grep {/flac$/i} @flaclist;
@filelist = sort @filelist;

# We need track offsets in 'frames'
# can get these from metaflac
my (@trackOffsets,@trackLengths,@trackSamples);
my $tn=0;
my $tmpl=0;
foreach my $fl (@filelist) {

$tmpl = `metaflac --show-total-samples "$flacDir/$fl"`;
$trackSamples[$tn] = floor($tmpl/1);
$trackLengths[$tn] = ceil($tmpl/588);
printf $trackSamples[$tn] . ":";
$tn++;
}

my $trackCount = $tn;
my $curoff = 0;
$trackOffsets[0] = 0;
for ($tmpl = 1; $tmpl <= $trackCount; $tmpl++) {
$curoff += $trackLengths[$tmpl-1];
$trackOffsets[$tmpl] = $curoff;
}

# Calculate the three disc ids used by AR
my ($discId1, $discId2, $cddbDiscId) = (0, 0, 0);

{
use integer;

for (my $trackNo = 0; $trackNo <= $trackCount; $trackNo++) {
my $trackOffset = $trackOffsets[$trackNo];

$discId1 += $trackOffset;
$discId2 += ($trackOffset ? $trackOffset : 1) * ($trackNo + 1);
if ($trackNo < $trackCount) {
$cddbDiscId = $cddbDiscId + sumDigits(int($trackOffset/75)+2);
}
}

$cddbDiscId = (($cddbDiscId % 255) << 24) + ((int($trackOffsets[$trackCount]/75) - int($trackOffsets[0]/75)) << 8) + $trackCount;

$discId1 &= 0xFFFFFFFF;
$discId2 &= 0xFFFFFFFF;
$cddbDiscId &= 0xFFFFFFFF;
}

print "\nChecking AccurateRip database\n\n";

# See if we can find the disc in the database
my $arUrl = sprintf("http://www.accuraterip.com/accuraterip/%.1x/%.1x/%.1x/dBAR-%.3d-%.8x-%.8x-%.8x.bin",
$discId1 & 0xF, $discId1>>4 & 0xF, $discId1>>8 & 0xF, $trackCount, $discId1, $discId2, $cddbDiscId);

my $arDiscNotInDb = 0;
my $arNetworkFailed = 0;

my $response = $lwpUserAgent->get($arUrl);

if (!$response->is_success) {
if ($response->status_line =~ m/^404/) {
$arDiscNotInDb = 1;
}else{
$arNetworkFailed = $response->status_line;
}
}

# Extract CRCs from response data
my $arCrcCount = 0;
my @arTrackConfidences = ();
my @arTrackCRCs = ();

if (!($arDiscNotInDb || $arNetworkFailed)) {
my $arCrcData = $response->content;
my $ptr = 0;

while ($ptr < length($arCrcData)) {
my ($chunkTrackCount, $chunkDiscId1, $chunkDiscId2, $chunkCddbDiscId);

# Force perl to interpret these values as signed integers
{
use integer;

$chunkTrackCount = unpack("c",substr($arCrcData,$ptr,1));
$chunkDiscId1 = unpack("V",substr($arCrcData,$ptr+1,4)) + 0;
$chunkDiscId2 = unpack("V",substr($arCrcData,$ptr+5,4)) + 0;
$chunkCddbDiscId = unpack("V",substr($arCrcData,$ptr+9,4)) + 0;
}

$ptr +=13;

if ( $chunkTrackCount != $trackCount
|| $chunkDiscId1 != $discId1
|| $chunkDiscId2 != $discId2
|| $chunkCddbDiscId != $cddbDiscId ) {

croak("Track count or Disc IDs don't match.");
}

# How if it flagged that a track is not in the database?
for (my $track = 0; $track < $trackCount; $track++) {
my ($trackConfidence, $trackCrc);

# Force perl to interpret these values as signed integers
{
use integer;

$trackConfidence = unpack("c",substr($arCrcData,$ptr,1));
$trackCrc = unpack("V",substr($arCrcData,$ptr+1,4)) + 0;
$ptr += 9;
}

if ($arCrcCount == 0){
$arTrackConfidences[$track] = [];
$arTrackCRCs[$track] = [];
}

$arTrackConfidences[$track]->[$arCrcCount] = $trackConfidence;
$arTrackCRCs[$track]->[$arCrcCount] = $trackCrc;
}
$arCrcCount++;
}
}


printf "Track\tRipping Status\t\t[Disc ID: %08x-%08x]\n", $discId1, $cddbDiscId;

# Calculate a CRC for each track
my $errLevel = 0;

# Calculate a CRC for each track
my @trackCRCs = ();
my $FH;
my ($accuratelyRipped, $notAccuratelyRipped, $notInDatabase) = (0, 0, 0);
for (my $trackNo = 0; $trackNo < $trackCount; $trackNo++) {

# Open a pipe to flac decode
open($FH, "flac -d -c -f --force-raw-format --totally-silent --endian=little --sign=signed \"$flacDir/$filelist[$trackNo]\" |");
binmode $FH;

my ($frame, $CRC);
$CRC = 0;
$CRC = processFile($FH, $trackLengths[$trackNo], $trackNo==0, $trackNo==$trackCount-1);

close($FH);

{
use integer;
$trackCRCs[$trackNo] = $CRC & 0xFFFFFFFF;
}


if ($arDiscNotInDb) {
printf " %d\tTrack not present in database. [%08x]\n",
$trackNo + 1, $trackCRCs[$trackNo];

$notInDatabase++;
} elsif ($arNetworkFailed) {
printf " %d\t [%08x]\n",
$trackNo + 1, $trackCRCs[$trackNo];

} else {

my $foundCrc = 0;
my $foundCrcMatch = 0;

for (my $arCrcNo=0; $arCrcNo < $arCrcCount; $arCrcNo++) {
if ($arTrackConfidences[$trackNo]->[$arCrcNo] != 0){
$foundCrc = 1;

if ($arTrackCRCs[$trackNo]->[$arCrcNo] == $trackCRCs[$trackNo]) {
printf " %d\tAccurately Ripped (confidence %d) [%08x]\n",
$trackNo + 1, $arTrackConfidences[$trackNo]->[$arCrcNo], $arTrackCRCs[$trackNo]->[$arCrcNo];

$accuratelyRipped++;

$foundCrcMatch = 1;
last;
}
}
}
if (!$foundCrc) {
printf " %d\tTrack not present in database. [%08x]\n",
$trackNo + 1, $trackCRCs[$trackNo];

$notInDatabase++;
}elsif (!$foundCrcMatch) {
printf " %d\t** Rip not accurate ** (confidence %d) [%08x] [%08x]\n",
$trackNo + 1, $arTrackConfidences[$trackNo]->[0], $arTrackCRCs[$trackNo]->[0], $trackCRCs[$trackNo];

$notAccuratelyRipped++;
}
}
}

if ($arDiscNotInDb) {
print "Disc not present in AccurateRip database.\n";
$errLevel = 2;
} elsif ($arNetworkFailed) {
print "Failed to get $arUrl : " . $arNetworkFailed . "\n";
$errLevel = 3;
} elsif ($accuratelyRipped == $trackCount) {
print "All Tracks Accurately Ripped.\n";
} else {
if ($notAccuratelyRipped >= 3) {
print "Your CD disc is possibly a different pressing to the one(s) stored in AccurateRip.\n"
}

printf "Track(s) Accurately Ripped: %d\n", $accuratelyRipped;
printf "**** Track(s) Not Ripped Accurately: %d ****\n", $notAccuratelyRipped;
printf "Track(s) Not in Database: %d\n", $notInDatabase;

$errLevel = 1;
}

print "\n\n\n";
}

sub processFile {
use integer;
my ($FH, $tracklength, $firstTrack, $lastTrack) = @_;
my ($frame,$CRC,$frameOffset,$frameNo,$sample,$endFrame,$frmloop);

$CRC=0;

if ($firstTrack) {
# Skip first 4 frames
if ($tracklength<=4) {
return 0;
}

if (read($FH, $frame, 4*2352) != 4*2352) { croak ("read failed.") };

if (read($FH, $frame, 2352) != 2352) { croak ("read failed.") };
$sample = unpack("V",substr($frame,2348,4));
$CRC += $sample;
$frameNo = 5;
} else {
$frameNo = 0;
}

if ($lastTrack) {
$endFrame = $tracklength-5;
} else {
$endFrame = $tracklength;
}


for ($frmloop=$frameNo; $frmloop<$endFrame;$frmloop++) {

if (read($FH, $frame, 2352) != 2352) { croak ("read failed.") };
$frameOffset = $frmloop * 588;

foreach (unpack("V588", $frame)) {
$CRC += $_ * (++$frameOffset);
}
}

return $CRC;
}



sub sumDigits {
my $n = shift;
my $r = 0;
while ($n > 0) {
$r = $r + ($n % 10);
$n = int($n / 10);
}
return $r;
}

Moderation: Codebox.

This post has been edited by greynol: Apr 9 2008, 06:29

[rohangc]

Thanks for ARFlac.pl Cerebus. This is exactly what I was looking for.

This post has been edited by greynol: Apr 9 2008, 06:12

[Jean Tourrilhes]

Hi,

First, thank you for this excellent tool. Despite all the limitation (not your fault), I've started using it.

I've started trying a few disks that I have ripped and for which I have the AR results. For most disks, it works ok. However, for a few disks, it calculate a wrong CRC for the first track.

Does anybody has any clue about this, and why it's happening. Maybe we could brute force a solution for that.

Second, in many cases I already have a EAC log, containing the TOC. Could that be used optionally ? The idea is that it may help cases where some tracks are missing, and maybe also fix issues with track 1.

Thanks for the attention, and keep up the good work...

Jean

[AliL]

Wow, I'd been wondering if there was a program out there that could do exactly this, brilliant! It is a bit slow though but I'm just grateful that someone actually put their time and effort into making this, thank you Cerebus (and Chris for the ortiginal arcue.pl script).

One thing though, as it's quite slow and I have a lot of albums to get through I was wondering if there was a way of it outputting the results to a log file or something similar as I could just point it to my files and let it get on with it overnight and as the command prompt only scrolls back a limited way eventually I'd lose some of the output.

Secondly, I have an album that says the tracks are accurately ripped but I get a confidence rating of -56. A bit odd I must say. Any reason this might be happening?

Thanks, AliL

P.S. I've thought of another question, however this is more related to the command prompt behaviour. Is there a way to batch add directories to the commandline rather than typing them in one at a time (or in my case dragging and dropping the path from the address bar into the cmd window)? Or is the script able to recursively check all folders within a directory (which would automatically fix the above question/problem)? Thanks.

This post has been edited by AliL: Apr 11 2008, 14:42

[Corwin]

There's something not quite right here... accuraterip-crcgen does exactly the same thing.

$ ARFlac.pl /amber/music/artists/Metallica/Metallica\ -\ 1984\ -\ Ride\ the\ Lightning/
/amber/music/artists/Metallica/Metallica - 1984 - Ride the Lightning/:12561444:17529456:13734504:18214476:10776864:11652396:17490060:23554104:
Checking AccurateRip database

Track Ripping Status [Disc ID: 000dd6b4-730b1e08]
1 Accurately Ripped (confidence 2) [2019a2aa]
2 Accurately Ripped (confidence 2) [201f0a88]
3 Accurately Ripped (confidence 2) [868abd27]
4 Accurately Ripped (confidence 2) [9bdc2556]
5 Accurately Ripped (confidence 2) [59d21bf6]
6 Accurately Ripped (confidence 2) [aa48f821]
7 Accurately Ripped (confidence 2) [61113500]
8 Accurately Ripped (confidence 2) [f14421bc]
All Tracks Accurately Ripped.

I merged the files together into a single file wav using CUEtools (on W2K in a virtual machine), then tried this:

$ ARCue.pl Metallica\ -\ Ride\ the\ Lightning.cue
Metallica - Ride the Lightning.cue:

Checking AccurateRip database

Track Ripping Status [Disc ID: 000dd7d4-760b1e08]

1 Accurately Ripped (confidence 34) [2019a2aa]
2 Accurately Ripped (confidence 33) [201f0a88]
3 Accurately Ripped (confidence 38) [868abd27]
4 Accurately Ripped (confidence 36) [9bdc2556]
5 Accurately Ripped (confidence 36) [59d21bf6]
6 Accurately Ripped (confidence 36) [aa48f821]
7 Accurately Ripped (confidence 34) [61113500]
8 Accurately Ripped (confidence 30) [f14421bc]

_______________________

All Tracks Accurately Ripped.



All the checksums are the same but the confidence count is about 15 times lower....

BTW this is from the original log file when it was ripped over a year ago:

Track Ripping Status [Disc ID: 000dd7d4-760b1e08]

1 Accurately Ripped (confidence 14) [2019a2aa]
2 Accurately Ripped (confidence 14) [201f0a88]
3 Accurately Ripped (confidence 17) [868abd27]
4 Accurately Ripped (confidence 15) [9bdc2556]
5 Accurately Ripped (confidence 15) [59d21bf6]
6 Accurately Ripped (confidence 15) [aa48f821]
7 Accurately Ripped (confidence 14) [61113500]
8 Accurately Ripped (confidence 11) [f14421bc]

This post has been edited by Corwin: Jun 29 2008, 02:39

[spoon]

The first has a different disc id.

[zer]

Ok, my programming skills are really minimal, I am not even sure how to properly add this option to the perl script,
but it is really easy to make it work on discs with some pregap.
simply initialize $curoff, and $trackOffsets[0] to the right offset sector instead of 0.

I am wanting to check a bunch check a bunch of old rips on linux.
I know CUEtools.NET does some real offset detection,
but without flac support on linux, it is cumbersome to use,
and I have not yet understood its algorithm.

but it appears to me that an ignorant approach will be fine
for 99+% of my rips with correct drive offset, but no cuesheet. (or without need to read cuesheet)
specifically, those with none, or a small silent pregap.

just noting that at least in my collection, some pregap lengths appear much more common than others.
for a subset of my rips (just those with eac logs w/ toc table section):
pregap len: count:
0 173
32 23
30 5
33 3
25,37,72,75 1 each

So, although an explanation of how real offset detection works would be nice,
I would appreciate if someone could help me make this scrips useful for bulk scanning
old rips that do not have AR results. An option to set the starting sector would work,
but maybe for this type of "cueless scanner" it would make sense to try at least a few
common pregap lengths if there is no match for the default assumption of 0?
maybe some more data on these pregap lengths would be useful,
but it appears to me that I could reduce the 'special case' from over 15% to under 1% by just doing a few more lookups.

what would be really great is adding the tags like dBpoweramp and CUEtools.NET do, but thats really over my head at this point.

[patmcg]

Is this script still current and working. Also, is there a tool that will detect if only the offset is wrong? Thanks.

[edwardar]

Is this script still current and working. Also, is there a tool that will detect if only the offset is wrong? Thanks.

Cuetools checks rips against the accuraterip database, and allows you to correct offsets too!

http://www.hydrogenaudio.org/forums/index....c=66233&hl=