Hydrogenaudio Forums > Searching for a MD5-mass-calculating program

Full Version: Searching for a MD5-mass-calculating program

Hydrogenaudio Forums > Misc. > Off-Topic

R.A.F.

Apr 4 2003, 00:33

I´m still serching for a MD5-mass-calculating program, which works recursively and writes in each dir the checksum-file. The "Easy MD5-Creator", which I used until now doesn´t hit my needs, as you must go into every directory to let it calculate. You get crazy, if you have to do this for 800 and more times.

Any idea?

budgie

Apr 4 2003, 02:49

I don't know... have you tried this one? This one could do the job for you.

http://www.md5summer.org/

arman68

Apr 4 2003, 04:42

On our production systems I use the command line fsum in scripts from http://www.slavasoft.com/fsum/overview.htm to manage our images (approximately 1.6 million files). It works great.

I also use it on my home system for CD/DVD backups to check the integrity of the files (I include one md5 checksum file at the cd root and keep a copy on the hard drive).

I also use m5summer when I need to show some fancy graphics to management (they usually prefer a progress bar rather than waiting a few hours for a script to finish

). There is nothing wrong with md5summer if your prefer it though, it does the same job as fsum very well. The only thing, as it is GUI based you cannot use it in scripts.

arman68

Apr 4 2003, 05:01

I have just quickly whipped a batch file that does what you want to do using fsum. Copy the code below in a text file, name it "make_md5.cmd" and drop it in your path, as well as fsum.exe

I have done some quick testing and it seems to work. Syntax is given when you call the file with no arguments:

CODE

@ECHO OFF
::
::___Author: arman68___Version 04/04/2003___::
::
IF {%1}=={} GOTO :syntax
IF {%2}=={} GOTO :syntax
SETLOCAL
SET source=%1
IF NOT EXIST %source% GOTO :no_source
SET filemask=%2
ECHO.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO Generating local MD5 checksums in all folders and subfolders
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO.
::_________________________Initiate recursion at top of tree
CALL :tree_walker %source%
TITLE Completed.
ENDLOCAL
GOTO :EOF
::_____________________________________________________________________________
::_____________________________________________________________________________
:tree_walker
::
SET source=%1
::__________________________Strip quote marks
SET source=%source:"=%
TITLE %source%
ECHO %source%
::__________________________Generate MD5 for all files in current dir
fsum -d"%source%" -jm -jnc %filemask% 1>"%source%\checksum.md5" 2>NUL
::__________________________Recursive call to walk down the directory tree
FOR /F "tokens=*" %%d in ('DIR /AD /B "%source%"') DO CALL :tree_walker "%source%\%%d"
GOTO :EOF
::_____________________________________________________________________________
::_____________________________________________________________________________
:no_source
ECHO.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO ERROR: The SOURCE directory cannot be found.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO.
GOTO :EOF
::_____________________________________________________________________________
::_____________________________________________________________________________
:syntax
ECHO.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO Syntax: %0 ^<SOURCE^> ^<FILEMASK^>
ECHO.
ECHO This batch file will recursively walk a directory tree and generate
ECHO a MD5 chesksum file for for all files in each directory. This MD5 checksum
ECHO file is placed alongside the files and can later be used to
ECHO check the integrity of the sales JPEGs by invoking check_md5.cmd
ECHO.
ECHO To use arguments with spaces, enclose them in ""
ECHO.
ECHO Example: %0 "C:\My Music" *.mp3
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO.
GOTO :EOF
::_____________________________________________________________________________
::_____________________________________________________________________________

atherean

Apr 4 2003, 05:07

Perhaps http://www.big-o-software.com/products/hksfv/ is what you're looking for.

R.A.F.

Apr 4 2003, 05:40

Thanks, friends. I checked them out. But neither MD5-Summer nor HKSFV do generate md5-checksum-files for each sub-directory. The only chance still is Arman68´s script. But Arman, can the md5-file take over the filename from the subdirectory (with suffix ".md5" then)? - If yes, it would be great! - But probably it´s too complicate to program, eh?

It´s really a pitty, that this "Easy MD5-Creator" is not able to write in each dir a md5-file. Because the frontend itself is nearly perfect.

rohangc

Apr 4 2003, 05:46

Try Brad Smith's Easy MD5 creator. You can download it here. I've been using it for a year now-never felt the need for anything else. If you are trying to create MD5 checksums for MP3s, then try Uberizer. It not only creates the MD5 files, but (to a certain extent) tells you if your MP3s are upto the mark.

See ya.

arman68

Apr 4 2003, 06:42

QUOTE(R.A.F. @ Apr 4 2003 - 11:40 AM)

Arman, can the md5-file take over the filename from the subdirectory (with suffix ".md5" then)? - If yes, it would be great! - But probably it´s too complicate to program, eh?

It is fairly easy to do, you need to modify check_md5.cmd:

In :tree_walker, after the line

CODE

SET source=%1

add the following line:

CODE

SET md5file=%~n1.md5

then modify the call to fsum (a few lines below) from

CODE

fsum -d"%source%" -jm -jnc %filemask% 1>"%source%\checksum.md5" 2>NUL

CODE

fsum -d"%source%" -jm -jnc %filemask% 1>"%source%\%md5file%" 2>NUL

arman68

Apr 4 2003, 06:46

Here is the other script which you need to check the existing files using the md5 checksum files. Save it as check_md5.cmd

CODE

@ECHO OFF
::
::___Author: arman68___Version 04/04/2003___::
::
IF {%1}=={} GOTO :syntax
SETLOCAL
SET source=%1
IF NOT EXIST %source% GOTO :no_source
ECHO.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO Checking MD5 checksums for all files in %source%
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO.
::_________________________Initiate recursion at top of tree
SET /A failed=0
SET /A missing=0
ECHO %DATE% %TIME% - Files that failed the MD5 check >md5_failed.log
ECHO. >>md5_failed.log
ECHO %DATE% %TIME% - Missing files >md5_missing.log
ECHO. >>md5_missing.log
CALL :tree_walker %source%
ECHO.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO Failed files : %failed%
ECHO Missing files: %missing%
ECHO.
ECHO Check files md5_failed.log and md5_missing.log for more details.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO.
ECHO. >>md5_failed.log
ECHO %DATE% %TIME% - Total: %failed% files >>md5_failed.log
ECHO. >>md5_missing.log
ECHO %DATE% %TIME% - Total: %missing% files >>md5_missing.log
TITLE Failed:%failed% Missing:%missing% - Completed.
ENDLOCAL
GOTO :EOF
::_____________________________________________________________________________
::_____________________________________________________________________________
:tree_walker
::
SET source=%1
::__________________________Strip quote marks
SET source=%source:"=%
::__________________________Check MD5 for all valid files in current dir
IF EXIST "%source%\checksum.md5" (
TITLE Failed:%failed% Missing:%missing% - Processing: %source%
FOR /F "tokens=1-4" %%r IN ('fsum -d"%source%" -c -jf "%source%\checksum.md5" 2^>NUL') DO CALL :parse_output %%r %%s %%t %%u
)
::__________________________Recursive call to walk down the directory tree
FOR /F "tokens=*" %%d in ('DIR /AD /B "%source%"') DO CALL :tree_walker "%source%\%%d"
GOTO :EOF
::_____________________________________________________________________________
::_____________________________________________________________________________
:parse_output
IF "%1"=="FAILED" (
SET /A failed=%failed%+1
ECHO %source%
ECHO FAILED : %3
ECHO %source%>>md5_failed.log
ECHO %3>>md5_failed.log
) ELSE (
IF "%1"=="NOT" (
IF "%2"=="FOUND" (
SET /A missing=%missing%+1
ECHO %source%
ECHO MISSING: %4
ECHO %source%>>md5_missing.log
ECHO %4>>md5_missing.log
)
)
)
GOTO :EOF
::_____________________________________________________________________________
::_____________________________________________________________________________
:no_source
ECHO.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO ERROR: The SOURCE directory cannot be found.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO.
GOTO :EOF
::_____________________________________________________________________________
::_____________________________________________________________________________
:syntax
ECHO.
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO Syntax: %0 ^<SOURCE^>
ECHO.
ECHO This batch file will recursively walk a directory tree and check
ECHO the MD5 chesksums against the existing files.
ECHO It reports failed and missing files, on screen and in logfiles.
ECHO.
ECHO Example: %0 "C:\My Music"
ECHO þþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþ
ECHO.
GOTO :EOF
::_____________________________________________________________________________
::_____________________________________________________________________________

arman68

Apr 4 2003, 06:51

Same here, if you want to use md5 checksum files named after the directory, you need to modify :tree_walker from

CODE

:tree_walker
::
SET source=%1
::__________________________Strip quote marks
SET source=%source:"=%
::__________________________Check MD5 for all valid files in current dir
IF EXIST "%source%\checksum.md5" (
TITLE Failed:%failed% Missing:%missing% - Processing: %source%
FOR /F "tokens=1-4" %%r IN ('fsum -d"%source%" -c -jf "%source%\checksum.md5" 2^>NUL') DO CALL :parse_output %%r %%s %%t %%u
)
::__________________________Recursive call to walk down the directory tree
FOR /F "tokens=*" %%d in ('DIR /AD /B "%source%"') DO CALL :tree_walker "%source%\%%d"
GOTO :EOF

CODE

:tree_walker
::
SET source=%1
SET md5file=%~n1.md5
::__________________________Strip quote marks
SET source=%source:"=%
::__________________________Check MD5 for all valid files in current dir
IF EXIST "%source%\%md5file%" (
TITLE Failed:%failed% Missing:%missing% - Processing: %source%
FOR /F "tokens=1-4" %%r IN ('fsum -d"%source%" -c -jf "%source%\%md5file%" 2^>NUL') DO CALL :parse_output %%r %%s %%t %%u
)
::__________________________Recursive call to walk down the directory tree
FOR /F "tokens=*" %%d in ('DIR /AD /B "%source%"') DO CALL :tree_walker "%source%\%%d"
GOTO :EOF
::_____________________________________________________________________________

R.A.F.

Apr 4 2003, 07:13

Okey, I try to fudge it together when I´m up again. Have to go to bed now.

Thx !

Heaven17

Apr 4 2003, 07:54

Case's 'Sweep' is your friend

http://www.saunalahti.fi/~cse/files/Sweep.zip

sweep fsum *.mpc

Apr 4 2003, 08:58

QUOTE(arman68 @ Apr 4 2003 - 02:42 PM)

QUOTE(R.A.F. @ Apr 4 2003 - 11:40 AM)

Arman, can the md5-file take over the filename from the subdirectory (with suffix ".md5" then)? - If yes, it would be great! - But probably it?s too complicate to program, eh?

It is fairly easy to do, you need to modify check_md5.cmd:

If only fsum had 'exclude' switch.
If I use *.* as mask and output md5 file to the same dir that is being processed, I'd like to exclude *.md5, naturally.

Maybe something like this then:
fsum -d"%source%" -jm -jnc %filemask% 1>"%TEMP%\%md5file%" 2>NUL & move "%TEMP%\%md5file%" "%source%"

goweropolis

Apr 4 2003, 10:48

Back to the original question, I don't know one for MD5s, but cSFV will resursively search and add a SFV file for each directory (recursive SFV creation).

Now my question to you guys, is there a way to recursively search folders and delete all SFV files (or any other type of file for that matter) in Windows XP? Maybe some kind of del command line or something?

R.A.F.

Apr 4 2003, 20:14

QUOTE(goweropolis @ Apr 4 2003 - 05:48 PM)

Now my question to you guys, is there a way to recursively search folders and delete all SFV files (or any other type of file for that matter) in Windows XP? Maybe some kind of del command line or something?

Just use Total Commander, goto "Commands" -> "Search" -> in the line "Search for" type in "*.sfv" and choose the wished directories, Total Commander shall search after these file-extensions. Then click on "use", highlight all the found sfv-files with the "*"-command and you can delete them by pressing the "DEL"-key on your keyboard. - Sounds now complicate, but if you´ve done it for some times, it will go quite fast. And it´s a thousand times faster, than looking in each subdir and delete it manually.

mmortal03

Apr 4 2003, 23:48

what OS versions will all these scripts work on?

Apr 5 2003, 03:26

QUOTE(goweropolis @ Apr 4 2003 - 06:48 PM)

Yep: (del /s *.sfv) will delete them from all subdirectories.

QUOTE(mmortal03 @ Apr 5 2003 - 07:48 AM)

what OS versions will all these scripts work on?

These work on XP (at least here):

Create md5's in all subdirs for *.*:

QUOTE(md5.bat @ system dir)

for /r %%I in (.) do fsum -d"%%~I" -jm -jnc *.* > "%tmp%\%%~nI.md5" & move "%tmp%\%%~nI.md5" "%%~I"
for /r %%I in (*.md5) do if %%~zI==0 del "%%I"

Check recursively all md5's, output results to temp txt and open it:

QUOTE(md5c.bat @ system dir)

for /r %%I in (*.md5) do fsum -d"%%~dpI." -c "%%~I" >> %tmp%\chk.txt
%tmp%\chk.txt & del %tmp%\chk.txt

Add to directory context menu:

QUOTE(contxt_mnu.reg @ double click)

Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\Directory\shell\MD5 Check\command]
@="md5c.bat"
[HKEY_CLASSES_ROOT\Directory\shell\MD5 Create\command]
@="md5.bat"

goweropolis

Apr 5 2003, 13:48

Thanks for the suggestions R.A.F. & ak. I really appreciate it.

Wow, bat files can do just about anything!

R.A.F.

Apr 7 2003, 01:38

Arman, this weekend I found the time to try your script. - But unfortunately it didn´t work. There just opens for just a moment the dos-window of fsum, and then it disappears again.
My syntax by starting the script was
f:\replaygain-directory\md5_sum.cmd
and all the directories, which I wanted to checksum, stood in this f\replaygain-directory. Also the fsum.exe-file. I also clicked on that md5_sum.cmd-file - but with the same negative result. What did I do wrong?

And I don´t manage to generate with cSFV (by Cruzer) SFV-checksum-files. Seems that I´m a too stupid for this program....

arman68

Apr 7 2003, 04:31

QUOTE(R.A.F. @ Apr 7 2003 - 07:38 AM)

Can you give me some more info? I am not sure I understand how you started the script. What I think you should do is:

1. put md5_sum.cmd and fsum.exe into your system path (eg: C:\WINDOWS )
2. open a command prompt
3. type: md5_sum "F:\replaygain-directory" *.mp3

(only 1 paramater, and it will then recurse through all the subdirectories). You can use * instead of *.mp3; in that case it could be a good idea to add a line adapted from ak suggestion to remove the self entry from the checksum file. I cannot do it right now, but if you want it I will have a look tomorrow.

Also, what OS are you using?

R.A.F.

Apr 7 2003, 05:41

Okay, I think I know what I´ve done wrong now: I forgot to add the file-ending "*.mpc" to the command-line. I´ll check it out the next few hours and tell you if it worked.

I use WinXP + SP1. So, my OS shouldn´t be the problem.

Update:
----------
Okay, I checked it out. Works great now. Even the filename from directory is taken over correctly.
Thanks a lot, Arman ! - Also for the fast reply. It saves me hours of real ultra-stupid work!

Maybe I´ll upload for all the other users, who ripp albums etc. this slightly modified script. It´s really a very useful thing.

R.A.F.

Apr 7 2003, 07:02

Two little things still are going wrong.... If there is the character "&" in the directory-name, the script refuses to generate the md5-file. For example in "Bob Marley & The Wailers - xxxxx" etc. And those dir-names appear quite often. Maybe it´s also the fault of fsum.exe. I don´t know. Same with the character "%".
And the second is, that if a point "." is in the dir-name, it stops naming the md5-checkfile after the point. E.g.: "Kendo Edition No. 5" becomes then "Kendo Edition No.md5", instead of "Kendo Edition No. 5.md5".

- Any idea how to correct these bugs, Arman ?

Jasper

Apr 7 2003, 09:55

If anyone is interested I have put a little program on my site (http://home.hccnet.nl/th.v.d.gronde/dev/misc/index.html) that outputs a file with filenames and hashes in such a way that it is easy to compare it to an older version (the filenames are sorted).

arman68

Apr 8 2003, 02:22

QUOTE(R.A.F. @ Apr 7 2003 - 01:02 PM)

I remember noticing the problem with & in filenames a while ago. I was not aware of % but that does not surprise me since it has a special meaning. The problem is with the Microsoft .cmd language. I will see if it can be fixed using `, ' or any other mean.

Same for the . in the directory name: I am using the built-in function that keeps the filename only. Therefore anything after the last . is considered as an extension. Since this is only used on directories, I will modify it to keep the extension as well. There might be a few problems with inserting an extra . when there is none in the directory name, but nothing that cannot be done.

Apr 8 2003, 02:48

Regarding dot in dirname,
SET md5file=%~nx1.md5
should do.

arman68

Apr 8 2003, 14:34

QUOTE(ak @ Apr 8 2003 - 08:48 AM)

Regarding dot in dirname,
SET md5file=%~nx1.md5
should do.

I think there might be a problem with this: when there is no dot in the dirname it would add one at the end. I will check it tomorrow.

Apr 8 2003, 14:35

Nope it doesn't.

_io_

Apr 9 2003, 12:26

Heres an installer that will do the job for you, its just some batch files and fsum wrapped in an installer. It will work with '&' and '.' in directory names. It works from the context menu, just right click on the directory that you want worked on.

md5.exe

Let me know of any problems

_io_

R.A.F.

Apr 11 2003, 10:12

Thanks, io, very much. I already MD5´d several hundreds of directories now. Well, I didn´t look so exactly on them, but everything seems to work. Have to take a closer look at directories, where the percent-char ("%") is written, to see if it works also with this character. One little thing should *maybe* be changed for me.... I´d like to have the "_"-sign in front of each MD5-filename, so that it´s sorted at the beginning of each dir. Can you upload a special version for me with this feature? Thx a lot !!

_io_

Apr 11 2003, 11:31

Here you go, i'd advise uninstalling the previous one before installing this one.

MD5 for R.A.F

_io_

R.A.F.

Apr 16 2003, 09:19

Thanks a lot, io.

R.A.F.

May 23 2003, 16:04

And another thing I noticed should be also included, _io_: Is it possible, that this MD5-program is generating checksums only for special file-types (like *.mp3, *.mp3, *.ape), which you should be able to choose before the gerneration-process starts? At the moment there are simply all files in each directory included, which I don´t want.

I hope you will ever read this, after the project closed such a long time ago....

This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.