The rumored test is finally here!! http://twolf1300.net/transcode/

With the rise in popularity of hardware mp3 players, mp3 has become the standard by which people store their CDs onto their computers. Unfortunately, the convenience of hardware players has discouraged the adoptment of other potentially better codecs, such as Ogg Vorbis or Musepack. Those who have a hardware mp3 player must choose between storing their music with either lossless codecs or in mp3. A third choice is to transcode from the other codecs. This is considered blasphemy in audiophile circles. The objective of this test is to help determine whether that notion is valid.

For those who were not aware of the discussion preceding this test, please read up http://www.hydrogenaudio.org/forums/index....topic=11734&hl=.

Thanks,
-Steve

sounds interesting
(though I'm probbably one of the ppl where point 1. comes true... )

Cool. B)

Sheesh...it's about time! (JUST KIDDING!)

I think with the release of several new portables that actually play something other than MP3, that transcoding is becoming more relevant. I've noticed more interest/questions about it in the past few weeks especially (the PC release of iTunes was a significant factor as well). Many people in the world have MP3 collections, and may want to transcode to another format or to a lower bitrate for their portable.

Thanks for the effort you put into this, Steve. I'll do my best to participate before the close date.

The following offsets should be appended to each config file:

Offset2 = 25
Offset4 = 125

These offsets are rather large, so they have the potential to cause ABX to have false positive results.

wavgain did a good job of leveling the volume.

ff123

Well I hope transcoding from Lame--alt-preset standard does well cause that's what I transcode from for my portable player.

The instructions say, in part:

"Do not worry if you can't tell the difference between the files, but please attempt to distinguish between the two files using the ABX portion of the ABC/HR program. ALL RESULTS THAT DON'T HAVE EVIDENCE OF AN ABX ATTEMPT WILL BE THROWN OUT."

There are 5 files plus a reference. We are supposed to ABX each of the 5 files against the reference, correct? How many ABX trials are we supposed to perform at minimum? I presume we are also supposed to rate the 5 files against each other in the ABC/HR portion?

ff123

Two questions:

1. How do I use the offsets referred to by ff123? Do I just add these to each config file.

2. Is it absolutely necessary to ABX each one? I'm one of the guilty ones who doesn't bother ABXing in a ABC/HR type test if it is incredibly obvious, which it has been with most of the tests done here recently... I've got it right every time during the last lot of tests.

Mind you it might not be the case this time around...

54 tests that's a bit much for me, I'll just say I found very small differences in a couple and couldn't tell the difference on the rest of the ones I listened to.

The start file has been updated already at my site. Hopefully Rjamorim will update his site soon. I don't know HOW ff123 managed to find the offsets, but I appreciate it.

As for ABX'ing, I think that a good minimum value is 8. There is at least 1 obvious crap file (I'm not saying which and I'd rather no one else did either). 8 trials for that one will take you all of 2 minutes tops. Since I know which one the crap sample is, I'll be forgiving/understanding if there isn't an ABX score associated with that file.

The rest are sufficiently difficult enough that you'll probably need 8 trials. Without that information, you could be just guessing (which is perfectly alright and there's no shame in this), but it won't be shown in the results. Like I said, the ABX attempts are more important than the actual score that you guys give. This isn't a "How bad is the difference test?" but more of a "Is there an audible difference?"

I didn't emphasize this enough in the start file, but I'm perfectly aware that 54 tests is a friggin' s***load.

But I'd rather you do (and send the results) to just ONE sample than none. One sample amounts to 6 tests, which will probably take about 30-60 minutes to do. And I can push back the test date to accomodate anyone who's working on it (assuming enough people ARE in fact working on it).

Very good work. I'll start testing some samples later tonight and try to finish as many as possible.

I updated it half an hour ago.

I used CoolEdit to inspect the length of the silence at the beginning of each file. However, one should also be able to use Schnofler's automatic offset finder to get the offset values.

ff123

Edit: I tried using Schnofler's offset finder (located in his abchr-java app in the "Setup Test" menu), and it came up with the exact same offsets.

When I checked with the Java ABC/HR program that I had, there was no offset calculator. But I just downloaded it again, tried it and there it was; and it worked.

The older one was included in the start file. I've updated the start file to include the newer version of Java ABC/HR and I've corrected some of the bash scripts I wrote (I can't believe I let those go by for this long).

The old start.zip file can be found at http://twolf1300.net/transcode/startOld.zip if you want to check out the older version of Java ABC/HR.

Just updated the start.zip package at my server again.

I downloaded the whole transcode test and zipped everything into one 35 MB archive. This file is now available through EDonkey / EMule / ... - perhaps it helps saving some bandwidth for RareWares:

ed2k://|file|Transcode.Test.zip|36309672|448B2EC30BB393B490DFAFC53DCF5277|/

Regards, fileman.

Edit: removed URL tag - ED2K-links aren't supported by the board software.

Should this thread be in Listening Tests area (also)?

why don't we use BitTorrent? It could very well save a lot of bandwith to dibron ... for fb2k also ... rarewares, etc ...

PS: BT is way faster than emule/edonkey ... i never get more than 10kb and 30 with BT ...

Feel free to use my BitTorrent Tracker:

Haha.. with 3 people sharing the file in either eMule or BitTorrent, the speed will just depend on their bandwith.. assuming those with eMule powershare the file..

wouldn't be better to test transcoding from LAME MP3 Encoder 3.90.3 --alt-preset xtreme instead (or also) from LAME MP3 Encoder 3.90.3 --alt-preset standard ?

These issues were discussed before in another thread. It probably isn't useful bringing them up here after the test is underway.

I think the various options were chosen based on typical use here at HA, and the general impression one gets is that --alt-preset standard is used more often here for lossy storage than xtreme, but I could be wrong.

Also, in my own transcoding experience, a sample that transcodes poorly from standard will also have problems from xtreme and insane, but hey you never know. My own testing was only on a limited number of samples.

I did choose the settings based on typical usage at HA. The only exception being MPC --insane, which I don't think is as popular as --standard. That particular setting was chosen only to see whether increased bitrates will ensure a transparent transcode.

As such, I didn't think that an mp3 -> mp3 transcode was a terribly good idea, but the results will be interesting, especially in comparison with the others. Most can't ABX Lame standard anyways, except with problem samples. Even then, problem samples don't always get better with xtreme.

I believe your batch files have a small problem with the last line, they don't wavegain anything (at least in my system)

two last lines from Sample01.bat:



I think it should be




Cheers, Joey.

[Edit: running WinXP]

Even if wavgain isn't doing anything, I don't think the batch files need to change. I checked the average rms levels of the left and right channels separately for each transcode plus the reference in one of the samples, and the volumes were the same within a tenth of a dB.

ff123

I'm averaging more updates than microsoft at this point.
Guess what I just did?

I wonder if I'm too scatterbrained to conduct this test.

Damn. I just found out that i don't have time to participate this test. But this war doesn't need one man...

Like I said, I can extend the test if need be. Right now, I have ZERO results, so if you need more time, I can easily extend the test end.

till when would you like to extend it?

I realy don't know next time when can i get my computer connected to internet after tomorrow. I'm moving now in new place and i just made contrackt with new ISP. They say that normaly it takes 2-5 weeks until connection is open...
But maybay i can download test pakage now and someday try to send results from somewhere. I realy like to put my share to this test, but don't whait for me.

Probably 12:00AM Dec. 1st. That gives over a month of time on very difficult material. After that, I'll either have plenty of results, or I'll call the test a failure due to lack of results. I choose that date specifically, because that's when I will have returned from the US holiday, Thanksgiving. Granted, that may be TOO much time, but I don't want to deal with the results immediately prior to my holiday.

Based on the likely amount of ABXing for each test group that will be required for this test (as discussed earlier), I'd also vote for an extension if it wouldn't cause you any schedule conflicts, personal or otherwise.

Also, as of now I'm not positive that I'd have time to complete and submit even one set of test results by the original end date, but if you extended to Dec. 1st, then I'm sure I could participate. My interest is high, but my time is somewhat constricted here lately.

And thanks for offering to accommodate those of us who may need more time!

I was going to conduce a Vocodec test by the end of this month, but now I'll probably reschedule it to January.

Roberto and Phong are the only ones I'm worried about in terms of scheduling conflicts. They're the ones with listening tests in the pipeline and the further I push back the test end date, the more they have to wait.

Personally, I'd be perfectly happy letting you guys take as long as you want, but there needs to be some obvious test end date so that Roberto and Phong can get their respective tests rolling.

I am confused about something... According to your test page, 5 codecs/settings are being tested (MPC @ -q5 & -q7, vorbis -q6, lame -aps, and wavpack lossy -320) against a direct encode to --preset medium, right?
Then why are there samples xxx_1.mp3 through xxx_6.mp3 PLUS an xxx.mp3 and six test per sample? Does this mean that one of the samples is actually another reference (direct encode)? If so, I think that is MEAN plus completely unnecessary; since that means six VERY difficult (or rather IMPOSSIBLE ) extra tests to prove what? That you can't ABX something against itself? I think its of no use either to discard people who take the test sloppily or can't follow instructions. Those will simply show up with ranked references or lack of ABX results. For those of us that are putting some effort to ABX the samples, I think it would be an utter waste of our time to put us to test something for as long as our patience lasts (which can be quite a while for some) just to get 5.0s.
Or is it just that there is a sixth codec being tested that you forgot to note down on the page? I kinda hope so.

I thought I mentioned that somewhere, but I guess I haven't. There is a 6th sample that's just a transcode of preset medium to itself several times. The point behind this is to give at least one sample that should be audibly different. A lower anchor. That's the crap setting that you guys should be able to ABX in about 2 minutes (counting setup time). If you CAN'T identify the crap sample, then you might want to reconsider taking this test.

Well, if you didn't before, you have some now ....although 1 test on one sample isn’t really statistically valid, now is it?

friedman.exe determines whether the results are statistically valid or not. But I would reckon that at least 10 results per sample are needed.

This test would be a lot easier with different samples.....I've done BlueMonday, VirtualReality and Waiting so far with pretty good results, but samples such as Talula sound pretty good at --alt-preset cbr 112 to my ears.....

edit: sthayashi - could we have an update on how many results you have so far?

[strongbad]A message to the peoples[/strongbad] : Come on! We need results to, ah, "seal the deal." - participate, it's hard, but I want to see this test succeed!

You bet. I have 3 results. I wish I had more. Hopefully I will after this next weekend.

I figured I would give people at least two weekends to try, but if there is enough demand, like I stated before, I'll extend the test until the end of the month.

Finally got around to downloading the samples... Two of the .sh files have the wrong filenames in them (01 and 04).

Also, has anyone ever gotten wavegain to compile under linux? I suppose the volume differences are neglegible here...

To quote a ninjai character, FAAAAHHHK. I knew about it and thought I had fixed it with one of the updates (in fact I'm damn near positive I did, so I wonder WTF is up).

The start.zip file has ONCE again been updated.

IIRC, some people managed to successfully compile it under OSX, so I'd imagine that Linux wouldn't be that difficult.
Thanks for participating.

@sthayashi
I hope you are still considering going to the end of the month. I'm tight for time here at the moment.

Den. >_<

Since a few regulars have asked for an extension, I shall grant them one. The test end date has been extended to Nov 30th.

Due to lack of interest and few results, the transcode test is now officially closed.

Feel free to contact me if you want to be tested with your own parameters (i.e AAC->mp3 or what have you).

Oh, man. That's too bad

I'm soo sorry. I haven't had time and had only gotten a couple of the samples done and didn't realize the deadline was the 30th. I hope you weren't like one or two result sets away from having enough.

I'm really sorry Steve. I had pushed myself to participate in this test, but my workload increased about threefold, and I never had enough time available. My time online and my participation everywhere else dropped a lot too in the past few weeks. The holidays get like this every year, with projects at work having converging deadlines, more people on vacation and fewer of us to pick up slack.

If you keep the links to the test packages working, then could people run the tests as they get time (more likely after the holidays)? Or without a time restriction would it be a pain to manage?

Well, I need to update the webpage, which I'll do when I return home from work. As is, I already have the files available on my site, but I was worried about exceeding my bandwidth limit. I guess that was an unfounded fear.

If you have a results file (or several results), please send them to me. I will respond to you and let you know what you could and could not hear.

I guess I shouldn't say that the test is cancelled, exactly, but the official end date is now undefined, and I won't publicly release results until I have enough to be statistically significant. Given the number of results I have now, it's not clear if I will ever have enough.

If you do contribute your own results, I'll tell you what you got and how it compares to others. At least that way, people will still have some incentive to participate. I will make some public generalizations about what I do have:
a) Every codec was successfully ABX'ed, to a limited degree (worst case was a pval of 0.006)
b) Half the results was what I expected. The other half was not.