IPB

Welcome Guest ( Log In | Register )

9 Pages V  < 1 2 3 4 > »   
Closed TopicStart new topic
Near-lossless / lossy FLAC, An idea & MATLAB implementation
2Bdecided
post Jun 13 2007, 18:59
Post #26


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



SebG,

Ah, I see. Well, it might be interesting to try. It sounds like you're tempted to do it! wink.gif


Josh PM'd me to point out that the FLAC frame/block size can be set from the command line using the -b command. I've just tried it, and it works as expected: with the MATLAB code changed to use 1024 sample blocks, it seems I get better compression, but I need to try more samples. On the one I tried (41_30sec) this shaved another 20% off, though that sample encodes 3% more efficiently in 1024 sample blocks than 4096 blocks anyway.

Cheers,
David.
Go to the top of the page
+Quote Post
jcoalson
post Jun 13 2007, 19:07
Post #27


FLAC Developer


Group: Developer
Posts: 1526
Joined: 27-February 02
Member No.: 1408



QUOTE (pepoluan @ Jun 13 2007, 11:21) *
Slightly offtopic: Where'd you get the gspflac.dll??? I wanna!11!!! w00t.gif
https://sourceforge.net/project/showfiles.p...group_id=165460

QUOTE (SebastianG @ Jun 13 2007, 11:42) *
Cool!. Could you clarify the "Notes" paragraph in the frame header section, please? What blocksizes are allowed if it's a variable length block stream? I'd use "1000-1111 : 256 * (2^(n-8)) samples" but it looks like they are not allowed.
all that convoluted logic will be going away with the next version of FLAC thankfully. I'll publish the details with the next release of FLAC (hopefully no later than july)
Go to the top of the page
+Quote Post
goodnews
post Jun 13 2007, 19:36
Post #28





Group: Banned
Posts: 232
Joined: 20-January 06
Member No.: 27228



QUOTE (jcoalson @ Jun 13 2007, 12:07) *
all that convoluted logic will be going away with the next version of FLAC thankfully. I'll publish the details with the next release of FLAC (hopefully no later than july)

Slightly OT: Josh, is there a Intel Mac OS X version of FLAC 1.1.4 for download yet? All is still see is the PPC Mac version. Thanks!

What features will be included in next version of FLAC that you mentioned?

This post has been edited by goodnews: Jun 13 2007, 19:37
Go to the top of the page
+Quote Post
halb27
post Jun 13 2007, 21:03
Post #29





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



I tried all the samples you provided, and couldn't abx them except for the second half of furious:
With my first trial I got at 4/4 with furious, but I missed several times with the following guesses.
With my second trial I got at 6/6, then 7/8, finally 8/10. Not a totally convincing result but maybe enough to show that furious lossy isn't totally transparent.
This is pretty similar to what I have learnt from wavPack lossy behavior for furious.
Anyway the difference is so subtle I can't really describe it. Just a minimal lack of precision may be. Not serious at all.

Can you provide Atem-lied, herding_calls, trumpet, harp40_1 please? These and all the other samples in the presumably more efficient short block version?

ADDED:
I forgot badvilbel. Can you provide badvilbel please?

This post has been edited by halb27: Jun 14 2007, 08:09


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
Bourne
post Jun 13 2007, 23:40
Post #30





Group: Banned
Posts: 735
Joined: 19-March 06
Member No.: 28599



I kinda talked about this once before... I called it Virtual Lossless...
But unfortunately someone cut me out saying: Lossy is virtual lossless...

Thanks for your detailed explanation.
And I see a diference between LOSSY and VIRTUAL LOSSLESS.
Go to the top of the page
+Quote Post
Mitch 1 2
post Jun 14 2007, 10:35
Post #31





Group: Members
Posts: 31
Joined: 3-October 06
From: Australia
Member No.: 35904



Using a two-part file extension (e.g. .lossy.flac) should solve the compatibility problem, at the expense of longer filenames. Proper tagging is needed, of course, as filenames alone cannot be trusted.


--------------------
lossyFLAC (lossyWAV -q 0; FLAC -b 512 -e)
Go to the top of the page
+Quote Post
2Bdecided
post Jun 14 2007, 11:53
Post #32


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (halb27 @ Jun 13 2007, 21:03) *
I tried all the samples you provided, and couldn't abx them except for the second half of furious:
With my first trial I got at 4/4 with furious, but I missed several times with the following guesses.
With my second trial I got at 6/6, then 7/8, finally 8/10. Not a totally convincing result but maybe enough to show that furious lossy isn't totally transparent.
This is pretty similar to what I have learnt from wavPack lossy behavior for furious.
Anyway the difference is so subtle I can't really describe it. Just a minimal lack of precision may be. Not serious at all.


Thank you for ABXing halb27.

I thought I could ABX the background noise at the end of Furious, but then failed. I'm not sure if I'm imagining it, or if it's nearly audible.


If you're in the mood to play, please try the attached files. I've played with the thresholding. I've also (intentionally) broken the lossy part by dithering the LSB itself so you can't cheat and look at the FLAC bitrate!

At least one of these files has more noise (so it's not as hard a job as it looks). At least one has less noise. So you should be able to ABX at least one, and maybe cannot ABX at least one. See what you think.

If you do have time to ABX, please decide upon the number of ABX tests before you start, and stick to that. As you probably know, selecting results or re-starting messes up the statistics.

Of course anyone is free to try.


QUOTE
Can you provide Atem-lied, herding_calls, trumpet, harp40_1 please? These and all the other samples in the presumably more efficient short block version?

ADDED:
I forgot badvilbel. Can you provide badvilbel please?


I'll do those as time permits. If I get chance before you've responded, I'll post them at the default settings. However, if your next response suggests I need to reduce the noise addition slightly, then I'll post them with a less aggressive setting.

Cheers,
David.
Attached File(s)
Attached File  1_Furious.flac ( 427.89K ) Number of downloads: 261
Attached File  2_Furious.flac ( 428.99K ) Number of downloads: 245
Attached File  3_Furious.flac ( 432.18K ) Number of downloads: 278
Attached File  4_Furious.flac ( 428.62K ) Number of downloads: 248
Attached File  5_Furious.flac ( 440.95K ) Number of downloads: 252
Attached File  6_Furious.flac ( 458.38K ) Number of downloads: 258
 
Go to the top of the page
+Quote Post
robert
post Jun 14 2007, 13:11
Post #33


LAME developer


Group: Developer
Posts: 783
Joined: 22-September 01
Member No.: 5



foobar has some problem with the sample 1_Furious:
CODE
Decoding failure at 0:01.486 (Unsupported format or corrupted file):
"F:\1_Furious.flac"


edit: Sorry, the downloaded file was broken on my side. I downloaded it again and Foobar plays it just fine.

This post has been edited by robert: Jun 14 2007, 13:17
Go to the top of the page
+Quote Post
halb27
post Jun 14 2007, 13:14
Post #34





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



Will try them tonight.

BTW I don't concentrate on the background noise but on the accuracy of the 'main signal' in the second half of the track.

As for abxing if the question is 'is track X transparent?' I always allow for a second trial in case I have the impression that there is a difference and get at a result like 4/4 before I start to go wrong (due to possibly fatigueness). The number of guesses is fixed before a test (usually 10 guesses, sometimes 8), but in case I don't see what to concentrate on I often give up within the first guesses (usually with several wrong guesses at that time).
Of course my insisting on going through the test also depends on previous experience with the sample. From furious I know it's a serious problem for wavPack lossy and I have an idea what to look for. I'm also more emotionally engaged cause this is one of the more serious problems to wavPack lossy - I can imagine to get a similar kind of music in real life encoding, and it's not just a tiny amount of altered or increased hiss/noise but inaccuracy - though in your case very tiny as well.


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
2Bdecided
post Jun 14 2007, 13:49
Post #35


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (halb27 @ Jun 14 2007, 13:14) *
As for abxing if the question is 'is track X transparent?' I always allow for a second trial in case I have the impression that there is a difference and get at a result like 4/4 before I start to go wrong (due to possibly fatigueness). The number of guesses is fixed before a test (usually 10 guesses, sometimes 8), but in case I don't see what to concentrate on I often give up within the first guesses (usually with several wrong guesses at that time).
I'm sure an ABX / statistics guru will be along in a minute to explain exactly why that alters the statistics. I just recall that it does, and in a way that makes it much harder to hit a given level of confidence.

I think you could say something like "I will do 8, and they will not count, then I will do 16, and they will count" if you stuck to it.

Of course, you can always listen carefully, and A/B (not X!) until you believe you hear something. Then, and only then, do the pre-decided number of ABX trials. Then there's no question.

Cheers,
David.
Go to the top of the page
+Quote Post
Mark0
post Jun 14 2007, 13:51
Post #36





Group: Members
Posts: 69
Joined: 15-August 02
From: Venice
Member No.: 3068



I may be misunderstanding something, but: why linking this to FLAC at all?
I mean: this is a "sound simplifier", so to speak, so it's output could be very well fed to pretty any lossless (or even non lossless, even if this does a lot of less sense) coder, right?

Bye!
Go to the top of the page
+Quote Post
Ariakis
post Jun 14 2007, 14:28
Post #37





Group: Members (Donating)
Posts: 145
Joined: 26-March 03
Member No.: 5677



It was originally designed to specifically exploit FLAC's wasted_bits handling for compression gain.
Go to the top of the page
+Quote Post
naturfreak
post Jun 14 2007, 14:59
Post #38





Group: Members
Posts: 175
Joined: 16-October 03
Member No.: 9338



My suggestion to further prevent confusion whether a FLAC file is from a lossless or lossy source:
Introduce a flag inside the FLAC (meta)data that indicate whether a file has a lossy or lossless source.

An user should be able to set that flag at encoding time only. It should be noneditable and unerasable inside the FLAC file (hex editor might be an execption).

This post has been edited by naturfreak: Jun 14 2007, 14:59
Go to the top of the page
+Quote Post
2Bdecided
post Jun 14 2007, 15:12
Post #39


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (Ariakis @ Jun 14 2007, 14:28) *
It was originally designed to specifically exploit FLAC's wasted_bits handling for compression gain.


Exactly.

However, it turns out you can use it with wavpack lossless too, by using the --blocksize=1024 switch to force wavpack to use 1024 sample blocks. (or 4096 sample blocks for the examples I provided on the previous page). Thanks to David Bryant for providing this, and other useful tips via email (I will reply properly David).

Cheers,
David.

This post has been edited by 2Bdecided: Jun 14 2007, 15:13
Go to the top of the page
+Quote Post
halb27
post Jun 14 2007, 15:39
Post #40





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (2Bdecided @ Jun 14 2007, 14:49) *
... I think you could say something like "I will do 8, and they will not count, then I will do 16, and they will count" if you stuck to it. ...

Hm... it's not like this: I say in advance I'll do 10 guesses with each trial in order to call two tracks abxable.
What shall I do in a situation when I have the impression (which doesn't count in the end but I can't ignore it) that there are audible differences, and this is backed up by the first guesses where I score 4/4? If after that I fail what does that mean? Failure can be due to the tracks not being able to abx, but also due to fatigueness or overconfidence according to the first results. Certainly this means I'm not very good at abxing, but with tracks hard to abx it happens to be like this - I can't change it. So what shall I do in such a situation? My solution is: I do a second trial and try harder. Can't see a better procedure. Taking the result of the first trial as the abxing result isn't the better alternative to me in case there's a suspicion that the tracks are abxable.
Sure if I allow for a second trial I could also allow for a third one, and so on. I see the point. But that's theory cause things are clear after the second trial.

This post has been edited by halb27: Jun 14 2007, 20:05


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
Mark0
post Jun 14 2007, 15:40
Post #41





Group: Members
Posts: 69
Joined: 15-August 02
From: Venice
Member No.: 3068



QUOTE (2Bdecided @ Jun 14 2007, 16:12) *
QUOTE (Ariakis @ Jun 14 2007, 14:28) *

It was originally designed to specifically exploit FLAC's wasted_bits handling for compression gain.


Exactly.

However, it turns out you can use it with wavpack lossless too, by using the --blocksize=1024 switch to force wavpack to use 1024 sample blocks. (or 4096 sample blocks for the examples I provided on the previous page). Thanks to David Bryant for providing this, and other useful tips via email (I will reply properly David).

Cheers,
David.

Exactly. What I was meaning (sorry, my English isn't very good) is that the "simplification" made by your tool can enable similar gain in compression ratio with other similar tools. For example (using one of the posted sample):

CODE
                      WAV     FLAC      TAK       RAR
01_41_30sec_lossy   5.168KB  1.957KB  2.119KB   2.755KB
02_41_30sec         5.168KB  3.473KB  3.284KB   3.633KB


Maybe two hours from now a new Lossless compressor will surface that enjoy an even higher gain than FLAC. In the end, I don't see the link between a generic tool that "simplify" a sound file, and a particular encoder (for witch the action of the tool may result especially beneficial).

On the same way, I don't see the point about flaggin in some way a lossless encoded file that had as a input a WAV file altered in some way.
Even without using this tool, original files can be altered in a number of ways (badly equalized, or they can contains click/pop, can have some noise reduction effects applied, etc.)...

Bye!

This post has been edited by Mark0: Jun 14 2007, 15:49
Go to the top of the page
+Quote Post
halb27
post Jun 14 2007, 15:41
Post #42





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (2Bdecided @ Jun 14 2007, 16:12) *
... However, it turns out you can use it with wavpack lossless too, by using the --blocksize=1024 switch to force wavpack to use 1024 sample blocks. ...

Wonderful.
Your idea is getting even more useful. Congratulations.


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
2Bdecided
post Jun 14 2007, 15:45
Post #43


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (halb27 @ Jun 13 2007, 21:03) *
Can you provide Atem-lied
I can't find it. Can you upload it please?

Cheers,
David.
Go to the top of the page
+Quote Post
Nick.C
post Jun 14 2007, 16:35
Post #44


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



So, how soon before an executable version of SoundSimplifierô ( wink.gif ) is released to an expectant HA community? Inquiring (impatient) minds want to know!


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
2Bdecided
post Jun 14 2007, 16:44
Post #45


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



I've uploaded herding_calls, trumpet, harp40_1, and badvilbel.


Here the lossy versions, and also links to the originals:

badvilbel:
Attached File  badvilbel_lossy.flac ( 1.38MB ) Number of downloads: 340
http://www.ff123.net/samples/badvilbel.flac

harp40_1:
Attached File  harp40_1_lossy.flac ( 860.51K ) Number of downloads: 400
ftp://ftp.tnt.uni-hannover.de/pub/MPEG/au...am/harp40_1.wav

herding calls:
Attached File  herding_calls_lossy.flac ( 640.72K ) Number of downloads: 340
http://www.hydrogenaudio.org/forums/index....showtopic=37002

trumpet:
Attached File  trumpet_lossy.flac ( 590.97K ) Number of downloads: 360
http://www.hydrogenaudio.org/forums/index....showtopic=39334


(I miss the HA problem samples library!)

Cheers,
David.

This post has been edited by 2Bdecided: Jun 14 2007, 16:45
Go to the top of the page
+Quote Post
TBeck
post Jun 14 2007, 16:54
Post #46


TAK Developer


Group: Developer
Posts: 1095
Joined: 1-April 06
Member No.: 29051



QUOTE (Mark0 @ Jun 14 2007, 15:40) *
...
Exactly. What I was meaning (sorry, my English isn't very good) is that the "simplification" made by your tool can enable similar gain in compression ratio with other similar tools. For example (using one of the posted sample):

CODE
                      WAV     FLAC      TAK       RAR
01_41_30sec_lossy   5.168KB  1.957KB  2.119KB   2.755KB
02_41_30sec         5.168KB  3.473KB  3.284KB   3.633KB


Maybe two hours from now a new Lossless compressor will surface that enjoy an even higher gain than FLAC. In the end, I don't see the link between a generic tool that "simplify" a sound file, and a particular encoder (for witch the action of the tool may result especially beneficial).
...

For optimum performance the intenal frame size of the encoder has to be taken into account when using the preprocessor. I suppose you haven't done this for TAK?

Possibly i will have to add an option to manually set the frame size for TAK files to get most out of the preprocessor. While TAK partitions each of it's fixed size frames into up to 5 variable size sub frames to adapt to signal changes, the "wasted bits" options works on the whole frame, which is too big (more than 4000 samples) to work well with the preprocessor.

Thomas

This post has been edited by TBeck: Jun 14 2007, 16:58
Go to the top of the page
+Quote Post
2Bdecided
post Jun 14 2007, 16:59
Post #47


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (Nick.C @ Jun 14 2007, 16:35) *
So, how soon before an executable version of SoundSimplifierô ( wink.gif ) is released to an expectant HA community? Inquiring (impatient) minds want to know!


I like the name!

I'm not keeping it back. I've attached the latest MATLAB script which I'm using to generate these samples. It executes very well if you have MATLAB!
(Though you need lots of memory for normal sized audio files, since there's no buffering, and you'll need to change waveread to wavread and wavewrite to wavwrite).

As for an efficient C/C++ implementation which could be compiled - that's beyond me.

I think we need some more listening tests before anyone puts that much effort in, but the job is open to anyone who wants it!

Cheers,
David.
Attached File(s)
Attached File  lossyFLAC2.txt ( 9.42K ) Number of downloads: 258
 
Go to the top of the page
+Quote Post
smok3
post Jun 14 2007, 18:12
Post #48


A/V Moderator


Group: Moderator
Posts: 1709
Joined: 30-April 02
From: Slovenia
Member No.: 1922



i couldnt reliably abx the 1st set of samples, but there were some weird negative results on some like 1/8 or 2/8.

This post has been edited by smok3: Jun 14 2007, 18:13


--------------------
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung
Go to the top of the page
+Quote Post
TBeck
post Jun 14 2007, 18:12
Post #49


TAK Developer


Group: Developer
Posts: 1095
Joined: 1-April 06
Member No.: 29051



QUOTE (TBeck @ Jun 14 2007, 16:54) *
QUOTE (Mark0 @ Jun 14 2007, 15:40) *

...
Exactly. What I was meaning (sorry, my English isn't very good) is that the "simplification" made by your tool can enable similar gain in compression ratio with other similar tools. For example (using one of the posted sample):

CODE
                      WAV     FLAC      TAK       RAR
01_41_30sec_lossy   5.168KB  1.957KB  2.119KB   2.755KB
02_41_30sec         5.168KB  3.473KB  3.284KB   3.633KB


Maybe two hours from now a new Lossless compressor will surface that enjoy an even higher gain than FLAC. In the end, I don't see the link between a generic tool that "simplify" a sound file, and a particular encoder (for witch the action of the tool may result especially beneficial).
...

For optimum performance the intenal frame size of the encoder has to be taken into account when using the preprocessor. I suppose you haven't done this for TAK?
...


I was curious...
I dowloaded "01_41_30sec_lossy" from here. Then i compressed it with TAK's default frame size and then with a frame size of 4096 Bytes. Results:

CODE
FLAC                2,004,157  Bytes
TAK Normal Default  2,023,188  Bytes
TAK Normal 4096     1,809,281  Bytes
TAK Turbo  4096     1,846,469  Bytes


This post has been edited by TBeck: Jun 14 2007, 18:19
Go to the top of the page
+Quote Post
Mark0
post Jun 14 2007, 18:36
Post #50





Group: Members
Posts: 69
Joined: 15-August 02
From: Venice
Member No.: 3068



QUOTE (TBeck @ Jun 14 2007, 17:54) *
For optimum performance the intenal frame size of the encoder has to be taken into account when using the preprocessor. I suppose you haven't done this for TAK?

Right. I have just compressed the two files "on a rush" to check if - as I supposed - the action of the SoundSimplifier™ smile.gif would be interesting also for other encoders.
And it was, so much, as your results show even more. Thanks for taking the time for the "optimized" encoding.

Bye!

This post has been edited by Mark0: Jun 14 2007, 18:37
Go to the top of the page
+Quote Post

9 Pages V  < 1 2 3 4 > » 
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th April 2014 - 12:04