Hey guys,

Is there any way to edit FPL files if the music files they referred to have been moved to another directory? I tried doing a simple find and replace and changing the paths in the FPL file, but it didn't work. Instead, it gave me an error saying the format was not supported, and the path was doubled. For example, instead of listing the path as "C:\Music\abcd\01.mp3", the path in the error message would be "C:\Music\abcd//Music\abcd\01.mp3" or something along those lines.

What am I missing?

Thanks,

nazgulord.

When you edit strings in an FPL file, you must preserve their lengths.

It's the old question of not having plain text files in foobar (the config file, the fpl files...)

Nazgulord shows with his example, why text files would be an advantage. Having binary files is an unnecessary barrier imo.

Without those binary playlists, loading and management of huge playlists would probably be very slow.

Perhaps I miss sth. but why not simply save the fpl playlist in the m3u or m3u8 playlist format and then edit it?

Text files are cumbersome to parse, unnecessarily large and require elaborate escape sequences to code some characters. If I am not mistaken, FPL can store all the tags of each file, including binary fields and fields with new-lines etc which would be awful in a text file.

If you are not mistaken: True, agreed.

Thanks for the replies, guys.

Foosion, I was curious as to what you mean by preserving the lengths of the strings? Does that mean that if the unedited version was 25 characters long, the edited one would also have to be that long?

nazgulord.

Correct.

If you need binaries to be stored in text, do a Base64 conversion. Libraries for converting from/to Base64-encoded binaries are freely and widely available.

Besides, if processing a text-based FPL takes 5 seconds, while processing a binary-based FPL takes 2.5 seconds... I would go with text-based all the time. 2.5 seconds is not *that* significant.

Geez.


Show us some quantitative timing and I'll believe you.

But, even a 100% improvement like I pointed above, is of no significance as the order of time is still seconds.

It will be different if, for instance, processing binary-encoded files takes 1 second and processing text-encoded files takes 3 hours 45 minutes 12 seconds.

Computers are fast enough today that even bubble sort is again popular. It's much easier to code and debug.

Somebody... please... make... it... stop... Oh... the... pain...

Yeah, sure, startup time is irrelevant, even if most of it is playlist load time bottleneck. I mean, you only reboot once a month anyway, right? And even if you do shut down, you are using hibernate or suspend to disk, right?



The important point for binary playlists is that you know the length of every field ahead of time, whereas with plaintext, you have to parse until you either hit some end of line marker, or the end of the file. And then you need some way to encode multi-line values. And then there's that base64 for all unknown types.




None of this makes any sense whatsoever. So it's better to code your own grossly inefficient methods when the language you are using has built-in methods (qsort, std::sort, what have you) which are much quicker? Sure, who cares if it's slow, you can't figure out why the built-in methods don't work for you, so why not write your own? The increasingly more powerful systems will cover your ass in the end anyway.

AHHGAGHAGAGAGFJKGKAFASHGAJSDGHSAJDGHASD

Like I said, if startup with binary file takes 20 seconds, while startup with text file takes 25 seconds, it's not that much different, I can live with that.


Like I said, how many minutes will it take to parse a plaintext file? And what's wrong with base64? You can also use quoted-printable if you want. Or even HTML-style ampcodes.


No, it makes much sense: Binary-encoded thingies makes debugging harder. That is similar to QuickSort vs Bubble Sort. Most Internet protocols (SMTP, POP3, HTTP) use text-encoding. Why? Because it's easier to debug. All you need is a capture of the stream, and without having to specially decode obscure binary codes you can quickly intuit where things start to go awry. It's not the most efficient, but remember that these protocols came from waaaay back when bandwidth are still a mere trickle compared to nowadays.

Now I have nothing against QuickSort; if implemented okay, then I'll use it. If it's buggy or doesn't fit my need, forcing me to write from scratch, I will choose something slower but easier to debug than something faster but hard to debug. Especially if we're talking a difference of mere seconds instead of minutes or hours.

Is there any reason not to use an XML list? It would seem to me that something like ODF, which is essentially an XML file in a compressed archive, would be ideal. This way, we could actually have the album art with the playlist itself, should we choose, all in 1 file. Then, editing the file would be as simple as Find --> Replace as well.

How did you test this?

You may rest assured that the developer tested different ways of implementing a playlist format and his decisions are not only based on his own experience but also on reports he got from testers with much slower hardware where it takes foobar2000 several minutes to start with very big playlists. Besides I don't see the point of this discussion: Everyone can save his playlists in the m3u(8) format if he needs them to be editable. The fpl playlists simply do what they are supposed to do: Providing a flexible and very fast way of storing file information, which can be efficiently parsed by foobar2000.

No I did not test this. Note that I wrote 'if', which means that in the hypothetical situation of...

I think a 'much slower' hardware which severely slows down text parsing compared to binary parsing will be unable to decode MP3 and Ogg Vorbis properly.

Yes, saving to a playlist, editing, and importing back is a workaround. The question is: Why binary in the first place.

The API already changes every minor version (i.e. 0.1). If something as sacrilegious as API can even change, why not the format of FPL, CFG, and other configuration files?

I don't think that this discussion makes much sense but I'm in the mood for that now.
Not at all. A PII is perfectly capable of playing back almost every available audio format with foobar2000. We are talking about really big audio collections and playlists with several thousands of files which become quite a bottleneck on such systems.
As I already said, it's about efficiancy. I don't see any good reason why a playlist format has to be manually editable since this is really rarely needed while you start foobar2000 several times a day and want it to start as fast as possible (at least I do).
Nobody said that the playlist format won't ever change. I just don't think that if it changes it will become a text based format (just some wild speculating from my side).

You're all forgetting that 99.9% of users don't care about manually editing FPL files. But everyone wants stuff to be fast. Binary playlists are fast.

The end.

Um, what is this? Cats versus dogs? (j/k)


We have a rule at the forums called TOS #8. We're supposed to be interested in hard facts, not speculation.

You might say that my first post was speculation as well, but I can tell you I have heard before that Windows's registry (which is largely based on plaintext) is slow.



A Pentium 60 can decode MP3 with DOSAMP or DAMP. I've heard a Pentium 133 can decode Vorbis on Windows 95. I'm pretty sure you'll agree they'll choke on Windows 2000, foobar2000 0.9, with 60000-file libraries and 52 dynamic playlists.



To make use of efficient data structures?



The API has been stable between minor versions. (0.8.2 to 0.8.3, 0.9 to 0.9.1.) If you mean major versions, it's similar enough for components to be converted to the new API.

The format of the CFG file has been prone to change in the early days. It's much better now. I'd say it's a good thing.

PS. The format of the FPL files has changed before, IIRC. (Or was that the database?)

PPS. But I doubt it will be changed to a text-based one.

When I said 'awful' I was thinking specifically of Base64.




All that is required to load a binary file is reading the file from disk into memory. Done. (Of course a few checks of data integrity and validity should be done along the way).

Reading an equivalent text file means reading the (definately larger, probably a lot larger) file from the disc, running a complicated parser on it, which has to check every character and interpret it.

A binary file is definately faster, and even if the difference is 0.5 seconds for you, its a matter of principle. If developers always did things the easiest simplest way, the time differences would soon add up and the software would be unusably slow.

You are very wrong. My configuration is used by Peter to tune the speed of FPL parsing. I've been working with Peter to optimize binary FPL parsing speed since foobar2000 0.23. The system in question is a 667Mhz Pentium III with 128MB of RAM. [Edit: FPL parsing as we know it now didn't exist in 0.23, however, I was already bitching about large playlist efficieny and RAM usage even then. A quick search shows posts as far back as 2001 on this exact topic from me. Fun fun.]

While this is "old" by todays standards, we were recently able to tackle a 0.9 FPL speed issue and reduce my FPL parsing time on startup from about a minute to about 20 seconds, and FPL writing on shutdown to under 30 seconds, where it was previously taking several minutes. You should see these improvements in the next release.

The test configuration is real-world, about 90 playlists, the larger ones containing about 50,000 entries, and the smaller ones containing only a few thousand entries each. The actual size of the FPL's range from a few KB to, currently, about 20MB. This is only representative of about 30% of my collection. As I recall, it was my configuration that prompted Peter to invent the metadb handle system in the first place. A few ~60,000 file playlists are nothing. foobar2000 has been carefully optimized to handle extremely large collections such as mine on modest hardware. This is something no other player is capable of on any platform.

iTunes, for example, uses an XML database, and it can take up to 30 minutes to make simple playlst and library changes using my configuration. This has been reported to Apple, who tells me to "reduce the number of items in my Music Library". Umm, ok.

Peter and myself have put in lots of time (literally, years of work) optimizing playlist read and write times. foosion has a playlist read/write benchmark component he developed to aid in this work. I carefully regression test each release. I hope I have given you some insight into why FPL files operate the way they do. They exist in the current format because of because 5+ years of constant testing and performance enhancements. Also, to answer your questions, the FPL format has changed, several times.

It's just not going to take any steps backwards, especially when it would mean sacrificing usability on large libraries. It is just simply impossible to get the kind of efficiency that is currently present using a text-based format. The current FPL format has grown out of years and years of hard work and sweat on the part of Peter and other developers. As much as you might wish we'd dump all that work, it's never going to happen. If it wasn't for FPL, I'd simply be unable to use foobar2000. This is likely the case with quite a few other users as well.

Ah, thanks for the insights.

I should have added "and 50 static text-based playlists" or something.

TOS #8 reads:
Since I am not talking about sound quality, I believe TOS #8 does not apply here, sorry

That's a valid point, yes. The question is, how much more efficient compared to a text file? ... Annnd I believe TrNSZ has answered this question, thanks.

TrNSZ beats me to posting as I was still struggling with mismatched bbcodes, heh

Hm. Whatever. IIRC it is a general convention that the first digit is called "major" version, the second digit is called "minor" version, and the third digit is called "subminor" version. But... this is slightly OT so I digress

What's so awful about Base64?

A lot larger? I think not. The most significant element of a playlist is already text-based, i.e. the full path of the media file pointed by the list. Growth will result from integer/real values which have to be either alphanumeric-encoded or Base64 encoded. And a parser need not be complicated, if the text-based file is designed properly.

Simplicity and ease of development/maintenance/use is also a matter of principle to me...

Well, it's been a fun, invigorating discussion The general consensus of this thread is that the playlist should stay binary as it is. It is of course the prerogative of the developers, and the explanation from TrNSZ is mighty satisfying even to me.

So I will not press my point further.

Gambit, you can sigh with relief now

Peace!

PS: I give up. I don't know what's wrong with this posting that the QuoteBoxes do not show. Sorry.
Gambit: Fixed quotes, IPB ist b0rk again.

No, but TOS #8 is the cornerstone of everything that is HA. We're still meant to be interested in objective data, not subjective opinions.

Well, this is slightly OT but...

Since I am not the one whose making the claim of faster binary processing, the burden is not upon me. I did post, back up there, asking a quantitave measure of speed, which TrNSZ has provided.

I am just hypothesizing that 'if' the improvement is 'as such', 'then' it is acceptable for me. But TrNSZ clarifies that the improvement is 'more than such', so the factuals/objective data has been presented and I concede.

Again, I do not wish to press my point further.

Peace!

PS: <Even_more_OT>What is it with cat avatars, guys??? </Even_more_OT>

Base64-encoded data exceeds the original in length by the ratio 4:3, and typically appears to consist of seemingly random characters. As newlines, represented by a CR+LF pair, are inserted in the encoded data every 76 characters, the actual length of the encoded data is approximately 137% of the original text data. Remember that FPL is already only a fraction of the size of the text data. Base64 is an extremely poor choice. I can make a safe assumption that Base64 format would cause my playlists (about 50MB of FPL files) to grow in size by at least three-fold.

Wrong, you are very unfamaliar with FPL files, and this is obvious. A simple test for you - take only a medium sized playlist of, say, 20,000 entries and save it and look at the size of the resulting FPL file. Then copy those 20,000 entries and paste them into the playlist again four times, so you have the playlist duplicated for a total of 80,000 entries, and then look at the size of the FPL again.

Simply stated, an FPL does not contain redundant data across entries. Your assumptions here are incorrect. There are many other optimizations as well that exist to reduce IO time. FPL writing/reading is a unique process in that it is optimized so that it is currently mostly bound to disk IO speeds, where all other playlist types on all other players are not.

Trying to use iTunes and waiting several minutes, sometimes up to half an hour for simple database changes to be re-written is absurd and in no way exemplifies ease of use.

BTW, I just added "everything" to iTunes.... and.... "lol":

1829MB? lol.

Edit: This is why a) I only use about 90 large playlists in foobar2000, and b) why the Media Library was invented and happens to be much more efficient than the library used in iTunes (or Winamp).

Sorry to pick on you again, pepoluan, but what are you smoking?! I just read this again and never laughed so hard. Who would implement a bubble sort over a quick sort?

a) A fast selection sort can be implemented in ANSI C in less than 7 lines of code by anyone having read an introductory C book and with zero prior computer experience. Wikipedia's entry for "selection sort" provides a C implementation with comments in 12 lines.

b) Why bother implemeting a bubble sort at all? Quicksorts have been standard in computing since the early 1960's and are part of ANSI C. qsort(), etc.

c) The only people who write bubble sort algorithms are middle school or high school students learning programming. A bubble sort is somewhat more complex to understand than a selection sort and is half the speed.

I really wonder where you get your information, or if you make it up. Again, I'm not trying to offend, but just pointing out how absolutely absurd this is.

Well it's only 'seemingly' random. 137% does not sound too bad; but how can the growth is three-fold?

The question is not to press my point, just curious. This is related to the next one.

You mean you actually did some compression in the FPL?

Now this intrigues me; can you elaborate?

Makes me glad I don't use iTunes


Performance-wise, I'd choose quicksort any day.

Can you give me the link? Anyways, Quicksort is actually exchange sort not selection sort. Which is quite interesting, as plain exchange sort is slower than selection sort, while optimized exchange sort (e.g. Quicksort) is faster than optimized selection sort (e.g. Shellsort).

Yup, that's the point. If it's been implemented correctly, I don't want to reinvent the wheel.

Yup, that's right. But it is simpler than Quicksort still, and it does not rely on recursion.

Nope, I'm not making this up. It was the result of a long discussion on teaching computer sciences at my university (I taught data communications 1 and nearly all networking-related stuffs there).

Redundant data is encoded similarly to run-length encoding, to avoid redundant IO. I don't recall when Peter implemented this. There is also experimentation underway to use compression to increase speed, since the process is mostly IO bound now.

This is not a new concept. NEXTSTEP from release 2 used on the fly compression/decompression of any memory going to and from swap space, because the CPU was faster than the slow disk IO speed and having less data to write increased performance, because the CPU could work faster than the disk. This is often used in networking protocols too, where wire speed is usually slow but the CPU is not, such as SSH compression or FTP directory listings using MODE Z.

Ahhh... okay, that's really informative. Thanks for elaborating.

But isn't compressed data more prone to bit-errors? How do you recover from errors?

(Again, not pressing my point here. I'm really curious and want to learn about this )

http://en.wikipedia.org/wiki/Selection_sort vs http://en.wikipedia.org/wiki/Bubble_sort ... See http://en.wikipedia.org/wiki/Bubble_sort#In_practice - "Bubble sort also interacts poorly with modern CPU hardware. It requires at least twice as many writes as insertion sort, twice as many cache misses, and asymptotically more branch mispredictions. Experiments by Astrachan sorting strings in Java show bubble sort to be roughly 5 times slower than insertion sort and 40% slower than selection sort."

A quick test on PowerPC shows the difference to be about 30%. On modern CPUs, simply looking at the code is NOT an indication of how it will perform, when you consider that all chips past the original Pentium are superscalar and use some sort of instruction pipelining and will execute more than one instruction per cycle. Optimizing for pipelining is a extremely simple matter. Just think that everytime a program branches, the pipeline is flushed. Apple's introductory document TN2085 (Tuning for G5: A Practical Guide - http://developer.apple.com/technotes/tn/tn2086.html) talks about avoiding pipeline flushes as well. This is common technique, not just for assembly programmers.

http://www.cs.duke.edu/~ola/papers/bubble.pdf

For a quicksort that you don't really need to understand recursion to read, see http://alienryderflex.com/quicksort/ - this is also linked from wikipedia.

I know this maybe sound very unhelpful, but if you are getting bit-errors while writing playlists to your hard disk, you have worse problems your FPLs. =)

Anyway, there is no kind of recovery from corruption in any other playlist type either, including the XML databases used by iTunes or M3U/M3U8 playlists.

With a compressed playlist, if it fails to decompress properly, you can at least know that corruption is present and refuse to load, where certain corruption with an XML or text-based playlist could easily pass through a parser.

Hm, you have a point there. Bit-errors in compressed data usually results in complete corruption of data.

But OTOH, bit-errors in uncompressed data means some data can still be salvaged. Error (hopefully) gets detected during usage of data, or each entry can be protected by simple CRC-32.

Anyways, <OT>thanks for the links. I am only vaguely aware of Combsort and it rivals Quicksort in performance (according to wikipedia there) while a lot simpler... I'll test this algorithm at home extensively.</OT>

I do believe that the compression being investigated does uses a checksum, optimized for speed. I'd have to ask. I'm not that famaliar with the actual file format on disk or the algorithms being used, I mainly just test the software.

Also, when it comes to teaching sorting algorithms, see http://www.cs.vu.nl/~dick/gnomesort.html for something simple, but not too effective for teaching.

LOL... thanks but no, thanks. It's enough for the students to learn one inefficient sorting algorithm. And now I need to integrate CombSort into the oh-so-limited timeslots given to me for teaching

I still don't really understand why this discussion even came up, as you can always save to m3u...

We have too much free time?

The thread starter IIRC was talking about editing a playlist. For instance you might want to search/replace in foobars database.fpl (instead of doing a rescan media library)

That's not really foobar's problem, you won't get bit errors on a modern system, the filesystem/network protocols/however you got the data should detect any corruption, and fail completely when they do so (if it is not possible to succesfully re-read/resend the data)