The HA "Posts" RSS feed doesn't work properly. Especially when used in Google Reader. Google Reader has 78 subscribers to this RSS (10 aug 09).

Below are notes what I've gathered during past two weeks.

First a GOOD example (tested on Aug 10):

REACT 2 Released thread, my last post on Aug 8 2009, 16:31 (HA server timezone?!): http://www.hydrogenaudio.org/forums/index....st&p=650450

Google Reader:
Received: Feb 3, 2009 10:40 PM (item sorted by this, I had to scroll about 2000!! RSS items before I located this!)
Published: Aug 8, 2009 6:31 PM (date tooltip shows this)

The item shows my last post but it's still sorted by the received date and not made unread again.

How insanely stupid is that?! What this means is that, all new posts to that thread between Feb 3 - Aug 8 have NOT been presented to Google Reader users because it doesn't mark back to unread nor change the "received" date either!

FeedDemon w/o synch sorts the item correctly by the Aug 8 date but it doesn't mark it as unread, so in order to catch it (+similar others) one has to turn the filter to "Show all" when reading that feed. PITA to change it for this feed only (I haven't got any other "forum" feeds) when reading, and PITA to locate which of the listed threads/topics/rss items are really new. And I'm soon moving to FeedDemon with Google Reader synch, so the problem will get worse as demostrated above.

So basically now only new topics are shown as unread in Google Reader. Or very old (~couple of months-year.. I guess it's the time when Google cache clears out old stuff) topics which are resurrected with a new post.

Just try it yourself, open in Google Reader and compare to e.g. Opera feed preview window or to the source code of the fetched RSS feed. Select one popular (/active) topic you're familiar with and see where you can find it in Google Reader.

------------------------------------

Let's look at one other RSS feed from a forum too: http://www.espooblues.net/keskustelu/rss.php

The other RSS feed has <lastBuildDate> not <pubDate> in the channel element (which one to use? or both? the specs doesn't tell you the difference properly). And it does NOT have <pubDate> nor <guid> inside ITEM elements (w/o the pubDate, the posts are not sorted properly). Also the <link> is unique (e.g. ...viewtopic.php?p=82328#82328).

This other feed shows EVERY post, even with Google Reader synch in FeedDemon (or just in Google Reader too of course). Shouldn't HA forum RSS feed act like this too?

------------------------------------

It seems that the culprit is <guid> element which's value doesn't change when a new post is published in HA RSS. I think that the description of this element tells the whole truth:

"guid stands for globally unique identifier. It's a string that uniquely identifies the item. When present, an aggregator may choose to use this string to determine if an item is new."

So, when the <guid> doesn't change in new items (new posts to a topic (title in RSS item doesn't change)), the Google Reader considers this as an old item already marked read.. but it still updates the content (<description> and <pubDate>).

------------------------------------

So what can be done?

Make <guid> unique for every item (i.e. every post) -> ensures that the item is shown unread in various RSS-readers!?

Please do something about this, if you can. I'm willing to test. Thank you.


Some items have empty <description>. Dunno why this happens, somebody else also noticed and wrote about this in the "Hydrogenaudio Server Switch" topic long time ago.

By the way, how often the feed is updated? TTL = 5mins?

------------------------------------

Here's my last post in "Hydrogenaudio Server Switch" topic. It never got any answer. Maybe this time? Please, I really want to fix it.. and I have put so much time to this already.

Nobody? Too big job? I really believe that the problem could be fixed by only changing (or removing) the <guid> element. I can test thoroughly if you don't have the time to do that.

guid is now constructed by taking a hash of the post link and pubdate.

I believe this should fix your problem.

I also added a "poster" field. The frequency of RSS updates is indicated by the TTL field.

Works like a dream now! THANK YOU SO MUCH!

BTW, "poster" element? Did you meant to use <author>? According to specs, it should contain email address but I've seen it used only with names. Dunno if that breaks anything though.

P.S. Tried to locate the culprit of the empty description element problem. It might happen because the xml parse fails to parse the text inside the <description> element. It seems that sometimes the text is wrapped inside <![CDATA[]]> and sometimes not. Maybe the forum software misses some "xml parse breaking chars" in the texts and doesn't wrap the text with CDATA. This might be the culprit. I don't see any problems wrapping all texts inside <description> elements. I'm not an expert on this issue though.

P.P.S. The "...&view=getnewpost" in links seem to work strangely, does it work like this: forum saves the user read topic last posts (when online.. or it uses cookies) and clicking a "getnewpost" link first goes to the next post of that saved one (if not saved, shows first post), but next "getnewpost" link click (right after the first click) directs to the last post of that topic. It sounds strange but it seems to be working like this now. Anyways, I don't have a problem with this, I can locate the correct place myself, BUT, now that the RSS feed is fixed to show all posts correctly in various readers, maybe the RSS <link> element link should be a link to the actual post of that RSS item*, not the "getnewpost" which goes whoever knows to what post in that topic.

* e.g. your last post, #3:

(had to use code because forum shortens long link texts)

Anyhoo, thanks a million again for the fix!

Hmm, this change was probably needed for the "Posts" feed, but doesn't work so good for the "News" feed. I don't want to get every single comment on an item in the news feed as an updated feed entry.

The purpose of the News feed should be the news i guess, at least that's the way every other "News" feed on the web works. If someone want the comments on the news feed then that should probably be integrated into the posts feed, or a maybe a new feed "News with comments". But please revert the behaviour of the news feed, it was already perfect before this change.

I'll second this. It's best to just see the first post of a new topic, and then no updates in a topic until there has been 3 or 4 days (or more) of inactivity.

The problem here is that both feeds are just using the same generic code.

Edit: hacked it in anyway

Thanks Garf. For the stab and the author fix. I see that you're getting more relaxed about stabbing the forum code.. hopefully this will lead to other stabs as well.. you know what I'm talking about. One can wish, right?

I'm also using RSS reader (FeedDemon) and now it looks better than before, especially update speed and was about time for adding poster name , but just wondering why some posts are without <description>, i.e. only get the thread title (this is not something new, as this happened before also)

XML has only <description></description> and I can't find any pattern why some post ends like this

Did you read my post #4? I tried to locate the culprit but couldn't pinpoint it.. I did wrote a hunch why this happens, check the post. I don't have a problem with the missing post content, I always click the title to open in external browser if I'm interested in some topic. Remember that the feed doesn't catch every post made to a topic.. so if you really want to follow a topic, open it in your browser.

Well, I didn't go through your #4 post but read your OP and browse quickly to see if this problem was mentioned and miss your <description> point.

I would like to suggest something about quoted text: can <!--quotec-->...<!--QuoteEnd--> be replaced with <i>...</i> or <b>...</b> or even better <hr />...<hr />. It will look much better I think


I guess that's just you. It would be correct to display the post or part of it at least. I use my reader for accessing many RSS channels, and it has integrated browser of course, but I would like to see the topic not just the topic title
And I didn't noticed lately missing post - maybe I oversee don't know

There are definitely some posts missing. But I can't say, when it happens. For example, I didn't get post #10 from this thread, but post #11. One thing I can say about it, is, that I should have got both posts with one polling request. But you can't say that this is a general rule, because mostly this works accurate.

For the topic-title-only posts I made the observation for one post, that it happened, when this post was edited between creation and polling. Bit I didn't look for this further, so this may not be the reason for this problem.

Ok. That is not the reason for the problem. I now got topic-title-only posts for this post and this post, which were not edited. The first two posts of the same thread were not got by me. Maybe this helps.

That is strange, I did get #10 and #11:

Feed Validator gives some error hints for the HA Posts RSS feed:

"Sorry, this feed does not validate."

Could this help to find potential reasons for the situation?

I think that this is OK idea, not replacing those tags but wrapping QUOTED text in it, and if it can be easily done it would work, although <content:encoded> is suggested as better solution.

Another thing is CODE wrapped text, i.e.:


Inline stylesheet doesn't have "codemain" class definition, as CODE is wrapped in it. So it looks like this:



I would suggest also here enclosing CODE text with <PRE></PRE> at least or specifying monospaced font as the parser would understand it and <hr /><hr /> optionaly for better looks or correcting the stylesheet.

Here is vBulletin RSS output example (it uses <content:encoded>, although that could be board related and maybe not so easy implementing it for IPB, but I guess there are external modules):

Believe me and fbuser, you're going to miss some items for sure. I'll try to explain and give examples.

Your first screenshot ("That is strange, I did get #10 and #11:") shows that you are using FeedDemon without GoogleReader sync, the reason you got those items is probably (I don't have enough info yet) that you kept your FD running at that time those were published and your FD is set to update that feed (or all) in short intervals. The HA feed has only the last post to a topic and how often you get the updated feed content plays a part with these missing items. If you keep FD (feeds without GReader sync (or without the soon or already shutdown NewsGator sync)) closed for many days, you would only get the last X items (varies per feed) published in HA feed.

Now, your last screenshots reveal that you're using GReader sync in FD for those feeds. This again changes the situation because GReader automatically updates feeds in 1 hour intervals* which means that within that update interval, only the last post is shown in GReader (or in FD w/ GReader sync). You can check this from GReader (you can't see the GReader update date/time in FD, only post publish date/time), there's no items with the same titles within one update interval in this HA feed. Example: this post is not in GReader because there's a reply to that post 20 minutes later and those both posts were in between the GReader 1 hour update cycle.

* At least this HA RSS feed we are talking about.. some feeds could have other update intervals based on the popularity/items published per day/?, I believe that this is not public information and Google only knows the full details. I don't know though whether FD with GReader sync forces GReader to update the feeds earlier or not, e.g. when starting FD or clicking the "update all" button.

The feed TTL value also could play a part in this problem. HA feed has TTL value of 5 minutes.

@fbuser (applies to others too): I take it that you're not using GReader or FeedDemon with GReader sync? And you had your RSS reader closed (not updating automatically) when the #10 and #11 posts in this thread were posted (those posts were 1h 27mins apart)? This could explain why you didn't receive the #10 post. I suggest you to check out FeedDemon.. the latest 3.0.x PreRelease versions have GoogleReader sync. GReader is nice because it's always updating your feeds even if you have your external RSS reader closed. E.g. my problem before this great GReader sync was that with high traffic feeds that had lots of posts per day (office hours) but the feed maximum items is 20 items, after I got home from work (I don't want to spend my work hours doing something that isn't in my job description) and opened my RSS reader for the first time, I only got 20 latest items from only past couple of hours, missing all the rest published that day (tens and tens of items). FD with GReader sync totally fixed that, now I can e.g. skip couple of days without worrying that I'll miss feed items (excluding of course items that fall under the problem we're discussing here).


There's only 1 problem, the <author> hasn't got an email address like the specs require. Hardly going to cause problems IMHO. Others are "interoperability" suggestions that IMHO don't have any effect on this missing issue. I also checked that these "interoperability" suggestions are not behind the missing <description> problem.

I believe you and I know that I'll miss posts because of update interval or using Google Reader sync (I meant to ask fbuser about that, but...). I was thinking that maybe some posts aren't published as some don't have description.

Then, you have nice ability to spot things:
You probably noticed that first screenshot is with FD 2.7 and the last two with FD 3. The reason is that I tried FD 3 and got annoyed by their ads window, so I revert to 2.7, but as you mentioned Google Reader I went back again on FD 3 to try sync (2.7 had NewsGator sync - discontinued), but on the screenshots there isn't any information that I use Google Reader sync and I didn't don't know why do you think that?


No, they are updated by Google Reader.

Syncing has it benefits, like when using different computers, but I read RSS only at home, set on custom intervals and I don't want to see deleted feeds also.


I don't know how TTL functions - if it caches feed for 5min it's great for me, because that is the interval I set for HA feeds, but still not sure if I'll get post replied after 2min?

Although you commented all back posts you didn't comment on CODE and QUOTE suggestions. At least that the inline stylesheet is missing the referenced elements, and if corrected readability can be much improved.

lol, my previous post doesn't have <description>

I don't know what you exactly meant by that, in any case, I wanted to say that those are the same thing, i.e. there's no special separate GR sync reason that you'll lose posts. So, you'll miss posts because of update interval. Period. The only difference is the update interval used.


Oh? Many people in the FD forums (and in Nick's blog) were very angry about the ads at first but now many say that they don't even notice them anymore. And you can cash out 10$ to get rid of those. You can use the license on different computers (it's not decided on how many but Nick said that they're going to be very lenient with that).

Besides, as you might already know, FD uses IE engine and if you have tightened IE rules (which everyone should do! You never know what app could use IE engine without you knowing it), i.e. blocked JS, ActiveX and rest of the crap that ~70% of all Windows critical updates are for, you only see a black content or text "how to get rid of the ads" in that ad place. Not very distracting at all.


Aah, you're not? Ok. The "mark read/starred/share/tags/etc." icons (after title) gave it away that you were using FD 3.0.x versions.. I just automatically thought that couple of those icons (e.g. starred, share) would not show unless you're using GR sync.


Are you sure? Since the GReader web app allows one to update a feed outside the automatic update interval. So basically the FD developer could have that option to use in FD.


But do you have your computer and RSS app always on? If not, you'll lose some old messages if the items published count exceeds the feed maximum items when your computer/RSS reader is not on. I tried to say this in my previous post. That's the only reason I'm using GReader sync now. I'm not currently using it on different computers.. but in the future I will.

Don't want to see deleted feeds? What this has to do with FD GReader sync? If I delete a feed in FD, the feed is removed from GReader too of course. At least with the latest PreRelease version it is. Maybe you used older version which has some GR problems? The FD forum tells that the GR sync has been constantly changed/fixed, sometimes even bigger changes, during past weeks (or should I say days, the release schedule has been super fast).

I'm using 15 mins custom interval in FD with GReader sync. It's more than enough, I don't want to monitor RSS feeds so intensively. And because this HA forum is the only feed I'll lose some posts from (I don't have any other "forum" feeds), I don't mind that the GR update interval is about 1 hour (it's more than enough to get all items even from high traffic "normal" feeds with low maximum item per feed setting). AND, I'm not following the topics in FD, but opening them in my browser, I really don't care if there's some missing posts (there's always at least the last one so I know which topics have new posts) because I read the whole topic online to the last post from where I last was.


The feed is published every 5 minutes, i.e. content changed. So I think you could lose some posts to same topic if those are posted within that 5 mins.


Well, to tell you the truth, I'm not interested about those.. and I'm swamped with things to do at the moment.. and guess how much time I've put to HA RSS feed problems.. LOTS.. I started in other topic several months ago (I have not constantly wrestled with this issue though). I've seriously started to think about starting a "cut down things to do and follow" process. It's stressing me out too much that I don't have enough time to do everything I want/"have to".. and I've already given up so much.

Why didn't I think of that? Thanks
Although I tried also to patch FD exe to use Mozilla Active X control, but I guess that is not possible with this version 3


"starred" is same as "flagged" was, and "share" is always there, but doesn't function if you don't use GR sync, and interestingly it works if you select to sync only root folder (Subscriptions) although you don't sync any RSS channel


Here is screenshot with GR sync and without. That's why I don't think you can manually update if GR doesn't update.



How can you change update interval? Through your Google account?


Probably I made some mistake, thinking that on resync GR revived feeds that I deleted. I tried now and it's OK. Thanks for TTL explanation also, I misunderstood that too.

As for using GR, I don't miss that. FD is started with Windows and it's in tray, and PC is almost always ON. If I'm away some days, I'll use browser to check if something "really important" had happened. Although it's good when using different PCs as said and for archiving purposes.


I may use your thinking for my browser avoidance, i.e. I use Opera when I need to, but I have bad habit of opening many many tabs and then loose the path(s)
It's easier for me to follow different forums/blogs etc. with RSS and if I want to reply or visit the page I use integrated browser - you know on what you are focused and don't waste your time. Sometimes of course I just surf

And last but not least I hope some site administrator will reply in this thread

Akkurat,

I am using the freeware Feedreader which claims to have some "unparalleled functionality".

For you may be of interest:

• configurable archive size (from 50 articles to unlimited)
• configurable update priority (lower, low, normal, high, higher)
• configurable update period (fastest = 10 seconds
• filter functions ("smartfeeds")
• saving articles as HTML documents (enables to view the source code)

I never used FeedDemon or Google Reader so I do not know which functions this software have in detail but I would be glad if Feedreader could help to solve your problem.

Just open "Tools -> Sync. options" and you can set how often FD checks if there's any updates in GR. It seems that FD doesn't ("update all" button, automatic update) force GR to update outside the automatic update interval.


We have different needs of course, all I tried to say that the GR sync is VERY useful even if you don't use FD on different computers nor archiving because you don't have to have your computer/RSS reader on almost always in order to catch all published items in various feeds. Excluding feeds like HA of course.

What problem? I've no problems with FeedDemon. Are you referring to the "feeds with high traffic but low maximum item limit" problem? That can't be "fixed" by using different standalone readers. It seems that I wasn't able to explain the issue to you.. I'll try again.

Lets take a feed that publishes about 100 items per day (but has 20 maximum items in its feed), most of them published during office hours (a news feed).

~7:00 feed starts to publish items.
8:00 I go to work, no RSS reading there.

- (7:00 - 18:00 the feed publishes about 80 items)

~16:30 I get home.
~18:00 I open my PC/RSS reader (no GR sync) which updates the feed BUT it gets only 20 latest items (from about past 2-3 hours) because that's what the feed is offering! I miss all morning and afternoon items! Here's where GR sync comes to save the day, GR is always ON and it updates/caches feeds in 1 hour intervals (might vary) and thus it caches all items (the feed doesn't publish over 20 items in 1 hour) and I'm not going to miss anything. All I've to do is open FD which syncs to GR and gets all the items. Handy.

IMHO FD with GoogleReader sync is far better solution (for a "normal" user) than any other standalone feed reader available. Back when I was selecting a feed reader, I did test many readers, Feedreader too, and ended up using GreatNews but I switched to FeedDemon after a while. I suggest you to try the new FeedDemon with GR sync (PreRelease version at the moment, just going final), you might like it.

This thread is getting seriously off-topic, and I can't really split it as the info is all mixed in. Can we stay on topic-please? You can discuss the merits of different readers in Off-topic if you wish.

Thanks.

hm..., I didn't noticed that. Now I use GR sync for all channels, except HA of course, which I set on 5min update interval


Can you comment on empty <description> element?

And I must correct myself: there is no inline stylesheet, as I looked at wrong place by mistake. The CODE and QUOTE can be easily corrected by adding this line in RSS XML output:

instead of this:

show be this:

Is someone willing to work on this issues?

Well I can.. or rather parrot what I wrote earlier:


I would try that. Hard to say anything else when black-box testing this problem.