Bandwidth-gobbling RSS aggregators: foiled!

0 comments suggest edit

This is great! Rather than wait for all the RSS Aggregators to properly use the If-Modified-Since header, implement it on the server instead via an IP address and User Agent combination. Now your first thought is probably “Wait, that’s not perfect. What about users of internet providers such as AOL which uses a shared pool of IP Addresses?”

True, theoretically there could be an instance where you don’t receive a blog entry because your IP and User Agent string just happened to match someone else. But really, how many AOLers are subscribing to RSS feeds in the first place? RSS is still mostly in the domain of the more technically sophisticated. Secondly (unfair cracks on AOL aside), the chances that two users with the same IP and User Agent requesting your pathetic little blog close enough together in time is probably very slight.

UPDATE: A commenter lamented that users behind a corporate firewall will lose out. This is a more likely scenario as your coworker is likely to subscribe to the same blogs that you do. My solution is to only throttle aggregators that misbehave (you know who you are). Or conversely, don’t throttle well behaved aggregators. This provides incentives for the misbehaving aggregator developers to fix their aggregators. RSS Bandit is well behaved in this regard.

Xeni Jardin: Last month, Cory posted an item about Glenn Fleishman’s analysis of the impact of RSS aggregators on his blogs’ bandwidth use. (Link to previous BoingBoing post). Now, Glenn updates us with this news:

I’ve run the latest statistics on RSS usage after adding a simple throttling program that uses a database to track the last access by an RSS aggregator (or anyone trying to retrieve a syndication file). One retrieval per file update is now the limit. I’ve seen my bandwidth use on RSS drop almost in half with no commensurate drop in actual users, and only a single note describing a problem in retrieving my feed (from a very old aggregator).


[Via Boing Boing]

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.



5 responses

  1. Avatar for Steve
    Steve December 8th, 2004

    Hmm shame - I agree that the bandwidth consumption sucks but you're losing people behind corporate or web-optimizing proxies.

    Why not implement at least TTL in your RSS feed?

  2. Avatar for Haacked
    Haacked December 8th, 2004

    Well I haven't implemented this process, I'm merely blogging about it. Granted that will be a problem if two RSS users subscribe to the same feed from the same office.

    However, the point of the article is that many aggregators are not honoring TTL headers etc...

    Perhaps the best solution is to only throttle on users using those aggregators. This will force the aggregators to fix the problem.

  3. Avatar for Jon Galloway
    Jon Galloway December 9th, 2004

    I think RSS is built to spill. As far as I can tell, no one else does.

    As someone who thinks about RSS and coded parts of my favorite RSS aggregator, can you tell me what's wrong with my thinking:

    Honestly, this keeps me up at night and I'd like to hear someone smart tell me what I'm missing.

  4. Avatar for Haacked
    Haacked December 9th, 2004

    I tried to go to your site and it said a ThreadAbortException occurred. "Thread was being aborted". Wow.

  5. Avatar for Jon Galloway
    Jon Galloway December 10th, 2004

    The whole site was down for a while. Back up now. Sorry you had to see the mess with the threads and all.