0xDECAFBAD - Tag: syndication

good gregarius

2006-08-18T04:43:20+00:00

Oh, and apropos my serial enthusiast ways, I've shacked up with a different news aggregator again. I was using newsRiver again, but the temporary lack of a spare server to run it on in the midst of the move had me feedless. So, after a brief flirtation with Bloglines - which went down promptly after I started using it - I switched using Gregarius. I have to say: This is a pretty keen aggregator. It doesn't support reading lists, but I have an itch to implement that support. Otherwise, the feed folders work pretty much how I'd like them to work - as a prioritized and stratified River of News. I might also think about hacking my evolving feed reader template as a drop-in theme for Gregarius.

Reading Lists, OPML, XOXO, Semantic Web, and Tools

2006-02-13T17:09:08+00:00

Listening to this podcast about OPML and Reading Lists and enjoying the various perspectives on RSS, OPML, and Semantic Web tech. It's also the first time I've heard Danny Ayers' voice, so that was pretty cool after having been a textual blog acquaintance for a few years now. As for the rest of the guys on the call, I'm not quite as familiar with all of them yet, but I'll be adding them to my Reading Lists shortly.

Apropos of this podcast, I've lately gotten a bit of a fresh take on the RSS/OPML versus XHTML versus Semantic Web tech merry-go-round. Here's the basic gist: Invest time into tools that solve problems first and formats that enable possibilities second, if you want to get any attention and subsequent help.

And then I say "tools", I don't mean a GraphViz construct that makes really cool charts for people willing to chase down and install the dependencies. No, I mean something like the OPML Editor. The OPML Editor comes in a single package, and you launch it from a single icon. The OPML Editor is a shitty piece of software that's nonetheless helping me achieve some practical goals that I've heretofore been vaguely stymied in reaching. The OPML Editor is making me think more favorably about OPML, despite OPML being a shitty format.

The utility of the OPML Editor is giving me the motivation to build filters and workarounds to transmute OPML into XOXO and RDF formats, but only when the need arises. I could also see putting an OPML-to-triples bridge in front of a semweb crawler or tool. And, there's no reason why outlines in the OPML Editor couldn't be rendered in XOXO and RDF formats by the tool itself, if there's something useful around to consume it.

On the other hand, I've done enough with RDF and XOXO to see the clear potential in both. With enough RDF data around, you can turn the whole wide web into a single massively networked database ripe for the querying thanks to a uniform data model and clear specs for representation of that model. With XOXO and microformats, you get a Sudoku-esque solution to providing both human and machine readable data in the same format and file, while sterring close to the inherent design principles of the web.

The problem with these formats, though, is that they open potential avenues but don't provide value until there's some foot traffic on the avenues. This is just establishing a world in which there can exist both chickens and eggs, but not doing much with respect to concrete instantiations of poultry. Now, to be fair, there have been efforts to make practical usage of RDF (see: MusicBrainz) and these are early days for microformats. But, I think a lot more effort needs to be given toward making useful, personally rewarding tools - and less effort given toward expressing awe toward the potentials given by formats and models. You can have both, but you've got to pick one to start with.

And, the reason you need to make tools first that use formats second is attention. (That's been a buzzword lately, hasn't it?) You need attention if you want adoption and help. And, you can get attention - and subsequent help - by scratching someone's itch.

Now, some people's itches are scratched by the fun of abstract symbol manipulation and the perception of elegance. Others' itches are scratched by getting work done, like writing or organizing thoughts or expanding awareness of more information sources. Personally, I've got itches of both of these varieties. The problem is that there are many fewer people with itches in that first category. And, no offense to anyone, that seems to be the bulk of the RDF and XOXO fanbase at the moment. For the second category, even if things are shitty, smoke a bit and catch on fire - as long as some work gets done at the end of the day, there's a lot of tolerance to go around and a lot of motivation to pitch in to improve the situation.

At the end of the day, I firmly believe that ATOM, RDF, and XHTML-based microformats trounce OPML, RSS, and the like in terms of clear definition, affordance in manipulation, and technical elegance. But, OPML and RSS are still winning in the world because the tools using these formats are scratching some damn annoying itches. And, though towers built atop shitty tools and formats seem Jenga-shaky, somehow they never quite come crashing down. Maybe they will come crashing down someday, but we seem to be getting work done in the meantime.

So, to wrap up: My take on this whole mess is that your format will gain adoption to the degree that the tools producing it help people get work done. I can't find tools today for RDF and XOXO that are as rewardingly useful as the OPML Editor has become. When that happens, the story may change.

Archived Comments

Edward Vielmetti
2006-02-14T02:32:49

There's a lot to be said for rough consensus and running code, and there's a lot more running code that speaks OPML than there is that speaks XOXO or RDF. You can just plain sit down and get work done and not have to entangle yourself in a microstandards battle. There's a certain appeal to that.
phil jones
2006-02-14T03:03:10

Same as it ever was :

http://www.laputan.org/gabriel/worse-is-better.html

(scroll down to section 2.1)
Phil Wilson
2006-02-21T20:39:35

I know they're very different things, but is there any particular part of the OPML Editor that doesn't exist, say, in something like NVU, or a stripped-down TinyMCE?

Ancient code drifting down the newsRiver

2006-01-28T06:08:57+00:00

So, while working through some writing blockages last weekend, I started doing some hacking on Dave Winer's newsRiver running on the OPML Editor platform. (Oh, and it is a platform, maybe more so than even emacs—don't let the moniker 'editor' fool you.)

Here's the thing: It was a lot of addictive fun, and it takes me back to the love/hate thing I had with Radio, back toward the first days of this blog. This Frontier / Radio / OPML Editor environment is undeniably satisfying for me to work in. It's like potato chips: Jump to an outline here, tweak some code there, mangle an outline node up there, reload a browser page, watch things go—lots of moments of instant gratification all building incrementally atop one another.

In one package—which reminds me a lot of Squeak—the OPML Editor bundles a crazy amount of integrated machinery for both Windows and Mac OS X:

UserTalk, a very capable scripting language
An IDE based on outlining, somewhat like Python's syntactic indentation
Persistent on-disk hierarchical hashtables (aka the Object Database)
Code and data all live in the same object database
A web server and a host of other networking code—including Jabber messaging, although I can't attest to its freshness.
Blog editing and no-effort external web hosting file synch
Hot patching and easy upgrades that most other software packages dream of
Years and years and years of (mostly) working code with datestamped change notes and living history.

And the thing is? This was the state of the art for Radio back in 2002—and earlier than that even for Frontier. Fast forward from then till today, and you'll find that Frontier is now Open Source—with many of the same sorts of warts Mozilla's open source release helpfully revealed back in 1997. (Some hero needs to come along and port this beast to Linux! Good luck!) And now, much of the guts of Radio are released as the OPML Editor.

In a lot of ways, this whole family of apps built on this common ancestry of code is really shitty software. Kind of like all that code on the Qeng Ho from Vernor Vinge's A Deepness in the Sky. This stuff crashes and sets itself on fire a lot, but then again it's amazing to see just how many wheels have been reinvented and re-reinvented since 2002 (and before) when I fire up this code today.

And now, it looks like the embers are stirring for this thing again. There's been a lot of time for a lot of angst to fade from a lot of places, and it looks like there's a new crop of fresh people rediscovering this stuff.

So, anyway, I like these potato chips. I think I want to start munching on them again, as time permits. Oh, and while I'm talking about potato chips—I have to say, don't pee in the potato chips and be nice to the guy holding the bag.

Archived Comments

Robert Brook
2006-01-28T15:03:50

Serious question: why not use Squeak?
l.m.orchard
2006-01-28T16:01:26

Robert: Short answer is that I saw Radio first for the RSS aggregator and blogging support, started hacking on that, and got sucked into other aspects of it along the way.

For Squeak, I saw some fun colorful things and kind of got the gist of the environment, but never got hooked on any daily uses of it after a weekend or two of playing. So, it fell into my "Learn Smalltalk" todo item that keeps getting pushed down the list by other things. That item's still on my list, though, and I mean to get to it someday...
Danny
2006-01-28T16:07:44

From what I've seen and read there are loads of good ideas in that platform, but there are two things I personally find extremely offputting : the pervading sense of Not Invented Here, and the sheer age (in the negative sense) of the stuff.

Now the code's been open sourced, maybe there's a chance of the tastier crisps (UK localization) catching up and being integrated with the rest of the world. Ok, very IMHO but : swap US-ASCII for utf-8, XML-RPC for doc/literal XML, RSS 2.0 for Atom, Object Database for triplestore, Python for UserTalk, outliner for modern IDE (although actually the outliner UI is probably one bit that might be worth preserving as-is) etc.

Well, ok, maybe it is a bit much to expect. Maybe a maximally interoperable version of the Frontier approach could be recreated (without reinvention) by plugging together various existing open source libs..?

For now I guess I'll have to make do with Web as Platform (and/or Squeak) ;-)
l.m.orchard
2006-01-28T16:28:59

Danny: Yeah, I agree with a lot of those points, and they fall into the 'hate' side of the 'love/hate' thing for me. (Well, more like 'strong distaste' than 'hate' anyway).

And I've realized that, since drifting away from Radio/Frontier, I've been recapturing it in various forms using tech like Python, triple-stores, mini-HTTP servers, and the like. And although these things have felt 'healtier' in a non-NIH, bigger community, more loosely coupled way... it's still a satisfying experience (like potato chips) to find it all buttoned up in one integrated package that works on most of the platforms I care about.

It's still kind of a writing-with-my-wrong-hand kind of pain in the butt to get all the loose coupled stuff working sometimes. And many times, there are very good reasons for thing being hard and lots of things need minding. Like, XML-RPC style web services versus REST:

REST is a cleaner, more web-natural approach—but sometimes it takes a lot of thoughtful meta-tinkering where that XML-RPC stuff would've had you on the road hours ago. Of course, years from now, the XML-RPC stuff might smell a bit while the REST stuff will have stayed solid and revealed further unexpected benefits... but bah. They both have their place.

It's a lot like baked versus fried. Anyway, rambling now.
Danny
2006-01-28T16:57:00

"It’s still kind of a writing-with-my-wrong-hand kind of pain in the butt to get all the loose coupled stuff working sometimes." Nicely put, yup. Bit of a challenge. Coincidentally, yesterday the idea of "Semantic Web in a Box" came up - maybe the place to start would be "Web in a Box"...
arbitrage betting
2011-06-23T02:16:57

Nice points there. I agree with you. Those steps dated in the past are still very useful and helpful until today. We just need to be careful which ones to use. Munch more potato chips! Cheers!

A bit of newsRiver hackery

2006-01-21T06:54:30+00:00

Update: I moved this code over to my space over at hosting.opml.org and added some updater code from Nicholas Riley's webdav tool, so that you should be able to just "Get Latest Code..." once you've got this installed. (Or reinstalled, if you grabbed last night's version.) Also, I fixed a bug or two.

So, I couldn't resist poking around with Dave Winer's newsRiver in the OPML editor. For whomever's interested in this sort of thing, here's the result of a couple of hours' hacking this evening:

decafbadNewsRiver.root

In this is a tool to be dropped into your OPML Editor install's Guest Databases / apps / Tools directory, right alongside newsRiver.root itself. You may need to restart the OPML Editor. Once installed, you should see a new "DecafbadNewsRiver" menu item under the Tools menu, allowing you to view your news in this new UI. Expect lots of bugs, though.

Basically, this provides an alternate DHTML outline rendering of the news items gathered by newsRiver—quite similar to what I did back in October for FeedSpool. I'm not sure how far I'll take this, if I take it any farther. I've got a bit of a notion to tackle some REST/AJAX interaction with newsRiver for on-the-spot item deletion, and maybe on-demand item content retrieval.

But, for newsRiver hackers: You might like what I've done with the viewNewsItems script under decafbadNewsRiverSuite. I've made an attempt to pull all of the HTML out of the midst of the processing code. That's was icky, and I hated it back when I first ran into it in Radio UserLand.

I've broken all of that out into smaller external templates for page, feed, and item under decafbadNewsRiverData. This should allow for easier styling and customization of the code that newsRiver produces, as well as maybe swapping themes on the fly with a preference that points the rendering process at a different root table for templates.

In any case, do what you like with this work. I hope it's useful to someone out there.

Archived Comments

Donovan Watts
2006-01-21T07:55:24

neat! i much prefer this interface. thanks much.

Use del.icio.us to build & share Reading Lists?

2006-01-19T20:08:37+00:00

As far as I know, the most popular link managment tool is del.icio.us, a tool I love for its power and simplicity. del.icio.us allow you to export all your links in RSS which is cool. So, I wrote a quick and dirty PHP script that converts this RSS export to an OPML list (see at the end of this post).

Source: vrypan|net|log » del.icio.us as feed manager

I haven't had much chance to dig into Dave Winer's new newsRiver tool for the OPML Editor, but I've at least installed it and nodded at it. It's not much more than the aggregator I used to use in Radio—but the big difference is the promise of fresh new development.

And yeah, I know, I groused about OPML versus XOXO not more than a few months ago. But, although I still would prefer cool things being done in XOXO, Reading Lists (in OPML) would appear to be one way to get the stratified River of News that I'm looking for. That is, a Reading List can be a news strata in an aggregator.

But, beyond my prioritized folders in NetNewsWire, a Reading List in OPML can be serialized and shared and subscribed. Yeah, aggregators can already import and export OPML subscription lists—but I've never seen this feature automated by subscription until now. This is the difference between an MP3 download and a podcast.

And, if you check out the above-quoted blog post, these could even be collaboratively maintained by way of del.icio.us and a simple RSS-to-OPML gateway service. I've got to say, this could be one of the niftiest bits of mojo to hit syndication and aggregation tech in awhile.

Smarty is on my evil list

2006-01-08T01:33:47+00:00

Although I have my grousing about PHP in general, I have to say that it makes much better template language as-is before pulling something like Smarty into the mix.

At this point, I consider Smarty to be evil—though I'm willing to admit that I haven't grokked it fully yet. And this is coming from a templating addict who has been at times in love with Template Toolkit in Perl, Velocity in Java, and Cheetah and Woven and ZPT/TAL in Python.

WordPress seems to have put native-PHP-as-templating to good use, so I've already got a favorite for comparison. But the use of Smarty in reBlog rubs me the wrong way. Of course, reBlog has a lot of highly-abstracted MVC architecture going on. And this, itself, is something I haven't much in my PHP explorations thus far. (In fact, I almost hate to see PHP chopped up in this J2EE-esque fashion, but that's another story. Oh, and reBlog rocks, by the way.)

But either way, Smarty makes this right-handed hacker feel like he's writing with his left. It seems to have a lot of weird conventions and terminology for which neither knowledge of PHP nor knowledge of other templating systems in general seem much help. It could just be a learning curve, but to me it seems like an odd sideways curve at the moment.

In fact, you know what? Smarty reminds me a lot of DTML in Zope. I hate DTML—it, also, is most firmly on my evil list.

Yuck. Again, it could be just that I haven't known Smarty long enough to like it... but yuck.

Archived Comments

Luka Kladaric
2006-01-08T11:55:40

I agree with you... I used Smarty in one project and am now using it in another work-in-progress... and I have to say, the decision to use it with the second was a bad one... Smarty is pretty cool for simple templating, but when you need raw power, it leaves you shipwrecked... I had to turn to some really black magic to get things done... Anyway, a rewrite of the first project is in negotiations and I'm 100% sure I will NOT be using Smarty there, either.
kellan
2006-01-08T14:34:57

Agreed. Using a templating languages with PHP makes no sense. (though I'm not sure the fact that WP does or doesn't do something makes it good design)
l.m.orchard
2006-01-08T15:03:33

Kellan: Oh, no, WordPress isn't an automatic gold standard for me... There's plenty that could be better about WP. (Not having checked out 2.0 yet) But, I just had a much better time setting up the templates for this blog than I have had in hacking around with templates implemented in Smarty.
Bob Aman
2006-01-08T17:41:17

Me, I just wish there was a templating system for Ruby for me to fall in love with. With all due respect to the Rails people, ERb just doesn't really do it for me.
Aristotle Pagaltzis
2006-01-08T21:04:14

Bob: port Kid to Ruby. :-)
Keith Gaughan
2006-01-08T21:28:05

What's moronic is that people go to all this trouble building templating systems when all you really need is extract(). Sad, really.
Bryce
2006-01-12T13:33:10

PHP as a template engine is more evil than Smarty. The main issue is that template authors may not be trustworthy. Once you include or eval() a PHP template the app is no longer in control and I don't see any way of structuring a PHP app such that it cannot be subverted from within.

The only defense seems to be using the tokenizer functions to build a PHP parser that validates templates against a "whitelist" of allowed operations. Savant's Restricted Compiler is about the only implementation out there, but it's not quite complete and seems to be more of a proof-of-concept than battle-tested code.

Whatever Smarty's faults, it solves the untrusted templates problem nicely.
l.m.orchard
2006-01-12T14:27:42

Bryce: Oh, I totally understand the implications of offering too much capability to template authors. That's just how much I dislike Smarty—I'm willing to risk it, rather than put up with that particular implementation. On the other hand, in my situation, I'm on the same team with the template developers, so we're all on the same team to sink or swim. I know it's not that simple, in principle, but it covers the 80% of the 80/20 equation.
Dougal Campbell
2006-01-12T15:25:25

Yeah, I was never able to get into Smarty either, and I've never been able to put my finger on why. Philosophically, I like the idea of using template engines, particularly if you're collaborating with a non-programming designer. I even wrote one (albeit a very simple one) myself, once. But I've never used any of the PHP template packages that I've looked at.
Bob Aman
2006-01-13T01:10:16

Aristotle: When TurboGears first came out, I almost did, believe it or not. But then I remembered that FeedTools eats all of the time I have available for open-source projects, and I figured someone else would probably do it for me anyways...
Aristotle Pagaltzis
2006-01-13T09:02:38

Hehe. Well, I plan to port it to Perl, in any case. :-)
Brian Grayless
2006-01-19T16:00:24

Smarty is really a waste of time. Some food for thought....

Smarty for Dummies?

Just write better code!!!
Tali Banana
2006-02-01T07:33:41
Well, quite opinionated I see.

Smarty solves some fantastic problems and the cacheing system is beyond fabulous at this level.

I've used Smarty on several projects and have been using it now exclusively for 4 years. Of course, I used others but it was getting sick coming to a new phase in a project and finding out the template system had become RIP....abandoned before their time.

Smarty has done two great things for me, and I'll speak about them until the day that PHP is abolished:
1. Smarty has reduced my code to a reasonably sized, scan-able, masterpiece. Having the logic and presentation seperate is so nice. I recently rewrote one of my old projects using smarty and all of a sudden it made sense! No more php in the html and no more html in the php. And that's something you can't solve without templates for you juniors out there.
2. Smarty's caching has turned monster db queries into non issues. Consider this. A top 10 list from all-time sales. I'm talking about searching a db with sales spanning 6 years for all items purchased and creating a top sales page. Easy! Smarty does it once and caches the results until I decide they need to be refreshed. Took a 4 second execution time down to less than 0.4 seconds. And all without having to implement a CMS or otherwise. Even if you have limitless resources to buy new hardware, caching like this still makes sense. And the intermediate php/html cache is brilliant. Everything is so much faster with smarty.
I can't deny, I love Smarty.
l.m.orchard
2006-02-01T11:43:36

Tali: I'm usually opinionated, but I try to listen & learn :)

The benefits you describe with Smarty would seem to come with any decent web development system. Separating logic and presentation are the general idea behind templating, and some sort of caching is a must no matter what you're using as a presentation-side tool. My beef is that, out of all the templating systems I've used, Smarty is one of the ugliest.
Tali Banana
2006-02-01T17:30:03

Smarty does need some standards redesign on the interface, as does php itself which has been well documented but slow going.

Coming from the PERL background and previous php template systems I've been very happy with Smarty in general, even if there are a few things that could be done to speed up the learning curve.

May be worthwhile jumping on the dev team and making some suggestions.
Aristotle Pagaltzis
2006-02-02T01:30:31

But why would you need a template system when you’re already working with PHP, which is itself a template system? Why do You don’t have to have different languages to be able to put the business logic in one file and the display logic in another.

Perl (not PERL), btw, has the Template Toolkit on the CPAN, which is the spiritual father of Smarty. (And it’s vaguely more defensible in Perl’s case becase Perl is not a template system; though constrated with Perl the deficiencies of TT’s mini language are just all the more painfully obvious.)

By all of which I mean to say that whether people use languages in stupid ways does not mean the language is the problem; not in either PHP’s or Perl’s case.

I also don’t see why you need a templating system to implement caching; in fact, having caching coupled with the template engine sounds like a bad distribution of responsibilities. (Indeed, Template Toolkit has no such facilities.) But there’s the Cache::Cache family of modules on the CPAN, which you can use to cache things completely independently of what is being cached – whether it be the output of a template, or the query results themselves, or anything else, which might not even have anything to do without any output you generate.
M.
2006-02-16T14:13:18

I've good experiences with the Expose PHP template engine (www.kwasd.nl/expose). It doesn't require you to learn a new language, because it uses plain PHP and still offers security, because it contains a tokenizer and, in fact, a small compiler. I used it for some projects and it really offered all I needed.

And for people who wonder why one would need a template engine: I work with designers and I don't want them to mingle application code. By setting up a shared folder which contains the templates, they can safely work on design and never need to touch my controller or module files. Especially since this template engine has a tokenizer/compiler, they can't call unsafe PHP functions (only those that you, the programmer, register).
Michal Migurski
2006-03-05T23:16:25

Smarty appealed to us because it offered a (relatively) clean way to separate the code from the HTML output. It's definitely got its problems. A large part of the choice was based on my perception of Smarty as having much thought & effort behind it, plus I like the idea of templates caching down to plain PHP. I've done a few PHP projects where I used PHP as the templating language, and frequently succumbed to the temptation to stash app logic into the templates where it didn't really belong. Smarty has a pretty extensive vocabulary, but has (so far) resisted the pull to incorporate more real-language features, like PHP did. It's worth remembering that PHP was originally supposed to be a template language for "real" code to be written in C. shudder

When we forked Reblog from Feed On Feeds, we were starting with raw-PHP display code mixed in with database calls, and it was a freaking disaster. One of my first design goals was to make it hacker-friendly, clarifying and documenting as many seams as possible so people could make localized changes more easily. I'd like to be using Python under the hood with Smarty on top, though there's a lot to be said for using commonly-known tools like Smarty in apps where you want to encourage modifications.

And Wordpress.... oh, Wordpress. My eyes roll back in my head when I'm forced to touch its convoluted innards. I pushed hard for a Reblog plug-in API specifically so I could cheerfully deprecate the old-style Wordpress plug-in. =)

OCD thoughts, syndicated

2006-01-07T21:49:04+00:00

Weirdest search referrer of the year so far:

ocd thoughts that i want to kill someone

The last hit on the first page of results goes to my blog entry, "Info Freako, or who’s already past arguing about syndication formats?"

To whomever you are: If this was a call for help, stay far away from syndication feeds. ("3000 TEST CASES!!!")

Archived Comments

Bob Aman
2006-01-08T17:44:34

The wierdest search referrer I've gotten so far was someone looking to "destroy all civilization," which, of course, led to my post on Civ IV.
Mark
2006-01-09T17:37:39

3315 test cases, but who's counting...

Sharing attention while reading feeds

2006-01-03T14:45:01+00:00

Instead of reading their individual selections of RSS feeds privately, everyone should be encouraged to publish those aggregated feeds on the Web. ... the simple act of publishing those aggregations then makes them available to others, and thus makes them amenable to network effects in a way that they never can be if they're kept private.

Source: » It's time to bury RSS | Software as services | ZDNet.com

My current two responses to the above:

I love my popular links summary feed.
Attention.xml is starting to make sense to me.

Putting these two together could be a very powerful tool for bringing network effects to feed reading.

First off, one of the most useful feeds I have is my private Popular Links feed. Touting my book again, I showcase the script behind this thing in Chapter 15. I've had this running for well over a year, and it's always my first stop in the feed reader. Almost without fail, this tends to surface what's buzzworthy.

Basically, this script scans all the current entries of all my subscribed feeds for unique hyperlink URLs found in descriptions and summaries. It collates all entries by these links, then sorts by the number of entries under each link. A threshold is applied, filtering for links pointed to by 3 or more entries.

At the end, I see a new feed entry displaying the most linked-to things of the moment. Think of this as a kind of real-time PageRank. How's that for network effects?

I think this is sort of how Memeorandum works, only this is constrained to the feeds in my subscription list. I've been considering making this script a full-on service: Upload an OPML export from your aggregator, get your own Popular Links feed. I've got all the parts laying around but I haven't yet had time to put them together—but if it sounds like something useful, and possibly worth clicking a Paypal donation button, let me know!

Secondly: Attention.xml makes a lot of sense with regard to the above-quoted article. When I first heard about Attention.xml, I merely cocked my head at it and made a confused sound. This was before I caught the microformats bug, and before I realized that I started reinventing it a bit in my own ramblings.

Basically, Attention.xml is a feedroll enriched with data about the entries you've lately read from each feed. It's in an XHTML-based format which—albeit ugly in my opinion and in need of more elegant microformat influence—is indeed viewable in a browser. In a sense, this format is an auto-blog of my feed consumption. I was looking for hacks for my NetNewsWire in AppleScript, when I found this Attention.xml generator. Seeing the output of that, it all clicked.

Combining Attention.xml with my Popular Links algorithm could be a very powerful thing, methinks. Rather than waiting for my friends to tip over the laziness point to blogging about something, I could digest their shared Attention.xml files and collate the links they've merely read. In this way, I could build an AttentionRank for various things, and cause the cream to rise to the top in my feed reader.

I'm pretty sure that I'm playing catch-up here, but this all suddenly seems hot to me. And not to mention, it seems neat that I have all the pieces laying around to build it. The only bad thing is that I just don't have the time to spare—I've already spent too much time writing this blog post!

Anyway, maybe you'll hear more from me about this soon.

Archived Comments

Daniel Nicolas
2006-01-03T15:39:45

I think you could get away with charging $5 for a service like that. Maybe even $10 if it was all web-based with cool ajaxyness flowing out of it.
Anne Zelenka
2006-01-03T19:26:21

That Ch. 15 script looks very useful, though I haven't had a chance to try it yet. The attention.xml thing... interesting but don't know how it would work with three-panel newsreaders like Bloglines. I'm not necessarily interested in a post just because I skimmed the latest articles in a certain feed. I didn't know before I clicked on the feed name whether I'd be interested in the latest stuff. Much of the time I'm not interested.

What I'd like to see is some smart recommendation engine that watches how I read, how much time I spend on a certain article display, whether I click through the links it has, and even provides a "boring" button like some other commenter suggested so it could do some Bayesian filtering. That's getting far beyond what you've described here but what you're describing is a start on it.

Another somewhat different issue is that Memeorandum works by limiting itself to a certain very popular and generic set of issues. Once you have one Memeorandum for politics or tech do you need another? Might individual quirks of attention (like the fact that you are interested in Detroit and I am interested in Maui) make sharing individual attention.xmls less useful? I've never really gotten the social sharing bug though so maybe I'm just showing a lack of imagination.

But just because Memeorandum already exists doesn't mean there isn't a real opportunity. While it's easy to find the hot conversations on politics and tech in the blogosphere, other domains are poorly aggregated and filtered, if at all. I follow mom blogs and feminist blogs and nothing like Memeorandum or digg or reddit exists for those yet there are regularly hot topics that ripple through them. What you describe could be used to build something for those and other domains on the web but is there room for everyone to do it individually? Or will people gravitate to one or a few sites that do a good job of highlighting popular and important conversations in certain well-defined domains?

These are just my raw thoughts on it... your ideas are inspiring.
Phil Wilson
2006-01-04T16:38:10

Presumably your subscription list needs to be of a certain size before you can start extracting interesting URLs or you'll just have 20 URLs mentioned twice and 50 URLs mentioned once.

If you were to just read the short head of your popular items list, would this actually be encouraging people to be infovores rather than normal information consumers (where you read or at least skim all of the items in your aggregator)? i.e. the data only starts working for you when you collect enough feeds. Gosh, you could have your own little long-tail on your desktop ;)
Bill Seitz
2006-01-06T16:45:16

have you look at at TailRank?

http://tailrank.com/

FeedMagick gains an RSS-to-JSON filter

2005-12-19T03:01:51+00:00

So, I put some more infrastructure behind FeedMagick and lifted a RSS-to-JSON idea from John Resig to transform from MagpieRSS parsing to JSON output.

Here's a spiffy JSON-enriched demo. It was surprisingly easy, all told.

I'm not sure how much time I'll have to really put into this project, but this is the first decent effort I've put into a URL-line suite in a long time. I'm trying to throw in some bits to make it easy to build, document, and use these feed processing commands.

Safari RSS database columns not so mysterious now

2005-12-13T21:08:06+00:00

The column names were inspired by the freebie game Monkey, Ninja, Pirate, Robot published by Atomic Sock Monkey. It’s fun! Try it!

The fact that there are weird column names was inspired by sqlite’s (then-)inability to add columns to existing tables. So the last time I had to break the schema to add a new column, I added four more as a reserve.

Needless to say, the contents, schema and very existence of that database are subject to change without notice, etc. etc.

Source: A comment by Jens Alfke

Remember when I wrote about the SQLite DB backing Safari RSS? I found a veritable zoo of natural enemies lurking in there. Jens Alfke was nice enough to leave the above comment today, and I thought it was worth calling out. I love hearing about this sort of trivia behind the code! :)

More on ignorant feed handling

2005-12-13T13:50:13+00:00

Part of the reason this whole must ignore thing with respect to feeds has me a bit fired up is because it seems like so few feed processing tools out there embrace this idea. And because of that, these tools are unfortunately brittle and prone to future shock.

For example, take Syndication.framework on OS X: Amongst the monkeys and ninjas and pirates and robots, you've got your standard title-date-link-description columns with a few other bits for good measure. But, where's the data from iTunes RSS extensions? Nowhere, gone, lost. If it was in the feed when Syndication.framework found it, it wasn't understood and so it wasn't retained after the parser finished chewing up and spitting the data into that DB table.

I've written about shared feed processing foundations before, but I don't think I've totally gotten the idea to gel in my head until now. Here's the thing: If you want feed processing tools that are useful for the general case, they have to be tolerant of things not understood. Rather than intrusively breaking apart and recasting feed data into a predetermined data structure, you've got to remain hands-off as much as possible.

This is what I did in FeedSpool. This code can subscribe to feeds, poll feed data periodically, and even work out which items in a feed are new—but it punts on everything else by only caring about where a feed starts and ends, and where its individual entries start and end. The rest is left in its original XML form. So, if there was data in there for iTunes? It's still there, because FeedSpool didn't know enough to do anything to it.

This is the major difference in feed processing model. Syndication.framework loses information when it encounters things it doesn't understand. FeedSpool retains the information, because it leaves things alone when it doesn't know any better.

Now, this is not to pick on just Syndication.framework. Despite the general-sounding name, this framework is pretty much just around to power Safari RSS and not iTunes or anything else. And as I said, pretty much every other feed processing framework and tool works in this manner. Just about everybody uses a destructive process when they parse and marshal feed data into local-idiom structures.

And this destructive process is fine when all you want to do is satisfy a specific purpose—display a few headlines, for example. But if you're trying to build general-case feed plumbing, this is unacceptable and untenable. If most everything in a feed item gets lost as it goes through a pipeline, we might as well abandon feed extensions entirely in favor of microformats stuffed into the description tag—an idea to which I'm not entirely opposed.

On the other hand, consider this: This past summer, Microsoft released an architectural overview about an OS-wide RSS framework in the upcoming Vista. Unfortunately, like many RSS and Vista related things from this past summer, all links have evaporated into a redirect to a single developers portal. I can't figure out why they didn't just redirect over to here, which took me a search and a few clicks to rediscover.

At any rate, there's one sentence in this overview that gives me hope for RSS as a general service in Windows Vista:

"It is also possible to access the item XML for applications that want to perform operations on the XML instead of using the item's properties."

So, dig that. If it works the way I hope it does, RSS in Vista will take care of the subscriptions for you, poll the feed data, grab new stuff—but then it leaves the data intact for you to process whatever new and unanticipated feed payloads that may arrive.

That's how it should work.

Archived Comments

Darren Chamberlain
2005-12-13T15:03:07

If you want feed processing tools that are useful for the general case, they have to be tolerant of things not understood. Rather than intrusively breaking apart and recasting feed data into a predetermined data structure, you’ve got to remain hands-off as much as possible.

It sounds to me like you're arguing for a triplestore and SPARQL. You'd stash everything in the triple store, and then your front end app just needs to be able to construct the appropriate query and process the results. More general and extensible than creating custom classes for filtering by specific fields, and I think if you were to ever write a general filter app where users can specify filter fields and values, you'd basically be reimplementing SPARQL.
l.m.orchard
2005-12-13T15:51:47

Well, a triplestore would be great if these feeds were RDF. But alas, with the exception of RSS 1.0, they're XML. I could play with trying to make transformations from XML to RDF, but that's getting back to a dangerously unlazy level of intelligence required to map unanticipated future feed extensions to RDF equivalents.
l.m.orchard
2005-12-13T15:52:52

(Although, it'd be really cool if syndication feeds were all RDF and not just XML. Triples would be a lot nicer to sling around than SAX parsing events.)
Darren Chamberlain
2005-12-13T20:09:47

Well, a triplestore would be great if these feeds were RDF. But alas, with the exception of RSS 1.0, they’re XML. I could play with trying to make transformations from XML to RDF, but that’s getting back to a dangerously unlazy level of intelligence required to map unanticipated future feed extensions to RDF equivalents.

Isn't there already a clean mapping between Atom and RDF? There's a list of integration ideas here.

On a side note, while Googling for Atom/RDF notes, I came across blogseive, which claims to be:

...a free web-based tool that creates new feeds by filtering, merging and sorting existing feeds. The BlogSieve engine accepts virtually every (valid) feed format, processed results are then exported into any feed format you choose.

I haven't tried it, but it claims to allow filtering.
l.m.orchard
2005-12-13T21:26:27
Isn’t there already a clean mapping between Atom and RDF?

Maybe, though I don't think it's official. And even it if is, it leaves out RSS. But even if it worked for RSS too, what about all the feed extensions that might be? I think RSS 1.0 had the right idea for extension modules in the RDF universe, but the world seems to be settling for XML.

I haven’t tried [Blogsieve], but it claims to allow filtering.

It does filter, but it does so destructively. (And that's not to mention the 7-8 step form I had to go through to start filtering. Definitely not a URL-line application.) But, with respect to their filtering and conversion, check out these feeds:
- http://del.icio.us/rss/deusx
- http://feeds.blogsieve.com/192/RSS0.91
- http://feeds.blogsieve.com/192/RSS1.0
- http://feeds.blogsieve.com/192/RSS2.0
- http://feeds.blogsieve.com/192/ATOM0.3
If you compare these to each other, you'll find information loss and even just plain corruption. The dc:subject elements encoding del.icio.us tags are gone, even in the RSS-1.0-to-RSS-1.0 transformation. And somehow, in the Atom version, they managed to jumble up titles and authors. Granted, my stuff doesn't do conversion yet, but I wouldn't want to do it like this.

I haven't tried it yet, but I'd have to guess that a podcast feed with iTunes and/or Yahoo! Media elements would get mangled in a very nasty way.
Sean Conner
2005-12-13T22:34:35

Wow, is it really 1985 again? I could have sworn all this was hashed out back then (only then it was a binary format, not a text-based format).

Sigh.
l.m.orchard
2005-12-13T22:43:22

Sean: Ooh, nice link! It's been awhile since I read about IFF, and I don't think I ever quite understood the concept.

But, this part certainly caught my eye:

Our task is similarly to store high level information and preserve as much content as practical while moving it between programs. But we need to span a larger universe of data types and cannot expect to centrally define them all. Fortunately, we don't need to make programs preserve information that they don't understand.

(Also, I miss my Amiga.)
Darren Chamberlain
2005-12-15T13:14:33

Maybe, though I don’t think [a clean mapping between Atom and RDF] 's official. And even it if is, it leaves out RSS. But even if it worked for RSS too, what about all the feed extensions that might be?

You could store the fully qualified entries, with the appropriate namespaces, and then define equivalencie using owl:equivalentClass. Then (I believe) a SPARQL query that extracted the rss:entry resources would also pick up the entries from Atom and the various RSS flavours. Although at that point you'd need a OWL-capable triplestore and library.

How to Hack RSS and Atom Feeds

2005-10-28T15:22:26+00:00

This article is an excerpt from the ExtremeTech book Hacking RSS and Atom. This feature explains what RSS and Atom feeds are and how you can build an aggregator, route feeds to your inbox and buddy list, subscribe to multimedia content, and more.

Source: ExtremeTech: How to Hack RSS and Atom Feeds

Woo hoo! Looks like the first chapter of my book has been excerpted on the ExtremeTech site. Check it out!

Google Reader vs Microsoft start.com

2005-10-26T16:13:21+00:00

You know how I dissed Google Reader last month? Consider it still dissed. In fact, checking in again on Microsoft's start.com makes me think Google has a lot to learn about making a feed aggregator. I still prefer my own in-progress aggregator UI, but start.com has me thinking about ideas to steal.

Archived Comments

Bob Lee
2005-10-26T18:12:20

How does start.com differ from Google's personalized home page (I don't use either, but you've piqued my curiosity).
l.m.orchard
2005-10-26T18:25:48

Well, from what I can tell versus Google's personalized home page, start.com seems more open to defining "Gadgets" along the lines of Widgets on Mac OS X Tiger. Though, that's really neither here nor there with respect to a "pure" feed aggregator. So, maybe start.com and Google Reader are apples and oranges to compare in the first place. (I have to admit I'd mostly forgotten about Google's personalized home page.)

But, the whole start.com UI thing just feels like a quicker, nicer presentation of feeds. I like the in-page detail overlay when you click on a headline, versus just skipping straight to the link. Google's UI feels pretty clunky to me.
Bob Lee
2005-10-26T19:03:19

Ah, the popup is kind of neat. I follow more than 250 feeds daily though. I need something like Reader (actually Reader wasn't fast enough, so I use NetNewsWire synced to Bloglines now).
l.m.orchard
2005-10-26T19:09:29

Oh, yeah, start.com is a bit too slow for me too—but it's faster than Google Reader. Currently, my favorite UI for speed is my own:

http://www.decafbad.com/blog_attachments/miniagg-1.jpg

But, I've been trying to think of ways I could tile feeds into columns to maybe pack more per page in front of my eyes. (ala start.com)
Nick
2005-10-26T20:55:32

I went from Bloglines to Googles for about 2 weeks, but I finally gave up on it, I just couldn't get used to the interface, and didn't like the way it that it just dumped the feeds at you without any order or control. I went back to bloglines for now, but I'd like to setup something on my own server.. (I picked up this really cool RSS book, just haven't had the time to read it yet.)

Could an iPod and AV cable fit in a DVD case?

2005-10-21T18:05:54+00:00

Being able to listen to audio while in the “natural” places you listen to audio (in the car, while walking/running, on the couch, etc.) is what accelerated audio Podcasts to the phenomenon it is today. Now that there is a way to view video the same way ... that’s going to accelerate that new type of user generated content as well. I’m positive. It’s a no brainer.

Source: Russell Beattie Notebook » My New iPod: Video Podcasting Is Going to Be BIG

I've been gradually easing into subscribing to vidcast feeds, starting to find some early greats out there like RocketBoom, Command-N, and OpenAlpha. I've also started hacking together feeds for great online video shows which lack feeds. And then, there's the occasional private hack I've done to record shows with a capture card or help catch up on TV shows I find on Usenet.

To watch all of these things in the living room, I use Xbox Media Center on my modded Xbox, networked to a file server in the basement. However, there are probably very few other people on the planet who'd bother setting up a rig like mine. So the combination of iTunes loading feed-delivered video enclosures onto an iPod equipped with an AV cable and a remote sounds like a genius combination for really jump starting the next phase of the vidcasting rise.

Archived Comments

Pete Prodoehl
2005-10-21T19:27:21

I really wish I had a enough spare parts of bucks lying around to build a MythTV box, because I could create quite a freaking videoblog aggregating machine out of it, and view it all on the big screen TV... Someday... Someday...
l.m.orchard
2005-10-21T20:29:08

Pete, well, I'm not doing all that bad with just an NSLU2, a 160 GB drive, and an Xbox. :)
Seth Mason
2005-10-21T20:59:13

Thanks for the tip about Xbox Media Center. I'll have to check that out.

How did you mod your Xbox? Or did I miss a post about it?
l.m.orchard
2005-10-22T01:43:09

Seth: Oh, I just went over to ModChipMan.com and picked up a Xecuter 2.6 and soldered it in myself. I upgraded the internal HD with one from a junker PC. Very cool to dump all my games onto the drive and leave them squirreled away in their cases. And then there's all the homebrew stuff like XBMC...
Seth Mason
2005-10-23T03:59:13

Thanks for the lowdown.
Steph_B
2005-10-25T18:48:47

I wanna thanks for the tip about Xbox Media Center too. It's very useful information for me because I'm going to create videoblog instead of my homepage.

Time-release Syndication

2005-10-21T03:45:54+00:00

Want to read Cory Doctorow’s new book, Someone comes to Town, Someone leaves Town, via RSS?

This chicklet will let you do just that…doesn’t matter when you first subscribe, this feed will deliver the book to your feed reader in the right order, a couple of chapters a day, over the next month. You could even subscribe via WINKsite and get it all on your mobile phone…

Source: Surfarama » RSS for serialized content

I've been toying around with a few ideas for something like this for a few years, but have never bothered to finish any of them. Russell Beattie had an interesting implementation of this too.

But, just in case I never actually do anything with this notion, here's a free idea—I wonder how many aggregators actually obey a 301 Moved Permanently HTTP status code?

Simon Willison wrote about this notion: A reader subscribes using a clean and simple feed URL, but the aggregator receives a 301 redirect upon the first GET. This redirect shifts the subscription over to a new feed URL with a unique ID for the reader, thereby uniquely tagging every new subscription to the feed. The aggregator should then continue to use this tagged URL for all future requests.

However, tagging a subscriber with an RSS radio collar implies that you'll have a database on the server maintaining notes tied to those IDs. That might be cool if I'm trying to gather some metrics, but for simple time-release syndication I don't really care about tracking a person. I want to track a start time.

What if this post-redirect feed URL had the time of initial redirect in it, and the time release was just a function of the duration since that initial redirect? Anonymous and no need for a database. My notion is that I'll just need a PHP script and a fully-formed "master feed" on which the time-release feed will be a time-bounded "viewport" informed by a little date math.

Or, maybe I'll have a pile of pre-prepared content in a folder and munge that into a feed. Imagine a podcast like Scott Sigler's Ancestor, all queued up and ready on the server, but doled out automatically on a weekly basis from the beginning from the point you subscribed. I'll leave as exercises for the reader issues of "obfuscating" URLs so that people can't easily skip ahead through the story.

Archived Comments

kellan
2005-10-21T04:16:34

My favorite example of this is Matt Webb's service which offers RSS feed of your personal light cone
Martin Atkins
2005-10-21T07:06:00

This scheme falls down in two ways. Firstly, people have a tendency to share URLs, and when they do they tend to just take whatever's in their user-agent (your timestamped URL) rather than your pretty entry point URL that lives only on some HTML page somewhere.

Secondly, lots of people use "shared aggregators", which reduces load on your server but requires that everyone is seeing the same content.

It's a nice idea, though. Its a shame that syndication tech ended up making it so tricky.
l.m.orchard
2005-10-21T10:54:22

@Kellan: Heh, I thought that one was kinda silly, but neat... It's in my subscription list.

@Martin: D'oh, that's right. Even podcasts are being heard via shared aggregators now like Odeo.
Pete Prodoehl
2005-10-21T17:14:17

Delta Park Project is a podcast that provides you with your own RSS feed url. I'm not exactly sure what they do with it, but for now I trust them not to do any evil.