0xDECAFBAD - Tag: xml

OPML reading lists in FeedMagick2

2007-10-17T07:22:47+00:00

For anyone who's interested: I've been hacking a little bit on FeedMagick2 again, with the latest addition being an OPML reading list feed blender.

What's an OPML reading list? Basically, it's the same as as OPML export of a feed reader's subscription list - only rather than doing a one-time import into another program, the OPML is itself treated as a live feed. A feed reader that supports OPML reading lists will continually check the list for updates and sync RSS/Atom feed subscriptions with its contents, maybe in a special sub-folder.

Here's a quick demo:

The itch I mean to eventually scratch is to replace the front page of decafbad.com with a live updating aggregation of the stuff I create and capture daily on the web. It'll be basically a self-assembling tumblelog pulled from many different services across the web. It'll also replace the footer of accumulated crud I've got on this very blog - which I thought was a good idea at one point, but now consider NeatLikeDigitalWatches.

With that in mind, the next thing I plan to develop is an hAtom module or XSL transform. This will turn the blended feed into an XHTML page. Maybe someday, hAtomic will launch, and I'll have a nice pretty style for the page too. Some time after that, I might work up a module that stows away dated historical archives of the feed and pages. I have further plans and ideas, but I'm trying to focus on the itchy spots first so that I might actually get something done in this round of serial enthusiasm.

Say hello to FeedMagick2

2007-04-30T03:06:32+00:00

Yeah, things have been basically silent around here thanks to post-work brain fryage and a general lack of things to say. Really, everyone else around the blogosphere seems to be covering things satisfactorily. However, I have been idly working on a new project over the past few weeks, namely a total rewrite and redesign of my format-ignorant feed filtering and munging kit dubbed FeedMagick.

You can find a demo installation of FeedMagick2 here and find it ready for checkout from SVN over here. It's basically just a step away from being a proof of concept, but I'm hoping to get around to fleshing out docs and battening down the hatches with tests. In any case, if my serial enthusiasm holds out, this thing could eventually subsume everything else I've done with feeds.

For now, peek at some of these highlights:

Master Personal Feed - One big feed blended from 10 other personal metadata feeds pulled from various Web-2.0-ish sites.
Feed to JSON via Magpie - Get feed data parsed by way of Magpie into JSON data structures
Flickr Favorites Feed - Feed of photos marked as favorites by a Flickr user, pulled via the API
jbox.com scraper - Pipeline composed of HTML Tidy and XSL to scrape jbox.com to build an RSS feed of new items for sale.

Beyond practical examples, there are some things under the hood that seem keen to me. Apropos of my pipes-via-web ramblings back in February, I'm trying out a few different approaches to pipelining feed content through processor modules. My original FeedMagick relied on feeding URLs to URLs as parameters. That, unfortunately, can be mighty cumbersome and inefficient. So, FeedMagick2 explores a few more approaches:

The first and obvious approach is to chain them together in a single script. So, I've got objects instances that pass content from one to the next. The thing is, the pipe works in reverse: The driver script asks the last module in the pipe for content, which then asks the one before it for content, and so on. At any point along the way, modules can cache the output of previous modules, and refrain from calling up the chain.
The second way to chain pipelines together is just like the first FeedMagick: Some pipelines start with fetching a URL. That can be an original feed, or a URL leading to the output of an antecedent pipeline. And, oh yeah, most pipelines are run via parameterized URLs, so there's that bit of handy recursion.
The third way to chain pipelines together is with HTTP POST: A pipeline can accept feed data via the request body of an HTTP POST, thus allowing antecedent pipelines (or even cURL scripts) to push data into the pipeline rather than getting pulled via URL. This is kind of like my years-old jiggery pokery with pipelines via XML-RPC, only much much simpler.

I'm also poking around at making all of the above available at the command line via PHP-CLI, and I'm having gratuitous fun exploring PEAR to roll my own stripped-down web framework. I still hate PHP, but I'm at least finding ways to entertain myself while I'm holding my nose. Of course, I find weird things entertaining.

And, as a side note, the only reason I'm using PHP is because I'd like to play around with the idea of the de facto WordPress installation requirements standard. That is: If you can run WordPress, you can run this. In reality, I don't think I'm there, but I'm hoping to get close. For one, I'm refusing to play with anything older than PHP 5.

Anyway, play with it, tell me what you think and give me a reason to keep hacking at it. :)

Archived Comments

jamesv
2007-04-30T11:39:15

Aw man, now I've got to port all the code you wrote while you were here over to this new hotness ;) I really like (and appreciate) the single script approach.

Is caching at a module level done automatically, or is that something I need to flag in my original call? Some installs of the original code base are now aggregating large sections of a pool of around 400 feeds, and eeking out even minor performances gains would be just lovely.
dRAUPP
2007-04-30T13:40:16

hawt.
l.m.orchard
2007-04-30T14:43:54

@jamesv: Take a look at the source to this big-ish pipeline, all the way at the end:

http://decafbad.com/2007/04/FeedMagick2/pipelines/masterfeed

You can basically slap a Cacher module at the tail-end or even middle of a long string of modules, and it'll cache the results of everything before it. I've got a cache lifetime set in the conf/config.php, and you can also set the lifetime in the Cacher parameters. There can multiple Cacher's per pipeline too.

This thing might not quite yet be even as stable / in working order as the original FeedMagick, but it might be worth poking at for you. :)
kellan
2007-04-30T16:02:31

For one, I’m refusing to play with anything older than PHP 5.

That must be nice.

And curse you, this looks interesting, now I've got to find time to look at it.
John Manoogian III (jm3)
2007-04-30T17:53:08

addendum to README:
Installation
cp conf/config.php-dist conf/config.php chmod a+w logs - RewriteBase /~lorchard/FeedMagick2 + RewriteBase /FeedMagick2
l.m.orchard
2007-04-30T17:58:49

@jm3: Ah! Good catch. I really need to eventually installer-ify that kind of stuff.
megalar
2007-08-05T12:25:34
Yesterday my host upgraded to php5 so I ran over to your svn dump and installed feedmagick2. After editing the $baseurl and rewritebase i tested it and it worked like a charm, so I took a nap. Upon waking I was gonna go play with it and see what hacks I could get going with it but something was fubar. Warning: fopen(/home/megalar/www/html/feedmagick/logs/feedmagick2-debug-20070805.log) [function.fopen]: failed to open stream: Permission denied in /usr/share/php/Log/file.php on line 216 are the errors I get as you can see @ my site. If it hadn't worked to begin with I would think module problems on my server. To my knowledge, and my host's knowledge, nothing has changed since the upgrade so I'm wondering if it is some sort of bug or a server problem that waited a few hours to reveal itself. The latter doesn't really make much sense to me but I can't rule it out since I'm not r00t on the box and am not 100% certain my host didn't bork something while I slept.
```
              anyhoo, your thoughts?
```

content sniffing sucks

2006-11-24T06:28:11+00:00

If you’re using FF2.0 go here and you’ll see why.
No, I don’t want to subscribe to a PHP template used to generate Atom feeds, thank you very much.

Source: snellspace.com » Blog Archive » Content Sniffing Sucks

I know this is just taunting the Happy Fun Ball I said I was done taunting , but there's a false-positive for ya. :)

XoxoOutliner and further outline addressing adventures

2006-11-15T08:07:12+00:00

Revised the addressing code a bit, adding a few new kinds of addresses and getting ready to support sub-outline updates. That is, fetch a sub-branch of an outline and then later post a change to that sub-branch using the same address. Needs more thought - ie. what happens if things move between fetch and update? - but here are a few more samples:

First is a straight linear index counting down from the top of the outline:
- http://decafbad.com/2006/11/XoxoOutliner/outlines/README;index:4?format=xoxo
Second is a navigation of outline structure, alternating numbers and letters:
- http://decafbad.com/2006/11/XoxoOutliner/outlines/README;level:3c4?format=xoxo

That's all for now. In my next round of enthusiasm, I may try stealing Tom Morris' Opath idea...

Archived Comments

Vincent D Murphy
2006-11-18T20:52:07

I think (and said as much on Tom Morris' site) that a fragment identifier would be a better solution, in which case Opath would be a fragment identifier syntax for OPML and XOXO. At least it would be the best solution from a REST/web architecture point of view..
l.m.orchard
2006-11-18T21:40:41

One reason I didn't use the #identifier URI syntax for suboutlines is because some gymnastics need to be done to get the hash through to the server from a browser. Otherwise, it gets treated as an in-page anchor. The semicolon syntax seems to work well for a set of path-segment parameters, and follows the standard (if I've read it correctly).

In either case, it works for me, and should be just fine in a REST context - the suboutline syntax here should always identify a single parent outline node as a resource, and will eventually work for GET / PUT / POST / DELETE.

Now I just need to implement a solution for the Lost Update Problem.
Michael Poremba
2008-09-18T23:45:56

Wondering if you ever completed your online outliner? Check out dynamiclist.com, a functioning but incomlete project I launched back in 2001. The editor is rich and works well. Been thinking of reviving now that all major browsers support the contentEditable tag.

XoxoOutliner and suboutline addressing

2006-11-13T09:34:02+00:00

Here's a feature I just hacked together for XoxoOutliner and plan to refine further:

Not entirely sure that this is how I want this to work, but these three URLs demonstrate the ability to address and fetch subsets of outlines. I'm hoping this will be a basis for selective transclusion in other outlines, or maybe even in a sidebar of a blog. (Which, depending on the blog software, might be built from outlines anyway.)

Sometimes the lazyweb delivers with a deluge

2005-12-19T23:15:03+00:00

Kellan: " '… and that’s not my code.' ouch"

Eric: "Right now, the only thing we have special for Magpie RSS is that we don’t serve Atom to that user-agent if the version is 0.5."

Rasmus: "I wrote a simple little PHP 5.1-based RSS parser a while back and it doesn’t have any problems with that ... feed."

Source: Comments on my previous post

Facepalm, headdesk. As it turns out, the issue I was having with my PHP XML parsing was that I was trying to cram raw gzip streams down its throat, caused by some odd things I was trying to do with HTTP headers copied cargo-cult style from a Python module I've tried reinventing in PHP.

In other words, completely braindead work by someone who should know better, even though he's relatively new to PHP—where he is defined as me.

But, kudos to the people of the lazyweb for taking the time to triangulate me and remove all doubt within scant hours of my post. Not counting a half-dozen private emails, I got very quick responses from the author of MagpieRSS, the CTO of FeedBurner, and the creator of PHP himself—each very nicely, clearly, and rightly saying:

"...and that's not my code."

Be careful when you attempt to invoke the lazyweb, because it might just respond!

Okay, okay, JSON is pretty hot

2005-12-19T04:17:55+00:00

The XML ~~purist~~ fanboy in me has had me pshaw'ing at JSON. But, now that the recent JSON release from Yahoo! reminded me of its existence and I gave it a shot myself, I have to admit that it's pretty hot—if only for the cross-domain bridging capabilities and the no-fuss parsing.

Although, I do worry about running into a poisoned payload someday that raids my cookie jar.

Archived Comments

fluffy
2005-12-19T06:50:57

JSON is currently the big Hot Thing at work. It's very good for some things, but IMO it's not nearly as robust as XML as far as generic interchange goes. Its big drawback is that it's not nearly as flexible, and since elements are either unordered or stored in an array, you have to agree upon the actual structure of the document before you send it across the wire (which isn't so much the case with XML where often all you care about is the nesting order which you can handle with XPath or similar).

It's GREAT for AJAX though, as long as you can trust the server of course.
Bill Humphries
2005-12-19T07:26:09

I thought AHAH was the current big thing. Just send HTML to the client and use Element.innerHTML.

Templates: Good or Evil?

2005-09-26T01:12:46+00:00

This cry and whine that draconian handling will break your page and make your users suffer for you if you have a single error is just another legacy of HTML we’ve gotten used to: our toolchains tend to be of the “glue strings together” (aka templates) variety. ... There should never be any part of your publishing toolchain just gluing strings together. Ever.

Source: Aristotle Pagaltzis in a comment on "The Future: HTML or XHTML"

There’s no rule that says templates must only be used to generate HTML. Indeed, many of the RSS and Atom feeds in the wild are generated from some form of template. They are never automatically-generated-behind-the-scenes using language bindings and are very rarely generated using some kind of DOM/SAX API.

Source: Web Services Infrastructure: Kid Templating

I've been meaning to write about this for some time now, though I've never had my thoughts together to any degree to mount a decent case for either side. Problem is, I haven't gotten much closer now, but I figured I'd at least post a few thoughts and conjure up a vague sketch of the issue.

You see, I think it all goes back to thoughts about which I posted almost three years ago. On the one hand, producing something like XML using "proper" DOM invocations and handwavings seems like the "correct" thing to do. Yet, on the other hand, using a templating system lets me get down to business much more quickly and with much more clarity and succinct code.

Yeah, templates provide a range of flexibility sufficient to aim the barrel at your own toes, while an API like the XML DOM keeps everything on the rails--but sometimes you know where you're going and don't need the rails to get you there. Furthermore, isn't it possible to make a template system that Does The Right Thing?

Anyway, it's rather apparent that I'm solidly in favor of templates: After all, a book of mine just hit the shelves which is just rife with template-based generation of RSS and Atom feeds.

My only issue, really, is that I feel vaguely guilty about it.

Archived Comments

rick
2005-09-26T02:27:37

Ruby has the nice Builder module for this (http://builder.rubyforge.org/). Ruby on Rails uses it for it's rxml templates. So, you still get the speed of templates, but you don't have to worry about those pesky xml rules. Here's a sample Atom 1.0 template used by Typo, a rails weblogging system: http://typo.leetsoft.com/trac/file/trunk/app/views/xml/atom10_feed.rxml
Aristotle Pagaltzis
2005-09-26T03:52:50

Using the DOM to build documents is awfully awkward. Don’t do that. Check Henri Sivonen’s suggestions just published in his HOWTO Avoid Being Called a Bozo When Producing XML. Instead of building a DOM, generate SAX events, using the program structure to ensure proper nesting – or instead of generating the SAX events all manually, generate them by parsing a static XML document and using certain interesting points in the stream (such as Processing Instructions) as hooks for inserting payload. The emission of synthesised SAX events can be generalised by writing a datastructure-to-SAX serialiser, so that you can build your data within your language’s native datastructures prior to outputting it, for maximum comfort. (Of course you serialise piecemeal too, f.ex. by outputting the head of a feed manually, then for each item, building the data structure and immediately serialising it, then emitting the final events to complete the document.) Depending on the complexity of the output, you could directly emit the otuput format or feed the events into to an XSLT transform that generates the full-blown thing from an easy to generate document structure. In both cases this can be done with or without the involvement of a serialiser as middle man. There are plenty of ways to make sure that the entire toolchain from one end to the other consists only of steps that conserve well-formedness, and they need not be any less convenient than using templates.
l.m.orchard
2005-09-26T04:16:35

or instead of generating the SAX events all manually, generate them by parsing a static XML document and using certain interesting points in the stream (such as Processing Instructions) as hooks for inserting payload.
Aristotle: Now here's where I think certain templating technologies get interesting, and may Do the Right Thing. My so-far-favorite templating kit, ZPT, wants well-formed XML as templates--although I think that restriction is unfortunately relaxed as a nod to HTML. On the other hand, my possibly-new-favorite templating kit, Kid, [demands well-formed XML as templates](http://lesscode.org/projects/kid/wiki/KidFaq#must-templates-be-well-formed-xml). In case you haven't played with them, these two template languages center around the idea that certain attributes on elements define where content provided in a data structure should be inserted / swapped in by the template engine. The engine handles character encodings and such to ensure that the well-formed template results in a well-formed document. Does this match up with what you say up there?
l.m.orchard
2005-09-26T04:28:49

I guess the other thing that bugs me about constructing documents via DOM or via generating SAX events—which I'd first discovered in this XML.com article by Uche Ogbuji—is how awkward and removed it is from the view-source XML I've gotten used to hacking around with. Of course, my distaste for programmatically generating things like this goes back to CGI.pm in my perl-hacking days. When I start writing code to generate data that could be mostly done with a template, it strikes me as tangling Model/View/Controller elements and introducing weird context shifts. (ie. h1() vs <h1 />) Not sure if that made sense...
Bob Aman
2005-09-26T05:40:24

Yeah, I was gonna say, Builder for Ruby solves most of the issues with using templates and xml. For RSS and Atom, there's also my Ruby FeedTools library, which makes it even more rediculously easy to get a valid feed up and running. I'm also very tempted to play around with Kid and see how I like it, and perhaps port it to Ruby. From what I've seen so far, it looks like perhaps the nicest templating system yet.
Aristotle Pagaltzis
2005-09-26T05:59:08

It does. I confess I actually kind of liked CGI.pm – because nesting is automatically taken care of without so much typing of end tags, and you can seamlessly weave maps into the code for output loops. I have to note though that I used it as a sort of template language in its own right – I never littered prints all over my code, I kept all the calls in a single place, in fact, usually a single expression. That is also why I find it strange that people create template languages for PHP, which was itself born as a template language. What matters is that the output generation is separate from the processing logic, whether or not the template and the code are in different languages. Anyway, I digress. What I wanted to say is, no, I hadn’t seen either ZPT or Kid. I had put off reading the lesscode.org article you quoted and went back to it after responding here. Now that I’ve read it, I admit I’m intrigued. I’ll have to look into Kid; it sounds like an interesting take.
l.m.orchard
2005-09-26T13:39:00

What matters is that the output generation is separate from the processing logic, whether or not the template and the code are in different languages.
Ah hah, yeah, that's what I was fumbling toward with the sleepy tail-end of my comment. Logic and presentation in separate blocks / files / etc. And then, my take is that if you're going to have your presentation/view separate from the logic/controller, you may as well code the presentation in a form as close to the goal as possible (ie. in XML or HTML, not in s-expressions or the logic implementation idiom)—especially since oftimes you've got separate people or teams working primarily on each. I suspect, however, that a templating system like Kid goes quite a ways toward solving the problem of "gluing strings together". I almost wish it had been further along / I'd been more aware of it before I'd reinvented my own wheels for my book using Python string templates and funky map classes.
Ryan Tomayko
2005-09-26T19:00:44

I suspect, however, that a templating system like Kid goes quite a ways toward solving the problem of “gluing strings together”.
That's the idea. The correctness of DOM/SAX based contruction with the ease-of-use of templating. The future of Kid is somewhat uncertain. I'm hoping to wrap up 1.0 and stabalize it for Kevin and TurboGears but what I'd personally really love to see is the general concept of "structured templating" reach a wider audience.
Ryan Tomayko
2005-09-26T19:02:17

Oops. Sorry about the rough formatting. For some reason I thought comments were in markdown. Edit away, Leslie.
l.m.orchard
2005-09-26T19:11:20

Ryan: Grr. Funny you should mention that—I thought comments around here allowed markdown, too. I think I need to fix that *and* display some copy explaining markdown availablility, since I think that was something about which Aristotle expressed some confusion/surprise before as well.
Aristotle Pagaltzis
2005-09-26T20:29:12

if you’re going to have your presentation/view separate from the logic/controller, you may as well code the presentation in a form as close to the goal as possible
Oh, I don’t disagree at all. I have been annoyed at the state of XML generation at large, myself. Kid looks very nice, except I have no use for it, being that I’m still a Perlista. (Python just doesn’t feel right to me – like wearing a badly fitting tuxedo.) I’m wondering how much work it would be to port it or a close copy to Perl…
I think that was something about which Aristotle expressed some confusion/surprise before as well.
Heh, yeah. Your MT installation permitted Markdown but no literal tags, which caught me off guard once or twice until I noticed. But thankfully it had a preview button, so I could figure it out. I hate how WordPress does not ship with preview button available and enabled by default, and the default configuration doesn't even mention the expected formatting anywhere either. I just suspected that your WordPress was vanilla and used HTML on that hunch – and it worked. Phew. Maybe you can install the gagdget that Ryan uses on lesscode.org? I love that.
l.m.orchard
2005-09-26T21:38:08

I just suspected that your WordPress was vanilla and used HTML on that hunch – and it worked. Phew. Maybe you can install the gagdget that Ryan uses on lesscode.org? I love that.

Your wish is my command. At least, in this instance. I pawed through Ryan's HTML source and got sufficient clues to install this thing and shamelessly steal a snippet or two. Let's see if this preview works and if it accurately reflects what this will look like when I post...
Alex Schröder
2005-09-27T11:22:06

I maintain Oddmuse, a wiki engine written in Perl, using CGI.pm to generate the HTML. Often people want me to switch to templates. At the moment, when you want to radically change the HTML for the stuff that is not the wiki content, ie. headers and footers, you need to override the Perl subs that I provide. Somehow that strikes me as natural and easy, but many of my users seem to disagree, preferring to learn a templating system instead of learning to write Perl code. I'm still undecided about the issue. At the moment I'm still sticking to the "write Perl code instead of templating" because being able to write Perl code will make so many other task easy.
l.m.orchard
2005-09-27T11:31:23

Alex: Well, from my perspective, I've worked on a lot of teams where you've essentially got two roles—software engineer (SE) and interface engineer (IE). Depending on the company and the team, the SE might understand Perl/Python/Java/SQL while the IE might understand HTML/CSS/Javascript/Flash. There's usually overlap, of course, but these are the roles on paper at least.

When you're trying to come up with an overarching framework which accomodates collaboration between these two sorts of people and their respective skill sets, it helps to have a bridge between the logic and the presentation which can keep the Perl out of the HTML and the HTML out of the Perl.

On the other hand... A project like Oddmuse likely has an entirely different user and developer base than the projects I work on at my day job :)

Hacking RSS and Atom is out!

2005-09-13T23:45:47+00:00

If you've been following along with me these past months, you might've gotten the gist that I've been working on a book. Well, lookee what landed on my doorstep late Friday afternoon--my author copies!

This package was actually supposed to have been delivered on Thursday. But, much to my consternation, I wasn't present to sign for the package. But now, I've got the surreal experience of holding a bound volume of around 600 pages with my name on the cover. And inside? There's all this stuff that once appeared on my PowerBook screen in MS Word and Terminal windows. (Oh yeah, and there're a few photos of my iPod thrown in for good measure.)

Funny thing is, this process started for me back in late December 2004. But, you'd barely know it, because--for whatever reason--I started off pretty cagey about the whole thing, and then plunged headlong into the effort. I did manage to make a quick announcement of authorship, but thereafter only managed to emerge for a few quick thought dumps.

I think part of my silence about things stemmed from a reticence to talk about or promise anything before I'd written it--and also because this has been my first time doing anything like this and I didn't want to flub anything up along the way. How I might've done that, who knows? But, I got a bit too busy to really worry about superstitions as the deadlines rolled on.

But now, it's out! And you should buy it! Lookee, lookee: It's already got a 5-star review from someone I don't even know!

For the moment, I'm going to wrap up this post, finish my dinner, and go for a walk with the girl--but, when we get back, I think I might post a few more details on what you'll find in my book debut.

Archived Comments

jose
2005-09-14T01:03:36

congrats, les, if anyone should have written that book it's you. i'm glad to see it's hit the shelves .. now to go find a copy and look it over :)
Jim Renaud
2005-09-14T01:54:39

Congrats Les. I am gonna need you to autograph mine!!! New site design, new box, new CMS... The world is your oyster!!! It's Les' world, we just live in it!!!
Matt Blackcustard
2005-09-14T02:52:08

Hell yeah!
Nick
2005-09-14T04:15:22

Just ordered mine...whens the book signing?.. Congrats man..
nf0
2005-09-19T21:41:49

I got my copy over the weekend. Some really good work!
l.m.orchard
2005-09-19T22:50:23

nf0: Hooray! Thanks for the kudos :)

Building an Address Book as a Modern Web App

2004-12-23T05:58:41+00:00

So, in the spirit of pico-projects, I've started building that address book application I mentioned awhile ago and I want to start writing about it as I go.

First off, hopefully you'll notice the quick diagram I threw together in OmniGraffle. This is a sort of rough sketch of the loosely-joined architecture I want to explore with this thing.

Data: This is where address book entries live.
Model: A set of objects encapsulating the data, this is how address book entries will be accessed.
REST API: Model objects exposed as resources identified by URI, serialized and deserialized as XML, and manipulated by GET / PUT / POST / DELETE methods.
XSLT Filter: XML data produced by REST API calls can be first passed through XSL at a given URL before being served up as a response.
HTML, CSS, JavaScript: Thanks to the XSLT filter layer, the XML vocabulary used to describe address book entries can be transformed into user interface presentation.
HTTP: Everything happens via HTTP...
Web Browser Client: ...and everything is viewed in a web browser.

Now, I call this a loosely-joined architecture because I want to stress that you should be able to swap out just about any part of this whenever you want.

Want the Data to be in MySQL? Fine. Want it to be in flat files? Fine. Just make sure the Model can cope while maintaining a consistent interface for the REST API. Want to change the user interface in the browser? Great-- ideally, all you have to do is change some XSLT files. I'm writing everything from the XSLT Filter down to the Model in Python. Don't like that? Fine. Rewrite it all in Perl, and hopefully everything from the XSLT up to the browser will still be useful to you.

At some point, you might even want to ditch the browser for a native desktop client. Fabulous! Just ignore everything past the REST API and HTTP, don't use any XSLT in the Filter, and use the API and XML directly.

I don't think any of this is particularly revolutionary-- although I thought it was when I first saw Amazon Web Services doing some of this, and I hope to throw a little GMail in as well. I hope that this will all be useful as I muddle through explaining what I'm doing. In the meantime, you can see me getting the stage set as I start checking things into my Subversion repository over here:

http://www.decafbad.com/svn/trunk/hacks/abook/

shortname=abook1

Archived Comments

fluffy
2004-12-23T07:35:40

Don't forget that with XSLT you could also rewrite it to SyncML and vCard and so on, so you could also sync it with external devices and iSync (assuming Apple finally opens up a third-party conduit system in Tiger, preferably one which doesn't require .Mac to function).
eliot
2004-12-23T09:12:34

I'm looking forward to what becomes of this little app. Address book apps seem to all be severely lacking (with the possible exception of OS X's app) and need some fresh thinking. Making the components flexible could be a great way for some change to come about. Keep us posted!
l.m.orchard
2004-12-23T12:41:13

Well, hopefully it's not too disappointing, but this address book app will be pretty anemic in terms of use as a serious app. Implementing an address book is just an excuse to run through the various technologies involved. However, there's nothing stopping anyone (including me) from enhancing the thing when I'm done and making it into a serious offering with vCard support and such.
Bill Seitz
2004-12-27T12:23:46

(Hmm, posted last week but it never showed up...) What are you thinking in terms of having this data store actually used by, say, your email client? Make it look like an LDAP server, maybe?
Chuck Conway
2005-01-02T20:37:00

How Funny! I've been thinking about doing the same thing. The only difference is I was thinking about doing address book, bookmarks and my RSS feeds. Those are the things I miss the most when I am away.
William
2005-03-05T05:16:08

I'm curious to see where this little app is going its still needs more work
Bill Seitz
2006-08-26T13:37:35

I see you working with Tarawa in there. Did you like it? Is it being worked on at all?

Cross-breeding XSLT and ZPT

2004-12-03T01:15:52+00:00

I've recently been doing some side work involving Zope and, along with the rest of the suite of technologies it offers, I've been happy to be working with Zope Page Templates again. I dabbled with them a bit when they first came out, and a Zope-free implementation named SimpleTAL was one of the core components of the iteration of my news aggregator which came before FeedReactor.

Out of all the templating and content generation approaches I've used, Zope Page Templates are my favorite yet. Pretty expressive, yet unobtrusive; nicely powerful, yet not quite something with which you'd want to write an entire application (and that's a feature, not a bug).

I've yet to be in a work-a-day team that uses ZPT-- but I can see where a lot of production, delegation, and integration issues would have gone much smoother had I used ZPT instead of Template Toolkit for the web app framework I created at a previous company. (Though I do have to say TT2 is very nicely done!) And where I am now, I spend most of my days trying to pummel ASP 3.0 pages into some semblance of logic/presentation separation-- I would certainly dive at the chance to dump VBScript and <% cruft %> for a bit of Python and ZPT. (But, you know, it's a living.)

A close second favorite is XSLT. I've really been hot on it lately, having worked it into the core of FeedReactor in place of SimpleTAL. And in other hacks, I've really come to appreciate it's role as a filter segment in pipelines between REST web services and URL-as-command-line invocations.

Granted, both ZPT and XSLT very different technologies, but they are often used in similar contexts. More than once, I've wished that XSLT was as simple as ZPT (i.e. less verbose and intrusive, more document centered), and I've wished that ZPT had some of the features of XSLT (i.e. ability to be used as a transforming filter).

Reading Ryan Tomayko's description of Kid got me thinking, and googling. One thing I turned up from a mailing list archive asked about an “XSL implementation of TAL?” It struck me as a tad nutty at first, but then I started having inklings that just maybe it could be done. (Whether it should be done, well...) But the kernel of the idea grabbed me: Instead of using TALES path expressions to look up values in Pythonic space, why not use XPath expressions to look up values from a supplied XML document?

This strikes me as such an obvious idea that someone has to already have done it and possibly rejected it for good reason. On the other hand, maybe this is the sort of thing Ryan's thinking about-- I wonder how hard it would be to hack this into Kid? It would give only a subset of XSLT's capabilities in trade for simplicity, and would only offer the “pull” approach, but it would give XML-pipelining to a ZPT-ish technology.

I think this is something I want to look into a bit further at some point.

shortname=crossbreedingxsltzpt

Archived Comments

Ian Bicking
2004-12-04T17:16:09

I've been meaning to write about this on my blog, as I've done something similar. Instead of using XSLT, I've used ZPT for the transformations. It was an application where the user created a document in a WYSIWYG editor, then we wanted to pull information out of the content -- like a table of contents, or a title. To do this I parsed the content into a DOM, then put some objects in the ZPT namespace that manipulated it. For instance, the ToC object took a tag name (through getitem) and returned a list of the content and id of those tags (it created ids if necessary, modifying the content). Then you could easily create a ToC by looking through and creating anchor tags from, say, all the tags in the document. It should be easy to expand with other transformations (all coded in Python, of course). The actual code was only like 20 lines of Python, maybe less, and easy to understand from both sides (ZPT and Python).
Petri Savolainen
2004-12-13T05:21:41

See http://zope.org/Members/DaddyGravity/PT_XPath It would be great to have that in SimpleTAL, too.
Ian Bicking
2004-12-13T11:43:33

Another thing you might want to look at: in the last few days there's been discussion on the ZPT mailing list about an extension to stylesheets (TERSE) for ZPT that introduces transformations.
Bertrand Delacretaz
2004-12-13T15:15:55

FYI, people from the Apache Cocoon and BXE projects are working on similar stuff, in the opposite direction: we're taking TAL-like templates and converting them to XSLT, adding simple "match" templates for declarative rules. It's only prototypes and experiments for now, but the results look promising. More info at http://wiki.apache.org/cocoon/HtmlToXsltExperiments and http://blog.bitflux.ch/archive/further-improvements-on-xsl-tal.html
Manuzhai
2004-12-13T16:07:54

I like XSLT a lot, myself, and I have advocated it in the past on my weblog, but it just doesn't seem to be very wide-spread. Which is a pity, I think the world would be much easier if more people used this *standard* way of templating. One of the bigger problems with it seems to be the verbosity, so I've been thinking of a more compact syntax, kind of like RELAX NG has the .rnc compact stuff. I don't know if it's very feasible, but it seems like that would be not very hard; just have some compact syntax which maps onto a real XSLT-sheet (it could be "compiled" and cached, if need be).
Colin Stewart
2004-12-13T23:47:18

Integrating an XPATH implementation into SimpleTAL shouldn't be too hard as the TALES and TAL implementations share a fairly simple interface. There are 6 methods you'd have to provide to the simpleTAL module and that's about it. I don't know if XPATH is the right approach though - how would things like tal:define work if the paths were pure XPATH instead of TALES? Being able to mix XPATH and TALES would work better I think. A more promising approach would be to integrate ElementTree so that the 'find*' methods were usable from within TALES. Making it so that '/mydoc/root/find/.//searchElement' works would be fairly easy, but getting '/mydoc/root/find/.//searchElement/attrib/firstAtt' to work would require more co-operation between ElemenTree and SimpleTAL. I'll have a think about this though as it sounds like a promising approach.
Paul Everitt
2004-12-14T00:45:18

Chapter 11 of Jeni Tennison's "XSLT and XPath On The Edge" book has a section called "Using Page Templates". It includes an example of using substitution points. Like some of the others in this thread, playing with doing merges using DOM IDs and other patterns. This lets you write "themes" that are simpler than even ZPT, as they contain no non-XHTML namespace elements. Thanks for the article, which also seems to have brought out some interesting comments and URLs!
Bill Seitz
2004-12-14T20:23:08

Don't forget you can run Python within IIS/ASP! http://webseitz.fluxent.com/articles/PythonViaIis
Colin Stewart
2004-12-15T22:49:30

I've put together an experimental build of SimpleTAL that integrates ElementTree to provide some of the XPATH syntax. It's just an experiment, but see what you think. (More detail here: http://www.owlfish.com/weblog/2004/12/15122004.html#20:59:59)

Next generation web apps using REST, XML, XSLT, and XmlHTTPRequest

2004-11-30T21:53:35+00:00

So, like I was saying: I've been working on FeedReactor and have been doing some things with it that I find rather interesting, independent of news aggregation.

One of the core goals I have for FeedReactor is to explore what it takes to build a web app that exploits principles of REST architecture. Having already sung the praises of XML-RPC, I wanted to get immersed in REST and see what all the hubbub was about. I've got some ways to go, but I think I understand the major concepts now, and it's a pretty nifty frame within which to work.

But, two other things I've added to my mix have really made things interesting for me:

XSLT filtering
The XmlHTTPRequest object

XSLT and REST make a really good pair, as Amazon Web Services already demonstrate. Inspired by that API (and earlier experiments), I use XML for all the input and output formats in my API and accept a query string parameter that contains the path to an XSLT file. When this parameter is supplied, the XML output by the API is first processed using the given XSLT. (Think of it like piping API output through xsltproc.)

So, with a properly constructed collection of XSLT, I can present a browser-viewable HTML user interface served up directly from REST API calls. Links, frame sets, and iframes present in the HTML lead the user from that call to the next XSLT-wrapped REST API call.

But, once the initial HTML-and-JavaScript payload reaches the browser, it gets better (ala Gmail):

On older browsers (if I happen to care about them), I can make new HTTP requests back to the server from JavaScript using iframes. In this case, XSLT filtering lets me retrofit the API's responses to the HTML-and-JavaScript crud I need to serve up to make things happen back in the browser client. Unfortunately, passing data to the API (which expects XML, not form submissions) is still a bit wonky and requires some hacks and exceptions involving hidden forms and such.

However, on the newer browsers, it's all about the XmlHTTPRequest object. With this facility, I can make clean asynchronous requests back to the REST API, including XML data in the request body if I feel like it. Responses are handled by JavaScript callbacks, which twiddle the browser DOM to update the user interface in response.

So, after the major initial contact with the API to supply the browser with HTML by way of XSLT, most future interactions take place in the form of direct calls to the REST API using XML. Although for some things, it's easier to just reload a page of HTML, it's nicer for most interactions to be handled via DOM manipulations in-place. I've been amazed at the Gmail-like responsiveness I get from FeedReactor when I'm skimming through news items, marking some as seen or flagged, and popping open the descriptions on others.

I suppose I shouldn't be amazed at the responsiveness, since I'm using some of the same techniques as Gmail. However, my daily-use installation of FeedReactor is presently running on an old 300Mhz Debian Linux PC at home, and it's taking me through the daily produce of 600 subscribed feeds faster than any desktop aggregator has yet. Of course, this is partly a product of my familiarity with the UI I've cobbled together, but... the server's running on a 300Mhz PC with 256MB of RAM! And the client is my 867Mhz G4 PowerBook, running Firefox or Safari, depending on my mood.

Although I can't see when I'll have time for it, I really want to explore this approach further using desktop apps on OS X and accessing the API from Flash movies (maybe using Laszlo). I'd also like to see how far I can go toward adapting the interface toward mobile devices like my Treo 600.

So anyway, this has been where most of my private hacking sessions have been taking me over the past year or so: combining HTML, CSS, DOM, JavaScript, XML, XSLT, and REST to build what I consider to be a next-generation web app.

Now, although I use FeedReactor on a daily basis to keep up with all my feeds, it's nowhere near any state suitable for public consumption. I add new subscriptions from a command-line script and still fiddle with the database directly for some operations. I'd like to have a personal-server version of it ready for use by some alpha geeks before or not long into the new year, but I'd like to share some of the things I've been doing with it before then.

With that in mind, I think I'll wrap up this entry and think about putting together a quick tutorial pico-project to demonstrate some of the concepts. Maybe an address book, or something equally simple-yet-useful.

Stay tuned.

shortname=nextgenwebapps

Archived Comments

Richard Rodger
2004-12-01T08:14:39

You should take a look at http://www.json.org which provides a really nice JavaScript-native way to exchange data using XMLHttpRequest.
mike
2004-12-01T14:23:44

This is def a powerful set of techniques. I hadn't worked with a lot of javascript in several years but recently I'd wanted to put a cleaner face on my proof of concept request icecast station: http://fiftyfly.mine.nu/RFM . An off hand comment somone had made regarding XMLHTTPRequest sounded so interesting I had to give it some thought. The result can be found here: http://fiftyfly.mine.nu/RFMamp.html . Faster, cleaner, far more flexible - this excercise has made me think about web services in a way I hadn't previously. Consider my eyes opened ;)

Using iTunes as a podcast aggregator, with a little help from XSLT

2004-10-08T17:07:49+00:00

So I had an idea for a quick podcasting listening hack on the way into work this morning. Check it out:

Take one list of RSS feeds in OPML.
Throw in a bit of XSLT.
Combine using xsltproc to make a playlist that works in iTunes.

And, oh yeah, I just happen to have an xsltproc web service laying around, so:

Supply a URL to your OPML in this form.
Get a freshly-built playlist.

Now, this has been barely tested and is the product of a ten-minute hacking session. There are likely an enormous number of things wrong with this. That said, iTunes does seem to open the playlist happily, and it looks like only new streams are added with repeated openings of the playlist.

You will want to be careful to ensure that your OPML is valid XML (mine wasn't, on initial export from iPodderX - escape those freaking ampersands in URLs already!), and I have no idea what would happen if any of the RSS feeds in your subscriptions turn up invalid.

Have I mentioned that, despite their unforgiving and sometimes fragile nature, I love XML technologies?

If this looks useful, maybe I'll work it over a bit more and pair it up with some python to handle actually downloading the MP3s and torrents.

Update: Oh yeah, and I'm expecting this will be useful with an iTunes smart playlist crafted along these lines:

Date Added in the last 1 days
Play Count is less than 1

Update #2: Another use I just found for this playlist, is on my Xbox Media Center. I generate this playlist via cronjob every few hours, and store it on an SMB share accessible to the XBMC. Voila! Listening to podcasts on my stereo system via the Xbox. Yeah, nothing big, just kind of nifty.

shortname=itunesxslt

Archived Comments

Kevin
2004-10-21T10:32:55

This is a great idea... My playlist seems to be formated all wrong, so i need to work on it. Any advancements on this one?

dbagg3: Please excuse the mess

2004-09-17T13:32:30+00:00

Wow. So it looks like there are some people starting to follow to what I'm doing with dbagg3, and they're showing me how woefully prepared I am for the attention from tinkerers who are actually trying to, you know, run my code. Things have been crazy busy for me at work, so I haven't been getting done what I've planned. But, I do need to pull a few things together and clean a few things up. I'll soon be answering the smattering of email I've gotten so far, but until then, a few quick thoughts:

My source control is a bit of a mess at the moment. Not only have I switched from CVS to SVN-- but even if you followed me in that migration, I've not kept committed code in working order. I already know that this is a horrible habit, but since no one's really been looking, I haven't been called on it until now. (Heh, heh--d'oh.) Planning this weekend (but hopefully today) to resolve this, so that moving forward, svn trunk will be (as far as possible) in a working state at any given moment.
I've hacked one of my dependencies, SQLObject, by applying a patch to support SELECT DISTINCT queries. This has understandably caused problems for some people who have no idea what I did. This patch has turned out to be essential, though I don't know if/when it will or would be included in a release of SQLObject. So... I wonder if I should dump my working copy of SQLObject into source control? Otherwise, applying the DISTINCT patch to your SQLObject install should work.
At some point very soon, I want to change the name of this thing to feedReactor. Yes, I know there's already a feedparser, and a feeddemon, and a feedburner, and someone's probably got a feedkitchensink in the works, but I like this name and want to run with it.

So, in the meantime while I straighten some things out, please excuse the mess and thanks for bearing with me!

shortname=dbagg3mess

Archived Comments

Christian Henz
2004-09-18T06:25:13

I think having a copy of your SQLObject tree would be most helpful. First of all, that patch applies (or at least _did_ apply a few weeks ago) to their development version, while you're using the 0.5 Version, right? I also 'back-ported' the patch to 0.5.2 (only the distinct part though), but I'm still getting errors... cheers, Christian
Christian Henz
2004-09-18T07:28:15

Woops, just tried it again with a fresh svn-upped dbagg3 and it seems to work now. Good stuff :-) cheers, Christian

Moving time: From CVS to Subversion

2004-09-16T15:29:04+00:00

So, I'm waiting for the other shoe to drop. After making sure things seemed reasonably stable post-server-move, I migrated my CVS repository here to Subversion. There were one or two tiny bumps in the road-- such as a default setting in Apache to deny access to anything starting with .ht (ie. .htaccess)-- but so far, so good. ViewCVS appears to support Subversion, and I've also discovered an alternate frontend called WebSVN. I like ViewCVS better, but WebSVN offers RSS feeds on commits.

One consequence to this move is that soon, when I take down everything related to CVS, I'll have plenty of broken links (since I frequently link to ViewCVS pages for my projects). So, I think my next step will be to set up some redirects to a reasonable number of things for continuity's sake.

So, anyway... if you've been keeping up with dbagg3, the action's not in CVS anymore. It's here:

http://www.decafbad.com/svn/trunk/dbagg3/

Alternately, you can also peek at things here:

(Can you see one reason why I like ViewCVS so much better?)

Archived Comments

kasia
2004-09-16T11:53:06

ViewCVS is much better than WebSVN, you could create RSS feeds with a relatively simple post-commit hook, or even just modify the email changes post-commit hook to also generate RSS feeds..
Ian Bicking
2004-09-16T12:50:14

One PITA with ViewCVS is that it has to have write access to the repository, and having multiple users access the repository can cause the permissions to get out of wack and make the repository unavailable. I don't know if WebSVN uses the client svn libraries (i.e., accesses the repository through the server, instead of accessing repository files directly), but if it does then I would probably stick with it. Though if you are using Apache, I suppose it's all being accessed as the same user, so it shouldn't be too much of a problem.
Simon Willison
2004-09-16T13:19:41

Have you looked at Trac? http://projects.edgewall.com/trac svn browser, bug tracker and wiki rolled in to one. It's very sexy.
Ben Weekes
2004-09-16T13:19:45

I suggest that if you want to avoid breaking URLs again in future you take the "svn" out of those URLs. I generally prefer WebSVN; it seems to fit the "Subversion Way" a lot better than ViewCVS, which bends it to a CVS way of thinking about things to some extent. Mostly, though, they're essentially the same thing. WebSVN's theme and UI needs a little work, but other than that I find it good. Providing both is nice, though! I'm guessing you're picking on WebSVN's horrible URLs... and I agree with you on that. It has been an intention of mine for a while to take a look at it and see if I can't use some mod_rewrite magic to make it nicer. I don't much like PHP, though, so I've not got around to it yet.