0xDECAFBAD - Tag: hacks

Social Media Cyborg

2011-11-04T21:12:34+00:00

TL;DR: Since Google Reader lost it's sharing, I've decided to revisit my "social media strategy". I'm plugging lots of things into other things. I feel like maybe I should write about this stuff.

Started diagramming all my social media flows on LucidChart.

This is what I do for fun

Since I've mentioned in a few places that I'm a weirdo, I thought I might spend some time explaining exactly what I mean by that. You see, one of my hobbies is turning myself into a human content filter for friends on the web.

And, honestly, having my sharing habits shaken up by the Readerpocalypse makes it fun again.

The image over to the right is an attempt I made to diagram some of my social media flow. It's incomplete, though: I'm a cheapsake and ran out of free objects in my diagram. So, I had to leave out at least a half-dozen more services and a dozen additional connections. (LucidChart is very nice, by the way.)

In other words, ifttt.com is awesome

In particular, I've finally spent some time with ifttt.com. It's like Automator for the social web. It lets you define event triggers from one site that cause actions on another site.

So, I've gone crazy and created a bunch of tasks that make things like the following possible:

Pinboard bookmarks based on favorites from Twitter, identi.ca, YouTube, Last.fm, and my Kindle highlights. I'm turning my Pinboard account into both my personal social media archive and a Google Reader sharing replacement, and I don't even have to beg for new features at Pinboard.
Tumblr photo posts based on Reddit up-votes, Flickr favorites, and Pinboard bookmarks tagged "to:tumblr". Since creating it, I've left my Tumblr almost entirely neglected—but now with ifttt.com, I can route almost every nifty image I find over to that thing. It's the right side of my outboard brain to the Pinboard left.
Published items from from my private installation of Tiny Tiny RSS end up shared on Pinboard. This pretty much scratches my itch for feed reader sharing, since Google nixed those features from Reader.
Starred items from tt-rss get sent to Instapaper, so I can read them on my Kindle later. If email delivery of documents start working, then I'll have a personal long-attention span newspaper delivered regularly to that gadget.
Posts from Google+ get copied over to Facebook, because I don't want to pick sides.
In fact, almost everything I do everywhere eventually ends up posted to Facebook, because I almost never post anything in person over there. I do have friends and family over there and respond to comments, though.

What's the point?

The important principle here is that I'm lazy. If there's a single button to click somewhere that expresses my happiness about a thing, I'd like to click it and have that happiness shared and archived elsewhere without me doing much (or anything) more. The scheme I have now feels pretty good for that, and ifttt.com does a huge amount to help.

The other important thing is that I am at the center of these connections. I don't live entirely on any single service, and anything I care about is archived where I can easily grab it for backups or mashups. Should any particular service node in this web fail, I can probably do without, find an alternative, or build one myself.

My goal is to eventually replace as many of these nodes as possible with self-hosted or at least Open Source derived services. (Though, of course, I'm also lazy and my friends are where they are. So, that goal will take awhile.)

One more thing...

I've got an idea for my next book simmering. And, since I've so far managed to write books about the major things I've done for fun, I feel I'd be remiss if I skipped this stuff. I mean, I sort of covered the same ground all in my first and second books, but things have progressed since then.

What this means is that all the above feels like a warm up for the coding and prosing I'll need to do. I just need to get my shit together and carve out the time to make it happen.

Ficlets enhanced author feed, an XSL scraper hack

2007-04-05T05:37:00+00:00

I've been trying to get myself serious about writing and even set up a personal slush pile for my output. Then, I found Ficlets, and spewed a few quick starter stories there. And then... I stopped. I'm hoping to pick it up again very soon, but I guess that's the nature of my serial enthusiasm—it doesn't just apply to hacking.

So, here's something ironic: I just spent a few hours tonight throwing together a hack for Ficlets. See, Ficlets runs on original stories, comments, ratings, and sequels and prequels to stories. You can get an Atom feed of an author's stories and a feed of comments—but it seems like there's no way to get notified of prequels and sequels, which are a very gratifying part of the whole shebang.

With that in mind, check out this RSS feed:

http://decafbad.com/2005/12/FeedMagick/www-bin/ficlets.php?author=l_m_orchard

That's a blend of all my stories, comments on my stories, as well as prequels and sequels found for my stories. Subscribing to that feed will give me updates whenever there's anything new in all the above. It's thrown together using a semi-crazy mix of my FeedMagick package for caching, and some XSL for scraping. If you'd like a feed like this of your own, just replace l_m_orchard for your own author name in the author parameter.

Please note, however, that this little service is hosted on my site and may go away at any time for any reason.

In case you're interested in what's under the hood, here's the quick and dirty XSL that's behind it:

http://decafbad.com/2007/04/ficlets.xsl

This thing's made possible because the Ficlets feeds are XML, and so are the XHTML pages happily infested with microformats. If they ever go invalid, this scraper breaks. But, that's the nature of scrapers, and it works for now. Oh, and although they provide Atom, this feed is RSS 2.0. Why? Because it was easier that way. I might put some more effort into an Atom feed, but my itch has so far been scratched.

Archived Comments

megalar
2007-04-05T09:31:48

I had already decided I was gonna grab your FeedMagick source and try to do something with it but this just gives me more incentive as a Ficlets lover. I recently read your thoughts on pipes and feel much the same way => meh to GUI, especially a buggy one. It seems much more satisfying to just tell php what I want done and watch it go rather than spend 20 minutes fiddling with a silly applet.
l.m.orchard
2007-04-05T14:04:06

@meglar: Well, just to warn ya - FeedMagick needs a lot of work. :) I've got ideas for it, but have yet to get back around to it.
Kevin Lawver
2007-04-05T14:59:37

That may be the coolest thing ever. Good job!
l.m.orchard
2007-04-05T15:14:00

@Kevin: One thing that might not be cool is that this XSL hits Ficlets.com once for each story in the feed to pick up the additional info, but I try to cache the feed on my end for about 20 minutes per author... so hopefully it doesn't put any more stress on your site than I might have done manually in opening all my stories in tabs to check for comments and prequels/sequels. :)
Kevin Lawver
2007-04-05T15:44:25

That's fine. I think we can handle it. We'll get prequels/sequels added to either the author feed or their own feed in the near future.
l.m.orchard
2007-04-05T21:05:26

@Kevin: Yeah, I think the bare author feed is a good thing to leave alone. But, a personal author feed with comments and prequels/sequels and possibly notes would be excellent! You know, basically, notifications of significant events that'd bring me back to the site.

I'm also thinking it might be neat to gently spider through prequels and sequels on stories to map them out and track progress, but that's just a future hackery idea.
Roger Costello
2007-04-06T13:54:55

Is there any way for me to view the documents you are mashing up to generate the resulting RSS document? I am particularly interested in seeing how Microformats are being mashed up. Thanks!
l.m.orchard
2007-04-06T14:25:44

Roger: Sure... Really, the only things I'm mashing up are the author Atom feed, like mine here:

http://ficlets.com/feeds/author/lmorchard

And then, I chase down each story in the feed, like this one:

http://ficlets.com/stories/1763

You can view source on a story page and see all the information sprinkled in there - look for 'abbr' tags and class names like 'pubdate' and 'hentry'. The pages are also valid XML, so they can be manipulated right in XSL.
Bill
2009-01-28T00:36:02

I'm sure that many Ficlets fans are sad that Ficlets.com is no more. We are in the process of constructing Blurbtree.com a site that might be helpful for them. It's not a replacement for Ficlets, but it's a clean start and we'll build the site based on the feedback we get from the online community. Please give it a try and let us know what we can do to make it fit your needs. You can also get more information at our Blurbtree FAQ.

i'm in the nintendo ds homebrew club now

2006-09-19T16:16:12+00:00

Passscard is the perfect flash card boot-up solution for DS LITE & NDS . It is compatible with most types of flash cards including M3, Neo, SC & G6 etc.

Source: Divineo.com - Product Information

I've just received a Passcard 3 in the mail. Nintendo DS homebrew, here I come! I'm particularly excited about NitroTracker and ScummVM. Though I've only got a GBA Movie Player and a 128MB CF flash card from a camera, that seems enough to run most of this stuff so far. Unfortunately, it just sticks out from the Nintendo DS Lite by an inch or so - almost half the depth of the handheld itself! I think one of my next purchases may be a more compact and capable SuperCard MicroSD, which fits flush in the Nintendo DS Lite and accepts MicroSD expansion cards.

Archived Comments

foosel
2006-09-24T22:43:08

Congratulations on the Passcard ;) I got one myself a couple of months ago, including a M3 CF, and playing Monkey Island on the DS rocks ;) I have to warn you though concerning SD based units, there were a couple of issues not so long ago with several homebrew apps (including Scumm), and write-support for SD is afaik still rather new in the standard fat-lib used by homebrew devs. So you might want to make sure that your most beloved or interesting apps will actually work like expected prior to spending loads of money.
Adam Mokan
2006-09-28T04:21:30

How does the Passcard 3 fit in the DS? I've heard that its a real tight fit and you really have to force it to "click".

I'm getting ready to break down and buy some homebrew stuff for mine.

Ajaxitagging

2006-08-16T13:12:07+00:00

Ever since I switched over to a new CMS back in February, I’ve been tagging all my journal entries. Until now, I haven’t been doing anything with those tags apart from exposing them in category elements in my RSS feed. Now that I’ve got a good head of steam going with my tags, I’ve decided to play around with them a bit.

Source: Adactio: Journal - Ajaxitagging

Book pimping time: In Hacking del.icio.us, I've got something not entirely unlike the above-linked hack. (Which, by the way, is good work!) Mine is much more basic, though - with less microformat, progress bar, and yellow fadey goodness. And, instead of AJAX, my hack uses JSONP. The Related Links hack in Chapter 9 of the book shows you how to include lists of the last few bookmarks you've posted under each tag / category / keyword used in a blog post.

Stupid fun with fax machines

2005-12-15T21:55:58+00:00

Random Stuff Sent From Your Fax Machine
Fax something interesting to 1-510-545-0990

Source: Fax Toy - Random Stuff You Fax To Us

It's fun to fax anonymously! And it's fun to have yet another opportunity for meta. First person to fax their ass from the Christmas party loses.

Yahoo for stalking iPods in the hands of couriers

2005-11-03T18:29:12+00:00

If the new Yahoo! Maps were really cool, they'd've already done a mashup with DHL to let me obsessively stalk the courier in whose truck my returning iPod repair currently rests.

Come on, get with the program! When we've got GPS watches, "Status: With delivery courier / Location: (blank)" is no way to track a package! In the meantime, my repetitive information injury is flaring up as I refresh, refresh, refresh, refresh...

Could an iPod and AV cable fit in a DVD case?

2005-10-21T18:05:54+00:00

Being able to listen to audio while in the “natural” places you listen to audio (in the car, while walking/running, on the couch, etc.) is what accelerated audio Podcasts to the phenomenon it is today. Now that there is a way to view video the same way ... that’s going to accelerate that new type of user generated content as well. I’m positive. It’s a no brainer.

Source: Russell Beattie Notebook » My New iPod: Video Podcasting Is Going to Be BIG

I've been gradually easing into subscribing to vidcast feeds, starting to find some early greats out there like RocketBoom, Command-N, and OpenAlpha. I've also started hacking together feeds for great online video shows which lack feeds. And then, there's the occasional private hack I've done to record shows with a capture card or help catch up on TV shows I find on Usenet.

To watch all of these things in the living room, I use Xbox Media Center on my modded Xbox, networked to a file server in the basement. However, there are probably very few other people on the planet who'd bother setting up a rig like mine. So the combination of iTunes loading feed-delivered video enclosures onto an iPod equipped with an AV cable and a remote sounds like a genius combination for really jump starting the next phase of the vidcasting rise.

Archived Comments

Pete Prodoehl
2005-10-21T19:27:21

I really wish I had a enough spare parts of bucks lying around to build a MythTV box, because I could create quite a freaking videoblog aggregating machine out of it, and view it all on the big screen TV... Someday... Someday...
l.m.orchard
2005-10-21T20:29:08

Pete, well, I'm not doing all that bad with just an NSLU2, a 160 GB drive, and an Xbox. :)
Seth Mason
2005-10-21T20:59:13

Thanks for the tip about Xbox Media Center. I'll have to check that out.

How did you mod your Xbox? Or did I miss a post about it?
l.m.orchard
2005-10-22T01:43:09

Seth: Oh, I just went over to ModChipMan.com and picked up a Xecuter 2.6 and soldered it in myself. I upgraded the internal HD with one from a junker PC. Very cool to dump all my games onto the drive and leave them squirreled away in their cases. And then there's all the homebrew stuff like XBMC...
Seth Mason
2005-10-23T03:59:13

Thanks for the lowdown.
Steph_B
2005-10-25T18:48:47

I wanna thanks for the tip about Xbox Media Center too. It's very useful information for me because I'm going to create videoblog instead of my homepage.

RedHanded » Announcing the MouseHole Proxy

2005-10-05T11:14:25+00:00

MouseHole is a scriptable proxy. Like Greasemonkey, but scripts are in Ruby.

Source: RedHanded » Announcing the MouseHole Proxy

MouseHole looks pretty freakin' sweet. It's doing one of the things I never got around to making AgentFrank do. Every now and then I get an itch to revisit that project (or, more likely, its Pythonic sibling).

Archived Comments

Paul
2005-10-05T13:49:56

Hey Mr. Hacking RSS ,

You may want to change the relative <href="af"> because it doesn't resolve very well from inside an RSS reader (like Bloglines).

;oD
l.m.orchard
2005-10-05T14:13:20

@Paul: D'oht. That wasn't supposed to be an <a href="af">.

Looks like I mixed up my ()'s with my []'s in Markdown when making the link reference. Should be fixed now!

Building an Address Book as a Modern Web App

2004-12-23T05:58:41+00:00

So, in the spirit of pico-projects, I've started building that address book application I mentioned awhile ago and I want to start writing about it as I go.

First off, hopefully you'll notice the quick diagram I threw together in OmniGraffle. This is a sort of rough sketch of the loosely-joined architecture I want to explore with this thing.

Data: This is where address book entries live.
Model: A set of objects encapsulating the data, this is how address book entries will be accessed.
REST API: Model objects exposed as resources identified by URI, serialized and deserialized as XML, and manipulated by GET / PUT / POST / DELETE methods.
XSLT Filter: XML data produced by REST API calls can be first passed through XSL at a given URL before being served up as a response.
HTML, CSS, JavaScript: Thanks to the XSLT filter layer, the XML vocabulary used to describe address book entries can be transformed into user interface presentation.
HTTP: Everything happens via HTTP...
Web Browser Client: ...and everything is viewed in a web browser.

Now, I call this a loosely-joined architecture because I want to stress that you should be able to swap out just about any part of this whenever you want.

Want the Data to be in MySQL? Fine. Want it to be in flat files? Fine. Just make sure the Model can cope while maintaining a consistent interface for the REST API. Want to change the user interface in the browser? Great-- ideally, all you have to do is change some XSLT files. I'm writing everything from the XSLT Filter down to the Model in Python. Don't like that? Fine. Rewrite it all in Perl, and hopefully everything from the XSLT up to the browser will still be useful to you.

At some point, you might even want to ditch the browser for a native desktop client. Fabulous! Just ignore everything past the REST API and HTTP, don't use any XSLT in the Filter, and use the API and XML directly.

I don't think any of this is particularly revolutionary-- although I thought it was when I first saw Amazon Web Services doing some of this, and I hope to throw a little GMail in as well. I hope that this will all be useful as I muddle through explaining what I'm doing. In the meantime, you can see me getting the stage set as I start checking things into my Subversion repository over here:

http://www.decafbad.com/svn/trunk/hacks/abook/

shortname=abook1

Archived Comments

fluffy
2004-12-23T07:35:40

Don't forget that with XSLT you could also rewrite it to SyncML and vCard and so on, so you could also sync it with external devices and iSync (assuming Apple finally opens up a third-party conduit system in Tiger, preferably one which doesn't require .Mac to function).
eliot
2004-12-23T09:12:34

I'm looking forward to what becomes of this little app. Address book apps seem to all be severely lacking (with the possible exception of OS X's app) and need some fresh thinking. Making the components flexible could be a great way for some change to come about. Keep us posted!
l.m.orchard
2004-12-23T12:41:13

Well, hopefully it's not too disappointing, but this address book app will be pretty anemic in terms of use as a serious app. Implementing an address book is just an excuse to run through the various technologies involved. However, there's nothing stopping anyone (including me) from enhancing the thing when I'm done and making it into a serious offering with vCard support and such.
Bill Seitz
2004-12-27T12:23:46

(Hmm, posted last week but it never showed up...) What are you thinking in terms of having this data store actually used by, say, your email client? Make it look like an LDAP server, maybe?
Chuck Conway
2005-01-02T20:37:00

How Funny! I've been thinking about doing the same thing. The only difference is I was thinking about doing address book, bookmarks and my RSS feeds. Those are the things I miss the most when I am away.
William
2005-03-05T05:16:08

I'm curious to see where this little app is going its still needs more work
Bill Seitz
2006-08-26T13:37:35

I see you working with Tarawa in there. Did you like it? Is it being worked on at all?

XPath based Python dictionaries, on loan

2004-09-01T10:47:41+00:00

So Kimbro Staken posted this nifty idea to build XPath based Python dictionaries to access XML data as a part of his incredibly nifty Syncato microcontent management system. Eventually, I've really got to break down and get that thing built and running on my server and my laptop-- it really seems like I'm reinventing so many wheels by not basing dbagg3 on it.

But, while I'm in the process of wheel reinvention, how about I borrow Kimbro's idea? I just threw together a quick class called XPathDict, based on libxml2. It works a little something like this:

feed_xd = XPathDict(file="sample-atom.xml")
for entry_node in feed_xd.nodes("//atom:entry"):
    entry = XPathDict(doc=entry_node.doc, node=entry_node)
    print "Title: " % entry['atom:title']
    if 'atom:author' in entry:
        print "Author: " % entry['atom:author/atom:name']

xml = """
   <dbagg3:user xmlns="http://purl.org/atom/ns#" 
            xmlns:dbagg3="http://decafbad.com/2004/07/dbagg3/">
        <name>deusx</name>
        <email>deus_x@pobox.com</email>
        <url>http://www.decafbad.com/</url>
        <dbagg3:prefs>
            <dbagg3:pref name="foo">bar</dbagg3:pref>
        </dbagg3:prefs>
   </dbagg3:author>
"""

map = (
    ('userName',  'a:name'),
    ('userEmail', 'a:email'),
    ('fooPref',   "dbagg3:prefs/dbagg3:pref[@name='foo']")
)

xd = XPathDict(xml=xml)
xd.cd("/dbagg3:user")
print xd.extract(map)

#    {'userName'  : 'deusx', 
#     'userEmail' : 'deus_x@pobox.com', 
#     'fooPref'   : 'bar'}

There isn't any spectacular code behind all this, and the idea was Kimbro's, but it's working. It's also incredibly convenient, especially with the little XML-to-dict extraction map method I whipped up. This would take a bit more work to pry it out of its current context, such as turning the hardcoded namespaces into an option, among other things. But, here's the code for you to peruse.

(I got hooked early on subverting in-built language constructs from perl's tie facilities, and C++'s operator overloading. Now I'm loving Python's special class methods. Someday, maybe, I'll actually get down to doing some work in LISP and wrap my head around some real language subversion.)

Anyway, while this is neither quite Native XML Scripting nor XML as a native language construct, it's getting there.

Archived Comments

Ryan Tomayko
2004-09-01T17:29:27

Funny that. I also have one that has survived a couple of failed apps. I have a hard time dropping it to be honest and just keep lugging it around to each new project. http://naeblis.cx/cvs/percolator/xb/lib/xpdm.py?rev=HEAD&content-type=text/vnd.viewcvs-markup It has some pretty big issues. Among other things, creating nodes with namespace support is a little.. ermmm.. not there. But it does a lot of things well like garbage collecting xmlDoc instances (freeDoc), copying nodesets between documents, encoding things when they need to be, etc. Anyway, I wonder if maybe we all might benefit by teaming up on this and try to define what a complete xpathish wrapper atop libxml2 should look like. And really, why limit it to libxml2? I'm of the opinion that the value here is an interface that embraces xpath. The fact that it's running on top of the blazingly fast libxml2 is nice but coding against the XMLTRAMP like interface is the value for me. So let me see if I can get some time together to whip up a quick comparison of the three implementations. I'll shoot that over to you and Kimbro and we can go from there. If these seem to work best as backyard APIs we like to keep close to us, we'll drop it. However, I think there's a good chance that we can all benefit by combining our efforts.

Making progress on dbagg3

2004-08-31T01:37:42+00:00

Work has been insanely busy lately, but I have made some more progress with dbagg3. The code is all in CVS, so feel free to take a gander-- I don't have a ton of time for a proper write up, but I do want to spew a little bit.

As per my previous musings on XML in a SQL database, I revamped the database. Now things are sliced up by feed and entry tables, rows in each containing a few metadata columns and then one big column for an XML dump. This lets me index on date and parent feed and such, meanwhile punting on the issue of dicing things like authors or content up further. And, as extension elements start to show up, this handling is dumb enough to simply store things it doesn't know about without mangling them. This is a very good thing and one of my big goals for this beast.

The other thing that I'm getting excited about is the REST API built atop the Atom store. Rather than spend time on proper documentation, here's a quick dump from the appropriate module:

URL: GET /feeds/
URL: GET /feeds/{id}.xml
URL: GET /feeds/{id}/{yyyy}/{mm}/{dd}/{hstart}-{hend}.xml
URL: GET /feeds/{id}/{yyyy}/{mm}/{dd}/{hh}.xml
URL: GET /feeds/{id}/{yyyy}/{mm}/{dd}.xml
URL: GET /feeds/{id}/{yyyy}/{mm}.xml
URL: GET /feeds/{id}/now-{nowoff}.xml
URL: GET /feeds/{fid}/entries/{eid}.xml
URL: GET /users/
URL: GET /users/{uname}.xml
URL: POST /users/
URL: DELETE /users/{uname}.xml
URL: PUT /users/{uname}.xml
URL: GET /users/{uname}/prefs.xml
URL: GET /users/{uname}/prefs/
URL: POST /users/{uname}/prefs/{pname}.{type}
URL: PUT /users/{uname}/prefs/{pname}.{type}
URL: GET /users/{uname}/prefs/{pname}.{type}
URL: DELETE /users/{uname}/prefs/{pname}.{type}
URL: GET /users/{uname}/subscriptions.{type}
URL: GET /users/{uname}/subscriptions/
URL: POST /users/{uname}/subscriptions/
URL: DELETE /users/{uname}/subscriptions/{id}.xml
URL: GET /users/{uname}/subscriptions/{sid}/{yyyy}/{mm}/{dd}/{hstart}-{hend}.xml
URL: GET /users/{uname}/subscriptions/{sid}/{yyyy}/{mm}/{dd}/{hh}.xml
URL: GET /users/{uname}/subscriptions/{sid}/{yyyy}/{mm}/{dd}.xml
URL: GET /users/{uname}/subscriptions/{sid}/{yyyy}/{mm}.xml
URL: GET /users/{uname}/subscriptions/{sid}/now-{hours}.xml
URL: GET /users/{uname}/subscriptions/{sid}/now.xml
URL: GET /users/{uname}/subscriptions/{yyyy}/{mm}/{dd}/{hstart}-{hend}.xml
URL: GET /users/{uname}/subscriptions/{yyyy}/{mm}/{dd}/{hh}.xml
URL: GET /users/{uname}/subscriptions/{yyyy}/{mm}/{dd}.xml
URL: GET /users/{uname}/subscriptions/{yyyy}/{mm}.xml
URL: GET /users/{uname}/subscriptions/now-{hours}.xml
URL: GET /users/{uname}/subscriptions/now.xml
URL: GET /users/{uname}/subscriptions/{sid}/entries/{eid}.xml

Hopefully, the structure of these URL patterns make a little bit of sense. The too-clever thing about these is that they're both documentation in the module's docstrings, and parsed out to register methods with automagically-generated regexes applied to incoming URL requests. (I may eventually realize just how stupid an idea this is, but not yet.)

This list is nowhere near complete or final or even all that well thought out yet. But, it seems to be working out pretty well so far, and it's so easy to tinker with the API to sketch out ideas in working code. Eating my own dogfood, my first browser window of the day tends to open on this URL:

http://localhost/~deusx/dbagg3.5/api/users/default/subscriptions/
now-12.xml?xsl=xsl/full.xsl&#38;content-type=text/html

This grabs the last 12 hours' worth of items from default's subscriptions, passing them through the XSL at xsl/full.xsl on the way to my browser with a content type of text/html. This tends to produce about 1000-1500 entries in about 15 seconds on my PowerBook, which is better than I'd expected.

Pretty soon, I'll be implementing the ability to post metadata onto feed entries under subscriptions. Then, I can mark items as seen, attach categories, tags, and notes. From there, I can exclude seen items from queries, produce new aggregate feeds based on my tagging or notes, among a few other ideas I've got stewing.

A little more work, and I think I'll be able to throw together the beginnings of a Bloglines-style three-pane browser interface, as well as improving the functionality of my own outliner-style display with XmlHTTPRequest-based calls to the API to enable refresh-free interaction. From there, I have some ideas for desktop apps and maybe even some tinkering in Flash. (Wow... has it really been over a year since I was writing about Flash & REST?)

And then, I want to implement the Atom API and allow users to create feeds to which they can post their own items and share read-only with others (or share writing with a group). From there, this thing can turn into a read/write Atom storage tank, serving both as an aggregator and a blog publishing engine, given the appropriate XSL work.

Lots of ideas stewing. Now I just have to get the time and possibly a new web server, since I'd like to eventually open up an installation of this to fellow tinkerers, but this poor little box can barely take what it's tasked with at present...

Oh yeah, and one other thing: I've been thinking about names better than dbagg3. The one that's sticking around in my head so far is feedReactor. What do you think?

shortname=dbagg3_makingprogress

Archived Comments

Alan
2004-09-01T03:39:10

I prefer the dbagg title to feed_anything_. There's feedster, feedburner, feedreader and I'm sure there will be a whole slew more. This needs an original name. It seems similiar to Urchin, which you earlier posted a link to, how about Starfish or maybe Seahorse. ;)

mysql and XML output

2004-08-23T05:09:51+00:00

So... How many of you have ever used mysql -X?

I just discovered it today, while screwing around with dumping database queries into Atom. While I'm not entirely sure it's what I need to use, this is pretty nifty:

$ mysql -Xp -udbagg3 dbagg3 -e '
> select id, title, modified 
> from feed
> order by modified 
> limit 4' 
Enter password:

<?xml version="1.0"?>

<resultset statement="select id, title, modified 
        from feed order by modified limit 4">
  <row>
    <id>527</id>
    <title>Channel Dean</title>
    <modified>2004-03-04 15:56:54</modified>
  </row>

  <row>
    <id>31</id>
    <title>chocolate and vodka</title>
    <modified>2004-07-21 21:30:08</modified>
  </row>

  <row>
    <id>183</id>
    <title>floating atoll</title>
    <modified>2004-07-31 14:09:27</modified>
  </row>

  <row>
    <id>24</id>
    <title>What's Your Brand Mantra?</title>
    <modified>2004-08-02 03:15:03</modified>
  </row>
</resultset>

Now, while I don't think that using this for dbagg3 is all that great an idea, it's something I need to remember for future shell and XSLT hacks...

Archived Comments

iamsure
2004-08-23T03:47:12

Is there a way to do so without the commandline option, ie, via a select call, etc?
bosshoff
2005-04-20T12:22:15

There is a better way, using mysqldump, explained here: http://insight.zdnet.co.uk/software/developer/0,39020469,2112200,00.htm
Rededog027
2007-07-19T23:51:15

That is awesome I haven't used it before but it is extreamly useful for creating changelogs from svn and bugzilla :)

Wish-of-the-Month Club, Part 3 of 3

2004-07-06T21:05:45+00:00

This is the exciting conclusion of the Wish-of-the-Month Club. Before continuing on, you may want to catch up with parts one and two.

Presenting the Results

Some ready-made files are available for this section: * wishes-ex5.xsl: The fifth iteration of the stylesheet in development. * wishes.html: Sample output in HTML

We've finally gotten together all the bits of information we need--wishlists have been queried; random items have been selected; and a shopping cart has been prepared. Now we just have to present the selections and a link to check out with the shopping cart.

First, locate the following line toward the end of the stylesheet as we left it in the last section:

    <xsl:copy-of select="$shopping_cart" />

Delete this, and let's replace it by building some HTML:

    <xsl:variable name="shopping_cart_purchase_url" 
                  select="exsl:node-set($shopping_cart)//PurchaseUrl" />

    <html xmlns="http://www.w3.org/1999/xhtml">
      <head><title>Wishlist Shopping Cart</title>\</head>
      <body>
        <p class="title">
          Here are your wishlist items
          <a href="{$shopping_cart_purchase_url}">
            <img src="http://g-images.amazon.com/images/G/01/detail/shoppingcart-header-02.gif" />
          </a> 
          items:
        </p>

We're using the exsl:note-set function again to access the contents of $shopping_cart with an XPath expression. We pluck out the value of the PurchaseUrl in the shopping cart and place it in the variable shopping_cart_purchase_url. Then, after a bit of HTML preamble, we borrow a shopping cart icon from Amazon itself to construct a link to which we can browse later to purchase the selected items. This HTML is very simple so far; it's likely too simple, so eventually you may like to toss some CSS in here to improve the looks of things. But, I'll leave that as an exercise for the reader.

Next, let's build a display of the items selected by iterating first through the wishlists:

        <xsl:for-each select="exsl:node-set($random_products)/wishes:wishitem">
          <div class="Detail">

            <p class="wishlistLabel">
              <xsl:value-of select="wishes:wishlist/@label" />
            </p>

This begins a block for each wishlist, starting off with a paragraph containing the label we gave each wishlist. Next, let's include a few details about the product chosen. Again, all of the bits of data included for each product are described in the AWS documentation in the Overview under Amazon Web Services Data Model. Checking that out, we can see that the data includes a URL to images of several sizes representing the product. Let's include the medium-sized image as a link to the product's detail page:

            <p class="Product">
              <a href="{Details/@url}">
                <img src="{Details/ImageUrlMedium}" />
              </a>
              <br />

We can also include the product's name as a link:

              <span class="ProductName">
                <a href="{Details/@url}">
                  <xsl:value-of select="Details/ProductName" />
                </a>
              </span>
              <br />

And, it would be nice to provide a listing of people involved in creating the product (ie. the artists and/or authors):

          <xsl:for-each select="./Details/Artists/Artist | 
                                ./Details/Authors/Author">
            <span class="Author">by <xsl:value-of select="." /></span><br />
          </xsl:for-each>

Note that here, the XPath selecting the data is just a bit more involved, since this information can be found in both Artist and Author elements. In another case, we might care to make a distinction, but it really isn't all that important for this project. The data model also provides an indication of from which catalog this product came, as well as its date of release. Let's include that for good measure:

          (
          <xsl:value-of select="Details/Catalog" /> -
          <span class="ReleaseDate">
            <xsl:value-of select="Details/ReleaseDate" />
          </span>
          )
          <br />
        </p>

Another thing that would be nice to know is how much this thing costs--we've got this information provided in the XML data as well, so let's include it:

        <p>
          <span class="PriceLabel">List Price:</span> 
          <span class="ListPrice">
            <xsl:value-of select="Details/ListPrice" />
          </span>
          <br />

          <span class="PriceLabel">Our Price:</span>
          <span class="OurPrice">
            <xsl:value-of select="Details/OurPrice" />
          </span>
          <br />

          <span class="PriceLabel">Used Price:</span> 
          <span class="UsedPrice">
            <xsl:value-of select="Details/UsedPrice" />
          </span>
          <br />
        </p>

Something to note about these prices, too, is that although the used price is listed, the shopping cart will contain new items from Amazon's shelves. You might want to compare these prices though, and make a change to the shopping cart when you get there, if a used item is acceptable. (Another good reason for manual intervention in our Wish-of-the-Month club.)

Oh yeah, and we should include one other bit of information:

        <p>(<xsl:value-of select="Details/Availability" />)</p>

This tells us whether or not this item can actually be bought, at present. Although we used this data earlier to try to filter out unavailable items, we should still display this information just in case we missed something.

Finally, let's clean up and finish the HTML:

      </div>
    </xsl:for-each>

  </body>
</html>

Running this stylesheet (wishes-ex5.xsl) should give you a page that looks something like this in a browser:

Scheduling Monthly Emails

Some ready-made files are available for this section: * wishes-ex6.xsl: The sixth (and final) iteration of the stylesheet in development.

That HTML we're producing is fine, but what we really want to do is get it delivered to us. We could set up a scheduled run that would periodically generate a page for us to visit, but the whole point of this is laziness. How about firing off an email with this content? There are two things to help us with this: RFC 1521 shows us how to construct email messages with a variety of content types; and sendmail will let us send these messages out. And then, with the help of cron, we can fire up this process every month.

Along with producing XML, XSLT can also construct plain text output--which is just what we need to create MIME email messages. RFC 1521 doesn't make for the most thrilling reading, but there are a few articles to be found that summarize things (such as this article and this article). To make a long story short, a basic shell for an email message using MIME to include an HTML part and a plain text part looks something like this:

To: someone@example.org
Subject: Some useful email subject
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="theBoundaryString"

--theBoundaryString
Content-Type: text/plain

Some plain text representation goes here...

--theBoundaryString
Content-Type: text/html
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Content-Base: "http://www.decafbad.com/"

<html xmlns="http://www.w3.org/1999/xhtml">
    <p>Some HTML representation goes here...</p>
</html>

--theBoundaryString--

I've snuck in the idea of providing both an HTML version (which we've already done) and a new plain text version. Depending on your email program and your preferences, one type might be more useful than the other. In any case, it's not all that hard to offer both here. To start sending these email messages, though, we'll need an email address. So, add that as an element in wishes.xml:

<wishes xmlns="http://www.decafbad.com/2004/05/wishes">
  <email>deus_x@pobox.com</email>
  <maxprice>15.00</maxprice>
  <associate>0xdecafbad-20</associate>
  <devtoken>D8HVH869XA0NP</devtoken>
  <wishlists>
    <wishlist label="Me">1QWYI6P2JF3Q5</wishlist>
    <wishlist label="The Girl">35OIOYWQ9XQAE</wishlist>
  </wishlists>
</wishes>

Let's extract this data into a global variable near the start of the stylesheet:

  <xsl:variable name="email_to"  select="/wishes:wishes/wishes:email" />

Start editing the final template of the stylesheet, inserting before the start of HTML content:

    <!-- Eat all the line breaks generated so far -->
    <xsl:text>To: </xsl:text><xsl:value-of select="$email_to" />   
Subject: 0xDECAFBAD's Amazon Wish-of-the-Month Club
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="theBoundaryString"

This is the header for the email. Up until now, we've been generating XML with the stylesheet and haven't cared very much about any extra whitespace or line breaks which might sneak into the output. However, in an email header, whitespace is important since a blank line is what's used to separate the headers from the body of the email message. So, any stray blank lines will cause what we might have meant to be headers to be interpreted as part of the message instead. Producing the first header in the email with xsl:text tags causes the XSL processor to throw away any leading whitespace which would have appeared before the first header.

Other than this little twist, the email header looks pretty much like the shell. We fill in the To address from the global variable $email_to and define a Subject line. The MIME-Version and Content-Type headers are what enable us to include both text and HTML versions in one email.

Now we can start into one of the parts:

--theBoundaryString
Content-Type: text/plain

This begins the plain text section of the email, using the boundary string as defined in the headers to delinieate the section's beginning. The section can also have its own set of headers, of which we use only one: Content-Type. Moving along, let's work on the text content itself.

Here are your wishlist items:

<xsl:value-of select="$shopping_cart_purchase_url" /><xsl:text>
</xsl:text>

No shopping cart image here, but this includes the human-viewable URL which leads to a shopping cart on Amazon.com. The usage of xsl:text here forces a line break where there otherwise wouldn't have been one with the usage of xsl:value-of. Now, let's iterate through each of the wishlists and list out the product details:

<xsl:for-each select="exsl:node-set($random_products)/wishes:wishitem">
---------------------------------------------------------------------------
<xsl:value-of select="wishes:wishlist/@label" 
       disable-output-escaping="yes" />
---------------------------------------------------------------------------

<xsl:value-of select="Details/ProductName" 
       disable-output-escaping="yes" />

<xsl:for-each select="./Details/Artists/Artist | 
                      ./Details/Authors/Author">
by <xsl:value-of select="."  
   disable-output-escaping="yes"/>
</xsl:for-each>

Catalog:      <xsl:value-of select="Details/Catalog" 
   disable-output-escaping="yes" />
Released:     <xsl:value-of select="Details/ReleaseDate" 
   disable-output-escaping="yes" />

List Price:   <xsl:value-of select="Details/ListPrice"  
     disable-output-escaping="yes"/> 
Our  Price:   <xsl:value-of select="Details/UsedPrice"  
     disable-output-escaping="yes"/> 
Used Price:   <xsl:value-of select="Details/OurPrice"  
     disable-output-escaping="yes"/> 

Availability: <xsl:value-of select="Details/Availability"  
       disable-output-escaping="yes"/>
<xsl:text>

</xsl:text>
<xsl:value-of select="Details/@url"  
       disable-output-escaping="yes"/>
<xsl:text>
</xsl:text>

</xsl:for-each>

Most everything in this stretch should look very similar to the HTML version we just finished. The biggest difference is that every bit of information pulled in using xsl:value-of is done using the disable-output-escaping option. When this is yes, things like ampersands are no longer escaped for valid XML output. Since this bit of the email is plain text, we don't want to see & in album titles, so this will cause ampersands to appear unmolested.

That's the plain text version finished. Now let's create the HTML version:

--theBoundaryString
Content-Type: text/html
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Content-Base: "http://www.decafbad.com/2004/05/wishes"

The boundary string appears again, signifying the end of the plain text section and the start of the HTML section. Headers appear here which specify that what follows is HTML; that it's encoded in 7-bit characters; that it should be included in the message display itself (rather than presented as an attachment to be saved); and that all relative URLs which might appear in the HTML should be treated as having a base URL as specified. This last part allows HTML in email to refer to images and other pages on another site without making all the URLs absolute.

We don't need to make any modifications to the HTML as we built it in the last iteration of the stylesheet, so we can just include it unchanged:

<html xmlns="http://www.w3.org/1999/xhtml">
...
</html>

--theBoundaryString--

This final appearance of the boundary string is bracketed on both sides by dashes, which indicates the end of the final section of the document. We should be ready to try this in combination with sendmail in a shell:

$ xsltproc wishes-ex6.xsl wishes.xml | sendmail -it

If everything has worked correctly, there should be an email arriving in your mailbox sometime soon. (Or in my inbox, if you followed the directions literally and didn't supply your own email address.) The options supplied to sendmail are fairly basic:

-i causes lines consisting solely of . not to be treated as an end-of-input signal.
-t causes sendmail to look in the message headers (ie. To:) for a list of recipients.

If you don't happen to have have sendmail available, you might want to look into what local mail programs you have available which can accept the output from the stylesheet.

Once you have this working, the final task is to schedule its monthly execution with your local cron installation. If you haven't played with cron before, there are many resources and tutorials available (here's one and here's another). You should add something like the following to your user account's crontab:

0 0 * 1 *  (cd /your/working/path; xsltproc wishes.xsl wishes.xml | sendmail -it)

The "0 0 * 1 *" indicates to cron that this set of commands should be run at midnight on the first of every months. Note also that /your/working/path should be replaced by the path to where you've been working during this project. And finally, I've renamed the final iteration of the stylesheet file to simply wishes.xsl.

Conclusion

So that's it--we have an XSL stylesheet which queries Amazon Web Services for products contained in multiple wishlists; selects a random item from each; prepares a shopping cart containing those items; and finally generates an email message containing both plain text and HTML presentations of the shopping cart and selected items.

Though this implementation serves the purpose I wrote about at the start of this article, there are definitely many areas where this can be improved upon or expanded:

Many people think Amazon is an evil company for their use of patents. I can't say that I'm entirely happy with them for this myself, but their AWS offering is just too nice to resist tinkering with. It might be interesting to investigate other retailers' wishlist offerings, where they exist, and to see how this idea might be made to work with other (or even multiple) retailers. Even better, come up with your own wishlist system, and a cross-retailer shopping cart.
I chose XSLT as the implementation technology because I thought it would be more natural to deal with Amazon's XML this way. There are, admittedly, a few awkward parts in the resulting stylesheet however. Sometimes it's good to see a project like this through, just to get a sense for where things do go awkward with a technology or my understanding of it. It could be interesting to transliterate this into a scripting language like Python or Perl, perhaps using the libxml bindings to do so.
The error and failure handling in this implementation are all but non-existent. Should anything unexpected happen while dealing with Amazon Web Services, the results aren't likely to be very pretty. You may want to consider improving in this area. One instance I identified was to report when the sanity limit was hit in looping through wishlist pages, versus an actual end of pages.
If you play around with making more wishlist queries using the techniques here, you might want to consider caching the full set of data pulled in by the multiple-page calls to AWS, in order to prevent hammering Amazon's servers with repeated requests for the same data, likely unchanged.
I still don't know why exsl:random doesn't work for me. Although I thought using a web service for random numbers was intereting, it would be very nice if I didn't have to use it.
The HTML presentation could certainly use some good CSS to make it more attractive.

Feel free to send me any suggestions, criticisms, or complaints related to this article!

shortname=wishofthemonthclub3

A mini-scraper for the playlist at radioio Rock

2004-06-28T23:17:36+00:00

Lately, my iTunes has been playing radioio Rock almost exclusively lately, but one thing that peeves me is that I don't seem to see the current song while the stream's playing. Instead, the radioio site offers a pop-up window that displays the last few songs in the playlist. However, I'm usually somewhere off in another window or a shell and don't really feel like popping over to a browser and navigating to the playlist just to see what this song is.

So, I wrote myself a little mini-scraper script:

#!/bin/sh

curl -s 'http://player.radioio.com/player.php?b=614&#38;stream=radioioRock' | \
    tidy -asxml --wrap 300 -q -f /dev/null | \
    xml sel -t -m "//*[@class='leadrock']" -v '.' -n \
        -o '    [http://www.radioio.com' -v '../@href' -o ']' -n

The output looks something like this:

[06/29 11:01:08] deusx@Caffeina2:~ % radioio

Vast - I Need To Say Goodbye
    [http://www.radioio.com...]
Cure - The End Of The World
    [http://www.radioio.com...]
Seachange - Avs Co 10
    [http://www.radioio.com...]
Pixies - Bam Thwok
    [http://www.radioio.com...]
Death Cab For Cutie - The New Year
    [http://www.radioio.com...]
Lovethugs - Drawing The Curtains
    [http://www.radioio.com...]

Oh yeah, and to run this script, you will need these tools:

Personally, I like the included URLs (which I edited here for length) since they launch a search for CDs by the artist. However, you can cut the output down to just the artist/title by removing the final line of the script and the backslash at the end of the line before.

If you like a different radioio station, say radioio Eclectic, you can change stream=radioioRock to stream=radioioEclectic in the URL above and change class='leadrock' to class='leadeclectic'. I could have parameterized these, but I'm lazy, and that was the whole point!

ShareAndEnjoy!

shortname=radioiorock_scraper

Archived Comments

Mike Carter
2004-07-01T07:39:45

I was listening to some streams today for the first time in a while and I was thinking how annoying it was not to know the track names. Thanks for posting this, it may come in handy!
Gnomon
2004-07-01T18:31:58

How is this the first time I've ever found out about XML Starlet? It's magnificent! The application is cool, too, but /man/ - I'm going to have fun with XML Starlet. Thanks so very much for the pointer, Les!
Jesse
2005-01-23T11:50:41

I was searching around with blowsearch.com today (which btw blows heheh) and found this entry. This earl might be of interest to ya: http://69.28.133.51:1040/7.html The db for searchplay and our players is only updated every 30 seconds, 7.html is near reatime right from our playback software. :) We used to have title streaming on our mp3 streams but are in transition of server and encoding software atm while we prepare to roll out our own software, and will be adding new formats such as heaac. :) Too much to talk about, we have a LOT of stuff planned for 2005-2006 that we've been working on (and patent pending) for years now. Definately going to be exciting time for net radio. Thanks for your interest Leslie. jesse chief tinkerer of toys, radioio

Wish-of-the-Month Club, Part 2 of 3

2004-06-28T01:44:51+00:00

Here's the next installment of the Wish-of-the-Month Club. You can revisit the first part, too, if you've missed it. I'd meant to post it within a week of the first part, so apologies all around to anyone who has been tapping a foot waiting for it. Enjoy!

Paging Through Wishes

Some ready-made files are available for this section: * wishes-ex2.xsl: The second iteration of the stylesheet in development.

Now we've got a way to make queries against Amazon Web Services, not entirely unlike what you might be used to if you tinker with MySQL databases on a regular basis. At this point, though, we still have a bit of refining to make to this query. If you take a look at the data produced by the query in its current state, and compare that to what you see on wishlists in your browser, you should notice some things missing.

If you look at my wishlist, you'll notice that items span several pages when visited by browser. As it turns out, AWS queries work in a similar fashion--each query returns only a limited number of items (about 10), and an additional parameter supplied to further queries is required to step through further pages of results. So, using what we've built so far will only get us to the first page of wishlist items; to get all of the items, we'll need a way to step through all of the pages.

In playing with this, I experienced a bit of hairpulling frustration: The AWS documentation, under "Generating Additional Product Results", claims that XML returned by the service will supply a count of the total pages available for a given query. And although I see this element present in other types of searches, the TotalPages element is absent when querying on wishlists. This may be a bug, or it may be an undocumented change in the service--either way, it was a surprise and leaves me with no official way to know how many pages I need to ask for in order to have a complete set of data.

With some further tinkering, though, I figured out a workaround: If a query is made for a page number beyond the final page, the XML returned will be a duplicate of the final page. Once I see a duplicate item appear, I know it's time to stop paging through results. This is completely undocumented behavior, and could break at any time (ie. if Amazon decided to start issuing an error for a page index out of bounds), but it'll work for now.

This calls for reworking the processWishlist template. For a given wishlist, it will need to iterate through a sequence of page numbers, requesting XML from AWS for each, stopping when the first duplicate page is found. Since XSLT is heavily steeped in functional programming concepts, this sort of iteration in XSLT is best done with recursion:

  <xsl:template name="processWishlist">

    <xsl:param name="wishlist" />              <!-- Wishlist ID -->
    <xsl:param name="max"   select="50" />     <!-- Arbitrary upper loop limit -->
    <xsl:param name="curr_page" select="1" />  <!-- Curr page # -->
    <xsl:param name="prev_first_asin" />       <!-- Keeping track of repeats -->

The first modification to this template is the addition of three parameters:

max provides an arbitrary upper limit to the number of pages through which this template will iterate.
curr_page contains the number of the page to be requested in this iteration.
prev_first_asin will contain the ASIN number of the first item from the previous iteration's page of results.

Next, we modify the URL used to query for wishlist data:

    <!-- Fetch the wishlist products -->
    <xsl:variable name="details" select="document(concat(
                  'http://xml.amazon.com/onca/xml3?',
                  't=',$associate,'&amp;',
                  'dev-t=',$devtoken,'&amp;',
                  'WishlistSearch=',$wishlist,'&amp;',
                  'type=lite&amp;f=xml&amp;',
                  'page=',$curr_page))//Details" />

The only addition here beyond the previous version is the page parameter in the URL. Not much mystery here--this parameter specifies which page of results we want. Now, let's build the loop:

    <!-- Snag the first item Asin -->
    <xsl:variable name="curr_first_asin" select="$details/Asin/text()" />

    <!-- If we haven't exceeded the loop limit, and this first Asin isn't -->
    <!-- a repeat of the previous loop (indicating we've run out of new   -->
    <!-- pages), then go ahead...                                         -->
    <xsl:if test="(($curr_page+1) &lt; $max) and
                  (string-length($curr_first_asin) &gt; 0) and
                  not($curr_first_asin = $prev_first_asin)">

We capture the ASIN of the first item in this page of results and check to see if we should continue. This if conditional first ensures that we're not past the sanity guard for loop iterations, makes sure that we actually got a non-empty current first ASIN, then checks our current first product's ASIN against what was passed in as the previous iteration's first product's ASIN. If this was the first time through the loop, this value should be empty and therefore wouldn't match the current ASIN. But, if we've gone past the end of results, the previous and current ASIN values should match, and the conditional will fail.

Moving along into the body of the conditional, we copy in wishlist products filtered on a price maximum, just as before:

      <!-- Copy products, filtering on a maximum price -->
      <xsl:copy-of select="$details/OurPrice[number(substring(
                   text(),2)) &lt; $maxprice]/.." />

Having done that, we move onto the recursive end of this template:

      <!-- Loop by recursion to get the next page -->
      <xsl:call-template name="processWishlist">
        <xsl:with-param name="wishlist"        select="$wishlist" />
        <xsl:with-param name="max"             select="$max" />
        <xsl:with-param name="curr_page"       select="$curr_page + 1" />
        <xsl:with-param name="prev_first_asin" select="$curr_first_asin" />
      </xsl:call-template>

    </xsl:if>    
  </xsl:template>

Here, the template makes a recursive call back to itself, passing through the wishlist ID and the maximum iteration count. Since variables in XSLT are immutable, meaning that their values can't be changed once they've been set, we can't increment $curr_page in-place like a loop counter in other languages--so, the current page count value is incremented and passed to the recursive call as a parameter. Finally, the current first item's ASIN is passed along, to become the previous ASIN for the next iteration.

Note that when the conditional fails--that is, if the loop limit is passed or a duplicate page is detected--the loop ends. In other words, nothing further happens and execution pops back up out of all the levels of recursion and the top-level template ends.

I wrote "when the conditional fails". This is a key point: for the loop to eventually end, this conditional must fail (or be made to fail) at some point, else this loop will happily progress through page requests forever. This is the reason for the $max parameter limiting the number of iterations, in case something goes haywire--like, oh say, a failure of our duplicate-page detection hack as a loop ending condition. A useful exercise for the reader might be to add some additional diagnostic code to report that the limit was hit versus a natural end to results.

Random Numbers

Some ready-made files are available for this section: * wishes-ex3.xsl: The third iteration of the stylesheet in development. * random-xml: A Perl CGI script used as a web service to generate random numbers.

Armed with a template that will query against the full set of items in a wishlist, we're ready to look into making a random selection from a list of products.

But first, we need to pick a random number. Unfortunately, there doesn't appear to be any random() function in the XPath or XSLT standards. There is a math:random() from EXSLT implemented in libxslt, but I seem to be having a bit of a problem getting it to produce anything other than the same sequence of numbers. I suspect there's a problem in seeding the random number generator, but I've yet to work out how to fix it. (Suggestions welcome.)

So, I cheated and made another workaround with a CGI script on my web server that generates random numbers in a simple XML document. Currently, it's hosted here:

http://www.decafbad.com/2004/05/random-xml

And this is what the script looks like:

#!/usr/bin/perl

use strict;
use CGI;

my $q = new CGI();

my $min = $q->param('min') or 0;
my $max = $q->param('max') or 1;
my $int = $q->param('int');

my $num = $min + ( rand() * ($max - $min));
if ($int) { $num = int($num); }

print $q->header('text/xml');
print "<rand>$num</rand>\n";

This is a very simple CGI. It accepts the parameters max, min, and int. The values of these parameters determine the maximum and minimum value for the random number returned, and whether or not it should be an integer. For example, the following URL should return an integer between 10 and 20:

http://www.decafbad.com/2004/05/random-xml?
int=1&#38;min=10&#38;max=20

Using this as a web service in the stylesheet with the document() function, we can get a random number. If you've got web space where you can host CGI scripts, I suggest you host a copy of this script yourself, since I can't guarantee how long mine will stick around. But, for as long at works, feel free to use the service from my server.

Moving along, let's add a new named template to the stylesheet, called randomWishlistProduct:

  <xsl:template name="randomWishlistProduct">

    <xsl:param name="wishlist" /> <!-- Wishlist ID -->

    <!-- Gather all the products for the current wishlist -->
    <xsl:variable name="products">
      <xsl:call-template name="processWishlist">
        <xsl:with-param name="wishlist" select="$wishlist" />
      </xsl:call-template>
    </xsl:variable>

Just like the processWishlist template, we start by defining the parameter wishlist to accept a wishlist ID. Using this ID, we call the processWishlist template itself and store the complete list of products queried from the wishlist into the variable $products.

    <!-- Count the products in the wishlist -->
    <xsl:variable name="max_products"
                  select="count(exsl:node-set($products)/Details)" />

This next step counts the number of products found in the wishlist. The one tricky bit here is the use of the EXSLT function exsl:node-set(): The $products variable contains what's called a result tree fragment, which is a kind of cross between XML data nodes and a plain old string. This type of data does not normally allow the full set of XPath operators to be used on it, so first we need to use exsl:node-set() to turn it into a full-fledged node set. Then we can look up the Details element nodes and count them.

    <!-- Conjure up a random index within the list of products -->
    <xsl:variable name="rand_product_num"
                  select="document(concat(
                  'http://www.decafbad.com/2004/05/random-xml?',
                  'int=1&amp;',
                  'min=1&amp;',
                  'max=',$max_products))/rand" />

Here is where the random number service comes in handy. The concat() function is used to build the URL to the service, with parameters specifying that the number should be an integer, and should fall between 1 and the number of products in the wishlist. The document() function grabs the XML document from the service, and the value is extracted from the single element the document contains.

There is an alternative to this last bit, should you happen to have a properly working math:random() function in your XSLT processor:

    <xsl:variable name="rand_product_num" select="round( math:random() *
                  $max_products ) + 1" />

If you can use this instead, you'll have no need for the random number web service. This version is obviously more concise, and doesn't require another trip out to a web service. You might want to try it--but if you find that you keep getting the same wishlist items selected, then you've run into the problem I found with the random number generator.

Now, let's wrap this template up by selecting an item:

    <!-- Copy the product as indexed by the random number -->
    <xsl:copy-of select="exsl:node-set($products)/Details[
                 position()=$rand_product_num]" />

  </xsl:template>

Again, we need to use the exsl:node-set() function to turn the result tree fragment in the $products variable into a node set, from which we select and copy the Details element whose position in the data is indexed by the random number we just selected. Just one last tweak needed to wrap up this iteration of our stylesheet. We need to swap out the call to the processWishlist function at the end and replace it with a call to randomWishlistProduct:

  <xsl:template match="/wishes:wishes">

    <xsl:for-each select="//wishes:wishlist">
      <wishes:wishitem>
        <xsl:copy-of select="." />
        <xsl:call-template name="randomWishlistProduct">
          <xsl:with-param name="wishlist" select="." />
        </xsl:call-template>
      </wishes:wishitem>
    </xsl:for-each>

  </xsl:template>

After these changes, you should be able to run the stylesheet ([wishes-ex3.xsl][wishes_ex3]) and get something like the following:

<wishes:wishitem xmlns:wishes="http://www.decafbad.com/2004/05/wishes">
    <wishes:wishlist label="The Girl">35OIOYWQ9XQAE</wishes:wishlist>
    <Details ...>...</Details>
</wishes:wishitem>
<wishes:wishitem xmlns:wishes="http://www.decafbad.com/2004/05/wishes">
    <wishes:wishlist label="Me">1QWYI6P2JF3Q5</wishes:wishlist>
    <Details ...>...</Details>
</wishes:wishitem>

This is similar to the output of the previous iteration of the stylesheet, but this time there's only one product selected at random for each wishlist.

Shopping Carts

Some ready-made files are available for this section: * wishes-ex4.xsl: The fourth iteration of the stylesheet in development.

By this point, we've been able to query and filter products in Amazon wishlists, and we've selected an item at random from each wishlist we've queried. Now, let's enable some purchases.

The AWS provides for Remote Shopping Cart functionality, whereby items can be added to an Amazon.com shopping cart programmatically. This is about as close as we can get to automating the purchase of items selected from the wishlists--there is no API functionality for actually completing the ordering of items. If you really think about it, this really is a good thing and should demand human intervention; we certainly wouldn't want this script going crazy and accidentally buying up everything on a wishlist.

Documentation for the AWS Remote Shopping Cart explains that a shopping cart can be created and items added with a URL like the following:

http://xml.amazon.com/onca/xml3?
ShoppingCart=add&#38;
f=xml&#38;
dev-t=[Developer Token goes here]&#38;
t=[Associates ID goes here]&#38;
Asin.[ASIN goes here]=[quantity goes here]&#38;
sims=true

Part of this should look familiar, so we already know what to do with the developer token and the associates ID. The last part, specifying product ASIN and quantity, can be filled out with information contained in the product records selected at random from the wishlists.

So, let's start by revising the template at the end of the stylesheet:

<xsl:template match="/wishes:wishes">

    <xsl:variable name="random_products">      
      <xsl:for-each select="//wishes:wishlist">
        <wishes:wishitem>
          <xsl:copy-of select="." />
          <xsl:call-template name="randomWishlistProduct">
            <xsl:with-param name="wishlist" select="." />
          </xsl:call-template>
        </wishes:wishitem>
      </xsl:for-each>
    </xsl:variable>

Here, we've taken what was the output of the previous iteration of the stylesheet and stuffed it into the variable $random_products. Next, let's fill in the blanks and build a Remote Shopping Cart URL:

    <xsl:variable name="shopping_cart_create_url">
      <!-- Standard AWS URL -->
      <xsl:text>http://xml.amazon.com/onca/xml3?</xsl:text>

      <!-- Add in the selected items -->
      <xsl:for-each select="exsl:node-set($random_products)
                            /wishes:wishitem/Details">
        <xsl:text>Asin.</xsl:text><xsl:value-of select="Asin" />
        <xsl:text>=1&amp;</xsl:text>
      </xsl:for-each>

      <!-- Wrap up with the shopping cart function and required tokens -->
      <xsl:text>ShoppingCart=add&amp;</xsl:text>
      <xsl:text>f=xml&amp;</xsl:text>
      <xsl:text>dev-t=</xsl:text><xsl:value-of select="$devtoken" />
      <xsl:text>&amp;</xsl:text>
      <xsl:text>t=</xsl:text><xsl:value-of select="$associate" />
    </xsl:variable>

Since simple XPath doesn't allow for the looping needed for multiple items, we can't just concatenate this URL together in a select expression like we did with the wishlist item query. So, we use xslt:foreach to build this with blocks of text using the xsl:text element. We iterate though the random products chosen from wishlists and add an ASIN for each to the URL with a quantity of 1. Then, we use the $devtoken and $associate variables to fill in their respective spots.

Note that this could have been written without using the xsl:text elements like so:

    <xsl:variable name="shopping_cart_create_url">http://xml.amazon.
    com/onca/xml3?ShoppingCart=add&amp;f=xml&amp;dev-t=<xsl:value-of 
    select="$devtoken" />&amp;t=<xsl:value-of select="$associate" />
    &amp;<xsl:for-each select="exsl:node-set($random_products)/
    wishes:wishitem/Details">Asin.<xsl:value-of select="Asin" />=1
    &amp;</xsl:for-each></xsl:variable>

This removes the clutter of all the xsl:text elements, but it would need to be piled all on one line in order to keep undesired whitespace from getting into the URL. I made a small attempt at wrapping this line here, but line breaks and spaces would leave us with a non-functioning shopping cart URL. It's up to you to decide which to use--personally, I prefer the xsl:text clutter for the ability to add in comments and clarify things a bit.

Finally, having built the shopping cart URL, let's use it to get a shopping cart and wrap things up:

    <xsl:variable name="shopping_cart"
                  select="document($shopping_cart_create_url)" />

    <xsl:copy-of select="$shopping_cart" />

</xsl:template>

As an aside, this part is pushing the concept of a REST web service a bit: In the REST philosophy, requests using the GET method (which is what document() uses) should only return existing resources and not create new resources or cause modifications to happen. Instead, these sorts of actions should use a POST request. But, since we've already accepted a few rough edges and workarounds in this project so far, we won't let a point of esoterica like that stop us. (That and, well, this is the way Amazon designed their web service, so we'll take what we can get.)

Once you run this iteration of the stylesheet ([wishes-ex4.xsl][wishes_ex4]), you should get something like this XML as output:

<ShoppingCartResponse ...>
  ...
  <ShoppingCart>
   <CartId>...</CartId>
   <HMAC>...</HMAC>
   <PurchaseUrl>...</PurchaseUrl>
   <Items>
    <Item>...</item>
    <Item>...</item>
   </Items>
  </ShoppingCart>
  ...
</ShoppingCartResponse>

The AWS documentation describes the vital elements here like so:

CartId - The Cart ID is the unique identifier for a given shopping cart.
HMAC - The HMAC is a security token that must be passed back to Amazon Web Services for using an existing cart.
PurchaseUrl - Use the purchase URL to transfer the remote shopping cart from your application to Amazon so that your application's users may complete their purchases. The purchase URL merges the remote shopping cart with the Amazon.com shopping cart.

So, in short, whenever we want to do any sort of manipulation on this Remote Shopping Cart via AWS, we'll need to remember and later supply both the CartId and HMAC found in the XML returned at its creation. And, once we're all ready to check out, the PurchaseUrl points to where we'll need to browse in person.

Stay Tuned!

This concludes Part 2 of the Wish-of-the-Month Club. Following this will be the final part, where we tie everything together and start firing off monthly emails!

shortname=wishofthemonthclub2