0xDECAFBAD - Tag: feedmagick

OPML reading lists in FeedMagick2

2007-10-17T07:22:47+00:00

For anyone who's interested: I've been hacking a little bit on FeedMagick2 again, with the latest addition being an OPML reading list feed blender.

What's an OPML reading list? Basically, it's the same as as OPML export of a feed reader's subscription list - only rather than doing a one-time import into another program, the OPML is itself treated as a live feed. A feed reader that supports OPML reading lists will continually check the list for updates and sync RSS/Atom feed subscriptions with its contents, maybe in a special sub-folder.

Here's a quick demo:

The itch I mean to eventually scratch is to replace the front page of decafbad.com with a live updating aggregation of the stuff I create and capture daily on the web. It'll be basically a self-assembling tumblelog pulled from many different services across the web. It'll also replace the footer of accumulated crud I've got on this very blog - which I thought was a good idea at one point, but now consider NeatLikeDigitalWatches.

With that in mind, the next thing I plan to develop is an hAtom module or XSL transform. This will turn the blended feed into an XHTML page. Maybe someday, hAtomic will launch, and I'll have a nice pretty style for the page too. Some time after that, I might work up a module that stows away dated historical archives of the feed and pages. I have further plans and ideas, but I'm trying to focus on the itchy spots first so that I might actually get something done in this round of serial enthusiasm.

Say hello to FeedMagick2

2007-04-30T03:06:32+00:00

Yeah, things have been basically silent around here thanks to post-work brain fryage and a general lack of things to say. Really, everyone else around the blogosphere seems to be covering things satisfactorily. However, I have been idly working on a new project over the past few weeks, namely a total rewrite and redesign of my format-ignorant feed filtering and munging kit dubbed FeedMagick.

You can find a demo installation of FeedMagick2 here and find it ready for checkout from SVN over here. It's basically just a step away from being a proof of concept, but I'm hoping to get around to fleshing out docs and battening down the hatches with tests. In any case, if my serial enthusiasm holds out, this thing could eventually subsume everything else I've done with feeds.

For now, peek at some of these highlights:

Master Personal Feed - One big feed blended from 10 other personal metadata feeds pulled from various Web-2.0-ish sites.
Feed to JSON via Magpie - Get feed data parsed by way of Magpie into JSON data structures
Flickr Favorites Feed - Feed of photos marked as favorites by a Flickr user, pulled via the API
jbox.com scraper - Pipeline composed of HTML Tidy and XSL to scrape jbox.com to build an RSS feed of new items for sale.

Beyond practical examples, there are some things under the hood that seem keen to me. Apropos of my pipes-via-web ramblings back in February, I'm trying out a few different approaches to pipelining feed content through processor modules. My original FeedMagick relied on feeding URLs to URLs as parameters. That, unfortunately, can be mighty cumbersome and inefficient. So, FeedMagick2 explores a few more approaches:

The first and obvious approach is to chain them together in a single script. So, I've got objects instances that pass content from one to the next. The thing is, the pipe works in reverse: The driver script asks the last module in the pipe for content, which then asks the one before it for content, and so on. At any point along the way, modules can cache the output of previous modules, and refrain from calling up the chain.
The second way to chain pipelines together is just like the first FeedMagick: Some pipelines start with fetching a URL. That can be an original feed, or a URL leading to the output of an antecedent pipeline. And, oh yeah, most pipelines are run via parameterized URLs, so there's that bit of handy recursion.
The third way to chain pipelines together is with HTTP POST: A pipeline can accept feed data via the request body of an HTTP POST, thus allowing antecedent pipelines (or even cURL scripts) to push data into the pipeline rather than getting pulled via URL. This is kind of like my years-old jiggery pokery with pipelines via XML-RPC, only much much simpler.

I'm also poking around at making all of the above available at the command line via PHP-CLI, and I'm having gratuitous fun exploring PEAR to roll my own stripped-down web framework. I still hate PHP, but I'm at least finding ways to entertain myself while I'm holding my nose. Of course, I find weird things entertaining.

And, as a side note, the only reason I'm using PHP is because I'd like to play around with the idea of the de facto WordPress installation requirements standard. That is: If you can run WordPress, you can run this. In reality, I don't think I'm there, but I'm hoping to get close. For one, I'm refusing to play with anything older than PHP 5.

Anyway, play with it, tell me what you think and give me a reason to keep hacking at it. :)

Archived Comments

jamesv
2007-04-30T11:39:15

Aw man, now I've got to port all the code you wrote while you were here over to this new hotness ;) I really like (and appreciate) the single script approach.

Is caching at a module level done automatically, or is that something I need to flag in my original call? Some installs of the original code base are now aggregating large sections of a pool of around 400 feeds, and eeking out even minor performances gains would be just lovely.
dRAUPP
2007-04-30T13:40:16

hawt.
l.m.orchard
2007-04-30T14:43:54

@jamesv: Take a look at the source to this big-ish pipeline, all the way at the end:

http://decafbad.com/2007/04/FeedMagick2/pipelines/masterfeed

You can basically slap a Cacher module at the tail-end or even middle of a long string of modules, and it'll cache the results of everything before it. I've got a cache lifetime set in the conf/config.php, and you can also set the lifetime in the Cacher parameters. There can multiple Cacher's per pipeline too.

This thing might not quite yet be even as stable / in working order as the original FeedMagick, but it might be worth poking at for you. :)
kellan
2007-04-30T16:02:31

For one, I’m refusing to play with anything older than PHP 5.

That must be nice.

And curse you, this looks interesting, now I've got to find time to look at it.
John Manoogian III (jm3)
2007-04-30T17:53:08

addendum to README:
Installation
cp conf/config.php-dist conf/config.php chmod a+w logs - RewriteBase /~lorchard/FeedMagick2 + RewriteBase /FeedMagick2
l.m.orchard
2007-04-30T17:58:49

@jm3: Ah! Good catch. I really need to eventually installer-ify that kind of stuff.
megalar
2007-08-05T12:25:34
Yesterday my host upgraded to php5 so I ran over to your svn dump and installed feedmagick2. After editing the $baseurl and rewritebase i tested it and it worked like a charm, so I took a nap. Upon waking I was gonna go play with it and see what hacks I could get going with it but something was fubar. Warning: fopen(/home/megalar/www/html/feedmagick/logs/feedmagick2-debug-20070805.log) [function.fopen]: failed to open stream: Permission denied in /usr/share/php/Log/file.php on line 216 are the errors I get as you can see @ my site. If it hadn't worked to begin with I would think module problems on my server. To my knowledge, and my host's knowledge, nothing has changed since the upgrade so I'm wondering if it is some sort of bug or a server problem that waited a few hours to reveal itself. The latter doesn't really make much sense to me but I can't rule it out since I'm not r00t on the box and am not 100% certain my host didn't bork something while I slept.
```
              anyhoo, your thoughts?
```

Thoughts on Pipes on the Web - Part II

2007-02-15T08:49:56+00:00

In the last post, I expressed concern for my own well-being over a lack of head-over-heels love for Yahoo! Pipes. On the surface, I think it's because it's not a freshly discovered concept for me, and also probably because I'm tired and a bit hacked out right now. But, I think there's a bit more to it.

First of all, I'm not impressed by GUI builders for most things. Don't get me wrong: That Pipes GUI is pretty sweet and quite an impressive implementation — but as a rule, such things don't quite get my geek on. So, there's that.

Another thing is that I've done something kinda like Yahoo! Pipes, called FeedMagick — only, it doesn't have a GUI and I mostly abandoned it after releasing the code and having used it for a project at my old job. It's one of those serial enthusiasms that I've figured I'd circle back to eventually. I still think it's a pretty cool idea.

All kudos to the Yahoo! Pipes team though: Unlike me, they've actually got a living and breathing project — which trumps a paged-to-disk serial enthusiasm most days.

So what else is really curbing my enthusiasm? Well, Yahoo! Pipes looks like a pretty self-contained pipes engine — data goes in one end, frobnication happens in the middle, data comes out the other side.

But, what I like about the notion of the URL-line is that you can take one URL and supply it as a parameter to another URL — making messy pipelined URLs while building a crazy web-wide distributed execution environment powered by HTTP and REST. This is the kernel of the notion that I think really excites me about pipe on the web — I just haven't had a chance to do much with it lately.

Depending on the perspective, true pipes on the web — that is, URLs fetching URLs — look to me like functional programming ala Lisp. Consider an idempotent GET request as a pure function call with no side-effects. Then, consider a GET request that accepts a URL as a parameter — it's a nested function call: the outer GET must make an inner GET to fetch the parameter-supplied URL. Give the inner URL another URL as a parameter, and you've got yet another nested function call.

But, if you like, ignore the theoretical benefits of functional programming — flip the nested function calls inside out and you've got a pipe. And, since you're using HTTP GET, you can get all the benefits of HTTP — like caching of execution results and a web full of distributed processing nodes, among other things.

I haven't explored Yahoo! Pipes deeply enough yet, so maybe I'm missing the features that pipe authors can used to call on other distributed pipe elements out on the web at large. But, I think that's what ultimately gets me psyched about pipes the web and hasn't yet for Yahoo! Pipes.

And, of course, I think I'm just a little hacked out and tired of whizbang new stuff right now. :)

Archived Comments

John Evans
2007-02-15T13:59:01

The thing that struck me about Pipes is that it seemed rather limiting to someone who can code but too geeky to someone who can't. Maybe I am wrong but I wonder that once the hype dies down who exactly the audience will be long term.
Stuart Robinson
2007-02-15T19:35:41

Don't be so apologetic! Conceptually it's cool, but the UI quickly becomes frustrating for anyone with text-based coding experience. I guess this is the web's visual basic and we are all going to be more interested in it's perl.

Thoughts on Pipes on the Web

2007-02-15T08:09:56+00:00

By way of my meme tracker, I found something like 15 entries linking to Yahoo! Pipes from my feed subscriptions. That's crazy — I've usually never seen more than 3-5 feeds linking to anyone thing, maybe as many as 8 if it's something really hot. I read about it over morning coffee and aggregator skimming; I heard about it on TWiT on the bus to work; I was told all about it in the hallway once I got to work. Yahoo! Pipes has been hard to miss.

With that in mind, it seems weird to me that I didn't get as excited about the release of Pipes as everyone else in my local blogosphere seems to have. Pipes has all the ingredients of something that should really set me off. Pipes, filters, mashups, RSS, Atom, RDF — (singing) these are a few of my favorite things! So, what gives? I'm genuinely worried that I'm sick, depressed, losing it, or otherwise replaced by a pod alien. Possibly all the above.

I knew about Pipes as an internal project at Yahoo! quite awhile ago — possibly even in my first week or two after having joined. I'm fuzzy on the timing here, but it's not important. Someone mentioned it to me at some point, I poked at it a bit from the internal network, and thought it was neat / keen. My main impression at the time was, "Cool, someone's doing FeedMagick, but done right and with a slick visual layout tool."

Then, I got really busy with del.icio.us and wandered off. So, the release wasn't big surprising news to me — my reaction was something like, "Huh, so they did a public release of that? Nice." Clearly I need more caffeine.

The thing is that the idea of pipes on the web is not itself a new concept to me. I think my XmlRpcFilteringPipes seems kinda silly in retrospect, and I'm all about the Power of the URL-line nowadays — but suffice it to say that I've been thinking about pipes on the web since as long ago as 2002 or so.

So, while I'm all jaded and apparently incapable of feeling the wonder, this seems to be one of the biggest revelations to people playing with Yahoo! Pipes:

Pipes? On the web? BRILLIANT!

So, it's not you, it's me. I think I've got a serious deficit of enthusiasm for what's really a pretty cool thing. I want to think a bit more about why that is.

Archived Comments

Kenneth
2007-02-15T14:06:37

http://yakubovich.blogspot.com/2007/01/visipipe-visual-version-of-core-utils.html

Might have been buried in all the buzz about Yahoo, but this sounds like a neat idea.

Kenneth Writer
Jeff Daly
2007-02-16T01:08:54

You're not alone. I was excited when I learned about Pipes, but ultimately I was let down by the lack of parsing tools. Pipes is seriously limited without any XPath/regular expression parsing capabilities. Fortunately, there is a post on XPath in the feature request page.

Pulling the Linkmobile back into the garage

2006-02-09T15:49:42+00:00

As a follow-up to Tuesday's post, I've disabled the nightly link post as of now and turned off the links category on the main page template.

I got a surprising amount of good and friendly feedback on the change in comments, IMs, and emails. And, the balance to me sounds like there are people who sorta liked the change and a lot of people just plain turned off by it—the nays were louder than the yeas. I was on the fence, leaning toward dislike, so down go the links.

I'd still like to reintegrate my link posting output on del.icio.us into this site in some other fashion soon, because that's where the bulk of my hasty blogging happens these days. In the course of writing this new book, I've got a panoply of other options available from sidebars to AJAX-powered geegaws. I've also considered pulling FeedMagick into the mix to offer a "build your own" remixed feed option, but I've not yet had the ambition to deliver on that.

So sorry link fans, it's back to the status quo for me, for now.

Sometimes the lazyweb delivers with a deluge

2005-12-19T23:15:03+00:00

Kellan: " '… and that’s not my code.' ouch"

Eric: "Right now, the only thing we have special for Magpie RSS is that we don’t serve Atom to that user-agent if the version is 0.5."

Rasmus: "I wrote a simple little PHP 5.1-based RSS parser a while back and it doesn’t have any problems with that ... feed."

Source: Comments on my previous post

Facepalm, headdesk. As it turns out, the issue I was having with my PHP XML parsing was that I was trying to cram raw gzip streams down its throat, caused by some odd things I was trying to do with HTTP headers copied cargo-cult style from a Python module I've tried reinventing in PHP.

In other words, completely braindead work by someone who should know better, even though he's relatively new to PHP—where he is defined as me.

But, kudos to the people of the lazyweb for taking the time to triangulate me and remove all doubt within scant hours of my post. Not counting a half-dozen private emails, I got very quick responses from the author of MagpieRSS, the CTO of FeedBurner, and the creator of PHP himself—each very nicely, clearly, and rightly saying:

"...and that's not my code."

Be careful when you attempt to invoke the lazyweb, because it might just respond!