0xDECAFBAD - Tag: json

Enter the LizardFeeder

2009-01-06T00:01:54+00:00

[caption id="attachment_1582" align="alignright" width="247" caption="The Mozilla Tree"][/caption]

Behind Firefox is Mozilla, and behind Mozilla is a community. And the Mozilla community acts a lot like an ecosystem, which can be visualized as a kind of living tree—not to confused with the mozilla-central tree. Oh yeah, and Mozilla is the name of both a Foundation and a Corporation.

Confused yet? If not, then we should talk so you can explain it to me, because it all looks pretty tangly and intertwingled to me. Nonetheless, it seems to work, and produces a good chunk of my favorite software and technologies.

There are many efforts to track what's going on—including planets and newsletters and bugzillas and wikis and repositories and tinderboxen. Some of these resources report on, or are driven by, the activity occurring in the others. Some are automated, and others are carefully stitched together by hand. None offer a full picture of what's going on in the Mozilla galaxy in a way that's casually comprehensible by a sane human being.

Of course, that's not a slight against any of these sites or the people maintaining them—extracting an overview from such an organic phenomenon is neither easy nor straightforward. But, it might be fun to try.

As an infovore and avid practitioner of continuous partial attention, my first impulse is to reach for a firehose and stick my head into the stream. Relax, defocus, and try to let my pattern recognizers do their thing—sometimes those pattern recognizers are in my head, and sometimes they're written in Python.

[caption id="attachment_1585" align="alignright" width="225" caption="Firefox Victory!"][/caption]

But, for Mozilla, I couldn't find a stream of sufficient volume or completeness to satisfy me or my robots. Happily, though, my feeding urge found itself aligned with a project to discover the patterns of contribution in the Mozilla community and to find a way to thank the contributors responsible.

So, while we're still working on the thank-you angle, allow me to introduce you to the Lizardfeeder. The LizardFeeder is a feed aggregator, whose source code is built atop Sam Ruby's Planet Venus. The LizardFeeder pulls together and archives activity streams from a wide variety of Mozilla community sources. Beyond the usual human-readable pages produced by a blog-gathering Planet, the LizardFeeder accumulates statistical and historical data meant for consumption and analysis by robots.

At present, the only robot navigating the LizardFeeder archives is an AJAX-ified user interface that animates the firehose as a near real-time or time-lapsed stream of events scrolling by.

This is just meant as a conversation starter, though. I'm hoping to gather feedback and find more sources, as well as to entice creative community members to come up with more sophisticated visualizations of this data.

So, take a look, check it out, and let me know what you think!

Archived Comments

dria
2009-01-06T01:26:18

Is there a list of what sources are already being read by the LizardFeeder anywhere? I scanned through the various links here but didn't see anything obvious :)
l.m.orchard
2009-01-06T01:37:15

Yeah, that part could use some improvement. There's a monster list here:

http://feeds.mozilla.com/sources.opml
Brian King
2009-01-06T10:11:46

Excellent stuff. I was going to ask about access to the list of sources for each category, but Dria beat me to it.
l.m.orchard
2009-01-06T15:26:50

Actually, it occurs to me that this config file might work as a more readable version of the list of sources:

https://svn.mozilla.org/projects/lizardfeeder/trunk/conf/config.ini-dist

There's also this, which is where most of the previous list came from:

https://svn.mozilla.org/projects/lizardfeeder/trunk/conf/hg-feeds.opml-dist
Austin King
2009-01-06T20:55:52

Thanks for the background surrounding lizard feeder. Great post!

+1 Dria and Brian

Maybe make the title of the link to the OPML more descriptive than just "Feeds" and/or link to it in the body of the UI too and write something around it to encourage other's visualizations.

Awesome work Les.
Sanjay Parekh
2009-02-04T20:38:04

Is the AJAX UI available anywhere for download? I'd like to hack it for another use altogether. Great visualization and great application. Good job.
Deen Seth.
2009-09-16T18:21:02

This is a very good idea. Can we accomplish the same result using Yahoo Pipe? Do you plan to add events from Bugzilla, and mailing list to the feed?

I am more interested in development activities. There aren't much in code category.

I am interested in analyzing development related events.

do not taunt happy fun JSON

2006-12-21T02:55:45+00:00

Allow me to repeat myself somewhat and clarify my general opinion of JSON and its use in cross-domain browser scripting:

Happy Fun JSON is not really an API. Happy Fun JSON is not a bold declaration of side-taking in the grand war of web service specifications. Do not base business models on Happy Fun JSON. Caution: Happy Fun JSON may suddenly accelerate to dangerous speeds or stop altogether. Happy Fun JSON contains a liquid core, which, if exposed due to rupture, should not be touched, inhaled, or looked at. Ingredients of Happy Fun JSON include an unknown glowing substance which fell to Earth, presumably from outer space. If Happy Fun JSON begins to smoke, get away immediately. Seek shelter and cover head.

Do not taunt Happy Fun JSON.

Having said all that? Happy Fun JSON is pretty fun to throw around.

Archived Comments

Geof F. Morris
2006-12-21T03:34:19

Did it hurt when Happy Fun JSON fell from Heaven? ;)
Robert Sayre
2006-12-21T03:55:47

Obviously, the new del.icio.us JSON url feed is crushing blow against XML Web Enterprise Services Declarative Net Neutrality.
jamesv
2006-12-21T15:14:24

I only wish you could pass Happy Fun JSON into Flash via FlashVars in IE. FF and Safari seem to have no problems with it.

It would make authoring a richer content version of sIFR much, much easier.
Jim Plush
2006-12-22T19:22:48

You should be banned from talking about JSON if you think you have to choose one or the other. A place for everything and everything in it's place.
Danny
2006-12-31T18:11:19

All this lynch talk - I just got throttled!

Running the script here on a series of URIs got me the 999 error (no sure how many had run). There's a 2 second pause between calls, I thought that would be polite enough. Still throttled now, maybe 10 mins later. Any suggestions for what would prevent throttling? How long it needs to recover?

btw, I think Happy Fun JSON is great, and not unreasonable for interop. But what isn't so good for interop is inventing a new HTTP code when there's 503 Service Unavailable (and a Retry-After header would be handy).
Damian Cugley
2007-01-22T14:16:27

@jamesv: The first thing I did with JSON at work was pass it in to a Flash movie via FlashVars. This used a JSON decode written in ActionScript grabbed from JSON.org, plus a JSON encoder written in C# (it did not take long to write because I did not attempt to serialize arbitrary objects). Some gratuitous URL-encoding was also involved. Compared with passing the same data structure via URL-encoded XML to be picked apart by ActionScript's DOM support, it was easy and much more like fun.

correlation is not causation

2006-12-21T02:01:38+00:00

By the by: Correlating the new del.icio.us JSON URL feed with Google's recent deprecation of the SOAP Search API is about as effective as noticing how often the clock reads 12:34 when you just happen to be looking at it.

Archived Comments

Joe Grossberg
2006-12-21T13:10:16

Because ... you say so? :)

Also, I think you misuse the word "correlation".

And I think there is a correlation -- both are indicative of a trend in "web services" today.

JSON and YAML are getting more popular in that arena and XML (especially SOAP) is something that a lot of programmers dislike workign with.

Tagometers and Travesties

2006-12-21T01:19:56+00:00

You know you've truly arrived in Silicon Valley when the darling new feature you just helped launch on a Web 2.0 site gets shredded in a rant by Dave Winer. :)

My day, having started with me in a foul mood, has been made. Pre-launch, the tooth marks in my tongue have gotten deep as I watch the comments pile up. It's exciting to have so many people debating something you've made. (Update: Seriously. I'm in a better mood now. It's made me laugh.)

Actually, it wasn't the feature itself that got shredded, so much as the new JSON data feed revealed before the intended star of the show made it on-stage, by way of a screencast preview on Y!DN by Matt McAlister and an amplification by Niall Kennedy.

The feed itself is mainly intended to power our new Tagometer page widget include thingy. Beyond that, leaving the feed decoupled from the include itself made sense to me, both for our own flexibility's sake and as a nod toward our mashup friends in the blogosphere and lazyweb.

You see, I like JSON. It was convenient in this use case of creating a remote blog widget fed by cross-domain sourced data. I, personally, didn't even have to reinvent anything that wasn't already implemented in assorted off-the-web tools. JSON worked for this purpose - it might not work for your purpose. ~~If it doesn't, please accept my insincere apology, but yours wasn't my primary goal. Attempts to string me up for this travesty will be met with giggles and pointing.~~ (Update: I realize I'm not the to-be-strung-up party responsible for the JSON travesty - I just couldn't resist being a smart ass for its own sake. I really do giggle and point a lot. My actual apologies for that.)

It's funny, laugh.

JSON URL feed is not really an API. JSON URL feed is not a bold declaration of side-taking in the grand war of web service specifications. Do not base business models on JSON URL feed. Caution: JSON URL feed may suddenly accelerate to dangerous speeds or stop altogether. JSON URL feed contains a liquid core, which, if exposed due to rupture, should not be touched, inhaled, or looked at. Ingredients of JSON URL feed include an unknown glowing substance which fell to Earth, presumably from outer space. If JSON URL feed begins to smoke, get away immediately. Seek shelter and cover head.

Do not taunt JSON URL feed.

Archived Comments

Dave Winer
2006-12-21T02:00:51

Not sure if I helped your mood or made it worse, but the rant wasn't about you.

http://www.json.org/xml.html

Tell me what you think of that bit of hyperbole. Someone there is trying to undermine a lot of work a lot of people did over a long period of time to get software to interoperate. When you say that's just my issue, you miss the point. It should be everyone's.
Geof F. Morris
2006-12-21T02:28:32

Aw, man. It should've been "Do not taunt happy JSON URL feed."
l.m.orchard
2006-12-21T02:47:44

Honestly, it brightened my mood. The smiley was neither ironic nor sarcastic. Something big I helped launch has been stirring an interesting discussion, however tangential my blame in the making.

Vanity and Carly Simon aside, I know I'm not the main perpetrator of JSON. I just couldn't resist an opportunity to be a smart ass. For that I have a sincere apology. I should probably strike it or delete it now that the giddiness of a major launch and a lot of attention has subsided.

But seriously, what I meant above is not that this is your issue, per se. This post is really more directed at some comments on your blog post, in bulk.

Our particular intentions behind using JSON here do not necessarily map to anyone else's expectation from a web service or standards-driven perspective. It seems like a lot of people are looking at this JSON data since its early release, divorced from it's main consumer: The Tagometer JS include. It works for that purpose - and incidentally might help others - but we're not working very hard to support other uses.

Apropos of that, I don't particularly buy the hyperbole about JSON being XML's superior in the general case. JSON's a brilliant hack, but it's got issues. And seriously, if anyone thinks that adding and removing script tags on the fly in a browser DOM is the best overall way to go about things... they should be strung up. It just so happens that right now, it's the only way to do some things.
timb
2006-12-21T11:22:52

hooray, i'm glad to see the json url feed.

however, i shouldn't have to generate the md5s of the urls i am interested in... i should just be able to give it the actual urls.
John Manoogian III (jm3)
2006-12-30T02:28:08

mmmmm, swilling the liquid core...!

Ajaxitagging

2006-08-16T13:12:07+00:00

Ever since I switched over to a new CMS back in February, I’ve been tagging all my journal entries. Until now, I haven’t been doing anything with those tags apart from exposing them in category elements in my RSS feed. Now that I’ve got a good head of steam going with my tags, I’ve decided to play around with them a bit.

Source: Adactio: Journal - Ajaxitagging

Book pimping time: In Hacking del.icio.us, I've got something not entirely unlike the above-linked hack. (Which, by the way, is good work!) Mine is much more basic, though - with less microformat, progress bar, and yellow fadey goodness. And, instead of AJAX, my hack uses JSONP. The Related Links hack in Chapter 9 of the book shows you how to include lists of the last few bookmarks you've posted under each tag / category / keyword used in a blog post.

Okay, okay, JSON is pretty hot

2005-12-19T04:17:55+00:00

The XML ~~purist~~ fanboy in me has had me pshaw'ing at JSON. But, now that the recent JSON release from Yahoo! reminded me of its existence and I gave it a shot myself, I have to admit that it's pretty hot—if only for the cross-domain bridging capabilities and the no-fuss parsing.

Although, I do worry about running into a poisoned payload someday that raids my cookie jar.

Archived Comments

fluffy
2005-12-19T06:50:57

JSON is currently the big Hot Thing at work. It's very good for some things, but IMO it's not nearly as robust as XML as far as generic interchange goes. Its big drawback is that it's not nearly as flexible, and since elements are either unordered or stored in an array, you have to agree upon the actual structure of the document before you send it across the wire (which isn't so much the case with XML where often all you care about is the nesting order which you can handle with XPath or similar).

It's GREAT for AJAX though, as long as you can trust the server of course.
Bill Humphries
2005-12-19T07:26:09

I thought AHAH was the current big thing. Just send HTML to the client and use Element.innerHTML.

FeedMagick gains an RSS-to-JSON filter

2005-12-19T03:01:51+00:00

So, I put some more infrastructure behind FeedMagick and lifted a RSS-to-JSON idea from John Resig to transform from MagpieRSS parsing to JSON output.

Here's a spiffy JSON-enriched demo. It was surprisingly easy, all told.

I'm not sure how much time I'll have to really put into this project, but this is the first decent effort I've put into a URL-line suite in a long time. I'm trying to throw in some bits to make it easy to build, document, and use these feed processing commands.