0xDECAFBAD - Tag: webdev

Putting clouds in boxes for webdevs at Mozilla

2011-10-02T22:45:37+00:00

TL;DR: At Mozilla, we're redoubling efforts to attract contributors. And, in the Webdev team, we're using virtual machines to make it even easier to get started hacking. It makes no sense to demand that a JavaScript ninja or a CSS artisan also be a Python-wielding sysadmin, so we're trying to build ready-baked VMs for our projects.

Lieferung 3 by erix on Flickr

Firefox is more than just a desktop browser, and Mozilla is more than just Firefox. An ever-growing mass of our code is going toward building web sites and services that build, ship, update, support, promote, augment, document, sync, share, identify, and report. (And I'm sure I'm missing a further cast of hundreds, here.)

Sighting storm clouds on the horizon

In other words, more and more of Mozilla is in the cloud. Unfortunately, many of these projects are really hard to get into as a contributor. It's often hard to find what needs doing in Bugzilla, and it's often hard to get the code up and running in a meaningful way without being both a bit of a sysadmin and a Mozilla insider.

This is a huge problem. Although Mozilla has a core group of employees, we're not nearly enough to build all of these projects and pursue the mission on our own. We've gotten to where we are thanks to the volunteer efforts of thousands of contributors—and the only way we'll get where we want to go is through the efforts of thousands (or millions) more. The good news is that is feels like we're waking up to this problem and starting to get really serious about it.

Sharing recipes for fluffy clouds

For my part, I've spent idle time over the past few years trying to work out how to make webdev projects more accessible to new contributors. We've come a long way from the days of just uploading WordPress to an FTP server; these things come with a lot of moving parts nowadays, and the INSTALL.txt files run long and consume weekends. And so, they filter out all but the most dedicated (or masochistic) volunteers.

For months, I spun my wheels playing with virtual machines and exporting appliances, doing a lot of things by hand. But then, earlier this year, I discovered both Vagrant and Puppet and had my mind blown. With those tools, I could share recipes for VMs right alongside my code.

I would never have to set up a webdev environment for myself by hand again—all the fiddly bits of installation and configuration could be codified, checked into github, and flushed from my brain. But, the real magic is that I could share this stuff. So, future coworkers and collaborators could benefit from the same reproducible dev enviroments—never having wasted brainpower and weekends on an INSTALL.txt to begin with.

This is a really big deal. I presented on it this past June at Open Source Bridge in Portland (slides). I've been talking myself blue about this stuff for months, in opposition to my usual serially enthuastic habits. But, it looks like I've infected my coworkers at Mozilla, because now we're talking about developing with Vagrant, Puppet, and Playdoh.

No assembly required

So, with Vagrant, Puppet, and Playdoh, we'll have projects that you can fire up with just a git clone and a vagrant up in a command line shell. Forget about MacPorts or Homebrew on Mac OS X, or whatever it is that people using Windows have to install. Take a bike ride or go to lunch, and you'll have a fully configured server running and source code ready for hacking.

I think we can do better, though. For one, spinning up a fairly complex virtual machine from scratch with Vagrant and Puppet takes a long time and a lot of bandwidth. This can be sped up by packaging pre-built boxes. So, one improvement we can pursue is to spin up VMs with Vagrant as a part of our continual integration and deployment processes and package boxes for public download. These are also handy on a thumb drive for sharing at hack events and meetups.

This still requires Vagrant and working with the command-line, though. Not everyone who has something valuable to contribute is a command-line jockey. And, a few of us webdevs have run into issues with Vagrant—it's still early days for that project, after all. Additionally, not everyone who could be a Mozilla contributor owns the kind of hardware necessary to support running a substantial virtual machine.

So, I've also started playing with Puppet by itself, to build plain-vanilla VM appliances and to spin up cloud servers on Amazon EC2 and Rackspace Cloud Servers. It's going pretty well, and it's conceivable we could do without Vagrant to script the build of downloadable VM appliances and to spin up disposable cloud servers.

So, now what?

There are still some issues to be worked out, so I've started a wiki page. And, we need to spread the word that we're doing this to see what people think—and more importantly, see if this helps attract more help. From my years at Mozilla so far and bouncing between projects, VM appliances are things I wish I'd had when faced with a new project. So, having been there—albeit with insider advantages—I'm pretty optimistic that this is a good step forward.

Of course, it's not the only step: The other thing I mentioned—and have conveniently avoided addressing again throughout the rest of this post—is the issue of finding things to do in Bugzilla. Well, we're working on that too. It's a bit beyond the scope of this blog post, but we have a Mozilla-wide effort to start classifying good first bugs for new contributors and identifying mentors to contact for help.

So... what do you think?

Moved to Jekyll and Disqus

2011-06-08T00:00:00+00:00

TL;DR: My blog is now produced by Jekyll, with comments hosted by Disqus.

This blog has transitioned between a handful of publishing platforms - including Movable Type, blosxom, and Wordpress. Well, apparently, it's that time again. I've been toying with making a change again for a few years, and finally just got all the pieces together over the past week. This blog is now just a pile of static files produced by Jekyll, and the comments are now hosted by Disqus.

This has a lot of interesting implications:

I can neglect my server for long periods of time without worrying that my long-outdated copy of Wordpress has been exploited. Really, there's no reason for me to have a full-featured CMS running here.
I could host my entire blog on Amazon S3 or something similar, and be more neglectful by reducing active code even further. The only crappy thing is that I don't think I can send decafbad.com directly to S3, and would need to redirect to a CNAME on a subdomain on blog.decafbad.com.
I can edit my blog directly on GitHub and, assuming I get the webhooks set up, have my server rebuild the HTML automatically on a git push. I could probably use GitHub Pages and the Jekyll support there, but I suspect I'll be doing some Weird Things that they won't handle. (And probably for good reason.)
Since Disqus has all my comments now, I can stop worrying about running that code on my server too. I'm a little antsy about putting that in the cloud, but the escape routes are well lit and I should be able to replace it easily if I need to. Self-hosted, JS-include-based comments is a project in my TODO list.
My writing here is now all stored in individual text files, formatted with Markdown, annotated with YAML metadata, and subject to revision control under git. This combination feels like it has a lot of longevity and potential for survival even beyond this current experiment with Jekyll
Writing, editing, and publishing entries here can now more closely match my daily reality of living in MacVim and git. It sounds lame, but the notion of swapping over to a Wordpress admin page and working in a browser textarea has often kept me from even starting a post.

But, some things are broken, now:

Fixed ~~No more tag pages. I don't think very many humans visit these pages, but a lot of search engines do.~~
Fixed ~~No more tag feeds. I expect to hear about this soon, since this blog gets syndicated to Planet Mozilla via my "mozilla" tag feed.~~
No more year / month archive pages or sidebar widget. Not sure how many people actually stroll down memory lane here on my blog, but I do have a gigantor huge list of all my posts evar, now.
A general lack of contextual links to related posts and such. Turns out that running the LSI option on Jekyll is completely horrible.
Regeneration seems less-than-happy in general, and I wish it did conditional regneration. That is, only produce the HTML for things that need it because dependencies have changed. No need to rebake the whole world just because I changed one file.

The bright side is that there are solutions to all the broken things, and now I have something to tinker with again on my blog. So, let me know if anything else looks broken, and hopefully this will get me spewing some more words out onto this thing soon.

Archived Comments

Les Orchard
2011-06-08T17:45:54

Testing a comment here on my new post.
bkclements
2011-06-08T17:59:01

Hi, did you happen to look at blogofile for similar functionality? I'm still trying to decide which way to go. I prefer ReST markup..
- Les Orchard
  2011-06-08T18:08:26
  
  I've had my eye on Jekyll for awhile now and have a few friends also using it, so I haven't really checked out any other packages. Might be nice to poke around at something based in Python since my Ruby-fu is not strong, but then getting stronger in Ruby is not a bad thing for me. And, for myself, I'm pretty stubbornly rooted in Markdown for all my writing online
BillSeitz
2011-06-08T22:01:56

What kind of metadata are you storing in YAML?
- Les Orchard
  2011-06-08T22:30:10
  
  Pretty simple stuff: Title, tags, etc. You can see the source for this post here on github, for example: https://raw.github.com/lmorchard/blog.decafbad.com/master/_posts/2011-06-08-moved-to-jekyll.markdown I had once-upon-a-time thought about storing posts as RFC 822 messages, with the metadata in headers like email messages. But, this "YAML front matter" approach of Jekyll is a bit more flexible
Dave Dash
2011-06-08T23:02:15

Les, Check out what I did - I actually host my site on github thanks to TofuMatt's suggestion. It works well and I have archives and whatnot. In fact, most of my links didn't break. This also means that I don't have to regenerate and sync, since github knows jekyll: https://github.com/davedash/davedash.github.com
- Les Orchard
  2011-06-08T23:56:45
  
  Hmm... I might have to try that, but on a subdomain like blog.decafbad.com. I have some non-blog stuff here on decafbad.com and don't want to turn the whole thing over to github
  - Dave Dash
    2011-06-08T23:59:33
    
    It caused a few things to happen - speed up my deploy time and force me to check things into git. Also my codez build tags and stuff...
tiernano
2011-06-09T07:19:54

So, how did you move directly from Wordpress to Jekyll? was it simple enough to migrate, or is it a pain?
- Dave Dash
  2011-06-09T13:51:27
  
  There is a migration script for posts. I may have had to adjust some minor things. Risqué has a migration for comments. it wasn't too bad.
  - tiernano
    2011-06-09T14:14:31
    
    Already using Disqus for comments on my wordpress install... It's the blog data itself that's the problem... Will have a look and play with the migrators...
    - Les Orchard
      2011-06-09T14:24:16
      
      FWIW, the script I tweaked and used for exporting from my Wordpress database is here in github: https://github.com/lmorchard/blog.decafbad.com/blob/master/_import/wordpress-decafbad.rb The only real pain was getting the SQL query to pick up tags the way I wanted. Otherwise, it's just getting things into text files.
      - tiernano
        2011-06-09T15:07:03
        
        Cool beans man. Will try that out when I get a few min...
Lee Roberson
2011-06-09T16:45:18

Google Reader keeps getting tricked into thinking several of your posts are new, might be some kind of issue with your pubDate field in RSS, not sure.
- Les Orchard
  2011-06-09T16:48:38
  
  Yeah, that's my fault. :/ I had initially exported all my posts missing the actual time part of the pubDate, and then just a little while ago re-exported with the proper datestamps
  - Dave Dash
    2011-06-09T16:59:50
    
    You are worse than Delicious!
    - Les Orchard
      2011-06-09T17:05:52
      
      That's probably my fault, too.
Lee Roberson
2011-06-09T16:45:53

Whups.
Lloyd Hilaiel
2011-06-09T17:45:00

Have you had much success in customizing the features and display of the injected DISQUS UI? Thile there are several knobs in their UI, but the only form of customization that's worked for me is fiddling with external javscript and CSS... Specifically things like disabling media upload and changing default sort, and threading seem to be no-ops.
- Les Orchard
  2011-06-09T18:12:39
  
  Hmm... I haven't even bothered to try customizing yet. I just dropped it in and it vaguely matched the rest of my page style, so I left it alone so far
Craig Maloney
2011-06-11T16:00:04

Oh c'mon, there's nothing in the world more fun than updating Wordpress blog every time someone manages to sneeze wrong and cause an exploit. Sheesh, where's your sense of discipline.
jolyonwagg1
2011-06-21T23:52:22

Love to use disqus, and would like to integrate it onto my Wordpress blog for comments, as it is so user friendly, but I am not to hot though on all the HTML stuff, any advice would be greatly appreciated, cheers from across the pond.
Johan Sundström
2011-08-15T04:23:30

Your sidebar archive widget was rather cool, layout wise; I bookmarked it as design inspiration, back when it existed.
- Les Orchard
  2011-08-16T15:58:00
  
  Hmm, I should see if I can find a way to revive that thing as a JS widget or something
Timo Zimmermann
2011-09-01T12:48:10

how do you feel about using Disqus for your comments? Any problems or the urge to write your own JS based commenting system? I just feel a bit uncomfortable with a 3rd party provider for something important like comments...
- tiernano
  2011-09-01T12:54:22
  
  Personally speaking, i have been using Disqus comments on my WordPress blogs for a while now (couple of years) and its working grand! don't have to worry too much about Spam, they manage backups, etc. you can even reply to, or delete comments (if spam does get though) by email, so you can manage them on your phone.

Firefox Sync server on Google App Engine

2010-07-05T23:00:56+00:00

TL;DR: I built an implementation of the Firefox Sync server API for Google App Engine.

To celebrate Independence Day, I figured I might take a shot at liberating Firefox Sync from the tyranny of Mozilla's servers.

Thus, over the past few days, I've built a sync server using the 1.0 Sync API, hosted on Google App Engine.

I lied about the tyranny thing, though—I just wanted to say something clever about the holiday. In reality, with respect to Firefox Sync, Mozilla has done all of the following:

Published the Sync API spec;
Released the source code for the server used in-house;
Explicitly included the option to use a custom server when setting up sync in the browser.

This means that, although Mozilla offers servers to go along with Firefox Sync, you're totally free to take your data elsewhere. Since your sync data is encrypted and practically opaque to the server, there's no direct profit for Mozilla in offering free sync hosting—not even through any clandestine data mining for devious purposes. It's just that sync makes Firefox a better browser, and somebody has to run some servers to make it work.

So, there's every incentive to make it easy for you to switch sync providers and stop freeloading on Mozilla's servers. Building a server on Google App Engine means I can freeload on Google's servers!

I kid, of course. No one's really complaining about freeloaders, and App Engine has quotas in place to head off any serious mooching—which is why I'm not telling you where to find my sync server deployed on Google App Engine, by the way.

No, I did this because:

Firefox Sync and Google App Engine are interesting and important technologies;
I've already done a bit of work on the PHP-based Firefox Sync server at Mozilla;
I really wanted to take a break from PHP and spend some time with my old friend Python.

There are, of course, a number of bugs in this server. But, it seems to be working between a number of machines and browser profiles I have at home. Things are really in need of optimization, it suffers from my inexperience with App Engine, and I keep running into those aforementioned App Engine resource limits—especially when updating or deleting large numbers of items (ie. 1000's to 10000's of items).

Pull requests and issue reports on GitHub are welcome!

A next step I'd like to take with this thing is to revisit another old friend, the desktop web app server. (Also known as the desktop website.) It seems to me that it would be interesting to scale this server down to a household appliance—say, just for use by my wife and I.

I'd be especially happy if the work I'm doing for a Google-hosted app could be self-hosted at home. Seeing as the development environment for App Engine runs on my laptop, I'm willing to bet I can hack the whole shebang into a simple, special-purpose app to download and double-click on a home desktop PC for use as your sync hub.

Anyway, check it out and let me know what you think.

Archived Comments

James Socol
2010-07-06T01:28:02

Assuming you already had a network and a Mac Mini or something at home, it would be pretty interesting to run your own small Sync server just to keep your home computers together. Maybe even use localtunnel for when you're away.
Mike
2010-07-06T01:36:37

You are a real-life hero!
l.m.orchard
2010-07-06T01:52:00

@James: Yup, exactly that. Maybe even roll in some SSL and UPnP port forwarding for easier setup behind a home router. And, if there were a Windows version, it could run on that cruddy desktop back in the spare room.

@Mike: Naw... this guy is the real hero, even John Lilly agrees!
Crash
2010-07-06T09:20:54

Is there already a App Id for this tool?
l.m.orchard
2010-07-06T10:18:57

@Crash: Sure, there's an app ID for my instance of this sync server. But, as I said in the blog post, I'm not sharing it. At least, not until or unless I get the quota usage down to a point that I wouldn't exhaust the free hosting limits.

It's pretty easy to deploy your own sync server on AppEngine with the source, though.
Peter Petrov
2010-07-06T13:18:11

@Leslie: The app ID of your instance is visible in app.yaml, so you've shared it anyway :)
l.m.orchard
2010-07-06T17:18:47

@Peter hah! Right you are! But, at least you had to look at the source of the app first to figure that out :) That is mostly the point of this blog entry after all. Anyone who does that and then uses my installation anyway will probably be sad when I regularly blow away data and eventually make it invite only
Tobias
2010-07-07T00:24:41

Neat! TyphoonAE http://typhoonae.googlecode.com might help you to build your household appliance.
Kumar McMillan
2010-07-07T20:59:25

Hey Les, this is super cool! Google App Engine's Datastore API is still very shaky though. In fact, it's been so bad lately that I've been considering porting one of my heavily used apps over to something else. Thankfully, this post suggests that fixing the Datastore is their top priority: http://googleappengine.blogspot.com/2010/06/datastore-performance-growing-pains.html
Robert Kaiser
2010-07-07T21:21:51

Well, I actually feel more comfortable with my data being on Mozilla servers than on Google servers - even with the decreased trust I have in the Mozilla organization, I'd trust it more than Google any day! ;-)
Chris Arndt
2010-09-28T11:06:55

Instead of porting GAE to your desktop, why don't you just port your app to plain Django? Django should run off your desktop with no problems.

Case Study: Building a Bookmark Management UI for Mozilla's BYOB

2010-06-22T22:16:24+00:00

TL;DR: I just wrote a long case-study about my process in building a bookmark management feature for Mozilla's Build Your Own Browser web application.

So, remember that tutorial I wrote about writing an article in Tinderbox? I'd mentioned that it was a digression from another, different article.

Well, I just finished a first draft of that article:

It's a case study about building a bookmark management feature for Mozilla's BYOB.

Light on code but heavy on narration, it's about what I did and what I was thinking while I did it. There are links to the end product for reference, but it isn't a demonstration of some new trick. Rather, it's a look at my process as a whole.

I wrote this, in part, for myself: The writing has helped me review things. But, I'm hoping someone else reads it, and then picks up something new or can offer some interesting critique in return.

And, because I'm curious and like looking behind the scenes, I'd love to see this kind of write-up from more webdevs. There don't seem to be a lot of detailed case studies versus quick one-shot tutorials on isolated techniques. It's probably because the writing is time-consuming, as is the reading. It's not in the genre of bite-sized attractions optimized for promoting blog traffic, but I'd like to see more of them all the same.

Using Tinderbox to write articles for the web

2010-06-07T04:41:03+00:00

TL;DR: I just put together a tutorial about writing an article in HTML using Tinderbox from Eastgate Systems, Inc.

Update: There's a Belorussian translation of this post provided by PC - very cool!

Tinderbox Icon by Bryan Bell, via flickr.com/photos/factoryjoe

I've been toying with a set of writing topics to get my blog going again, apropos of my notion of writing for myself. But, I like to write at length and have finally decided that life's too short to write in browser <textarea>'s, no matter how cleverly augmented.

I did buy MarsEdit 3, which is a fine app for dashing off short entries. But, for longer things, I found myself doing the actual writing elsewhere and pasting the text into MarsEdit for posting later. I even wrote and posted a whole fanfic novella this way, and it was a silly way to do things.

Well, one of the places where writing actually happens for me over the past few years is within Tinderbox documents. And, as it turns out, Tinderbox has some powerful HTML export facilities that I've left underused until some things finally clicked for me.

So, I thought I'd try turning that experience back upon itself to produce something worth writing. Without further ado, here's the source in GitHub, and the article exported to HTML for easy reading.

The funny thing is, I started working on this tutorial in the middle of writing another article. So, this post is a particularly advanced form of Yak Shaving. I hope it ends up handy for someone, and I hope it leads to the real articles I'm hoping to produce.

HTML5 drag and drop in Firefox 3.5

2009-07-15T15:26:40+00:00

Oh hey, look! It's another blog post—and this one is cross-posted on hacks.mozilla.com. I won't say this is the start of a renewed blogging habit, but let's see what happens.

Drag and drop is one of the most fundamental interactions afforded by graphical user interfaces. In one gesture, it allows users to pair the selection of an object with the execution of an action, often including a second object in the operation. It's a simple yet powerful UI concept used to support copying, list reordering, deletion (ala the Trash / Recycle Bin), and even the creation of link relationships.

Since it's so fundamental, offering drag and drop in web applications has been a no-brainer ever since browsers first offered mouse events in DHTML. But, although mousedown, mousemove, and mouseup made it possible, the implementation has been limited to the bounds of the browser window. Additionally, since these events refer only to the object being dragged, there's a challenge to find the subject of the drop when the interaction is completed.

Of course, that doesn't prevent most modern JavaScript frameworks from abstracting away most of the problems and throwing in some flourishes while they're at it. But, wouldn't it be nice if browsers offered first-class support for drag and drop, and maybe even extended it beyond the window sandbox?

As it turns out, this very wish is answered by the HTML 5 specification section on new drag-and-drop events, and Firefox 3.5 includes an implementation of those events.

If you want to jump straight to the code, I've put together some simple demos of the new events.

I've even scratched an itch of my own and built the beginnings of an outline editor, where every draggable element is also a drop target—of which there could be dozens to hundreds in a complex document, something that gave me some minor hair-tearing moments in the past while trying to make do with plain old mouse events.

And, all the above can be downloaded or cloned from a GitHub repository I've created especially for this article—which continues after the jump.

The New Drag and Drop Events

So, with no further ado, here are the new drag and drop events, in roughly the order you might expect to see them fired:

dragstart: A drag has been initiated, with the dragged element as the event target.
drag: The mouse has moved, with the dragged element as the event target.
dragenter: The dragged element has been moved into a drop listener, with the drop listener element as the event target.
dragover: The dragged element has been moved over a drop listener, with the drop listener element as the event target. Since the default behavior is to cancel drops, returning false or calling preventDefault() in the event handler indicates that a drop is allowed here.
dragleave: The dragged element has been moved out of a drop listener, with the drop listener element as the event target.
drop: The dragged element has been successfully dropped on a drop listener, with the drop listener element as the event target.
dragend: A drag has been ended, successfully or not, with the dragged element as the event target.

Like the mouse events of yore, listeners can be attached to elements using addEventListener() directly or by way of your favorite JS library.

Consider the following example using jQuery, also available as a live demo:


    Drag me!
    Drop here!

Thanks to the new events and jQuery, this example is both short and simple—but it packs in a lot of functionality, as the rest of this article will explain.

Before moving on, there are at least three things about the above code that are worth mentioning:

Drop targets are enabled by virtue of having listeners for drop events. But, per the HTML 5 spec, draggable elements need an attribute of draggable="true", set either in markup or in JavaScript.

Thus, $('#newschool .dragme').attr('draggable', 'true').
The original DOM event (as opposed to jQuery's event wrapper) offers a property called dataTransfer. Beyond just manipulating elements, the new drag and drop events accomodate the transmission of user-definable data during the course of the interaction.
Since these are first-class events, you can apply the technique of Event Delegation.
What's that? Well, imagine you have a list of 1000 list items—as part of a deeply-nested outline document, for instance. Rather than needing to attach listeners or otherwise fiddle with all 1000 items, simply attach a listener to the parent node (eg. the

Using dataTransfer

As mentioned in the last section, the new drag and drop events let you send data along with a dragged element. But, it's even better than that: Your drop targets can receive data transferred by content objects dragged into the window from other browser windows, and even other applications.

Since the example is a bit longer, check out the live demo and associated code to get an idea of what's possible with dataTransfer.

In a nutshell, the stars of this show are the setData() and getData() methods of the dataTransfer property exposed by the Event object.

The setData() method is typically called in the dragstart listener, loading dataTransfer up with one or more strings of content with associated recommended content types.

For illustration, here's a quick snippet from the example code:

var dt = ev.originalEvent.dataTransfer;    
dt.setData('text/plain', $('#logo').parent().text());
dt.setData('text/html', $('#logo').parent().html());
dt.setData('text/uri-list', $('#logo')[0].src);

On the other end, getData() allows you to query for content by type (eg. text/html followed by text/plain). This, in turn, allows you to decide on acceptable content types at the time of the drop event or even during dragover to offer feedback for unacceptable types during the drag.

Here's another example from the receiving end of the example code:

var dt = ev.originalEvent.dataTransfer;    
$('.content_url .content').text(dt.getData('text/uri-list'));
$('.content_text .content').text(dt.getData('text/plain'));
$('.content_html .content').html(dt.getData('text/html'));

Where dataTransfer really shines, though, is that it allows your drop targets to receive content from sources outside your defined draggable elements and even from outside the browser altogether. Firefox accepts such drags, and attempts to populate dataTransfer with appropriate content types extracted from the external object.

Thus, you could select some text in a word processor window and drop it into one of your elements, and at least expect to find it available as text/plain content.

You can also select content in another browser window, and expect to see text/html appear in your events. Check out the outline editing demo and see what happens when you try dragging various elements (eg. images, tables, and lists) and highlighted content from other windows onto the items there.

Using Drag Feedback Images

An important aspect of the drag and drop interaction is a representation of the thing being dragged. By default in Firefox, this is a "ghost" image of the dragged element itself. But, the dataTransfer property of the original Event object exposes the method setDragImage() for use in customizing this representation.

There's a live demo of this feature, as well as associated JS code available. The gist, however, is sketched out in these code snippets:

var dt = ev.originalEvent.dataTransfer;    

dt.setDragImage( $('#feedback_image h2')[0], 0, 0);

dt.setDragImage( $('#logo')[0], 32, 32); 

var canvas = document.createElement("canvas");
canvas.width = canvas.height = 50;

var ctx = canvas.getContext("2d");
ctx.lineWidth = 8;
ctx.moveTo(25,0);
ctx.lineTo(50, 50);
ctx.lineTo(0, 50);
ctx.lineTo(25, 0);
ctx.stroke();

dt.setDragImage(canvas, 25, 25);

You can supply a DOM node as the first parameter to setDragImage(), which includes everything from text to images to canvas elements. The second two parameters indicate at what left and top offset the mouse should appear in the image while dragging.

For example, since the #logo image is 64x64, the parameters in the second setDragImage() method places the mouse right in the center of the image. On the other hand, the first call positions the feedback image such that the mouse rests in the upper left corner.

Using Drop Effects

As mentioned at the start of this article, the drag and drop interaction has been used to support actions such as copying, moving, and linking. Accordingly, the HTML 5 specification accomodates these operations in the form of the effectAllowed and dropEffect properties exposed by the Event object.

For a quick fix, check out the a live demo of this feature, as well as the associated JS code.

The basic idea is that the dragstart event listener can set a value for effectAllowed like so:

var dt = ev.originalEvent.dataTransfer;
switch (ev.target.id) {
    case 'effectdrag0': dt.effectAllowed = 'copy'; break;
    case 'effectdrag1': dt.effectAllowed = 'move'; break;
    case 'effectdrag2': dt.effectAllowed = 'link'; break;
    case 'effectdrag3': dt.effectAllowed = 'all'; break;
    case 'effectdrag4': dt.effectAllowed = 'none'; break;
}

The choices available for this property include the following:

none: no operation is permitted
copy: copy only
move: move only
link: link only
copyMove: copy or move only
copyLink: copy or link only
linkMove: link or move only
all: copy, move, or link

On the other end, the dragover event listener can set the value of the dropEffect property to indicate the expected effect invoked on a successful drop. If the value does not match up with effectAllowed, the drop will be considered cancelled on completion.

In the a live demo, you should be able to see that only elements with matching effects can be dropped into the appropriate drop zones. This is accomplished with code like the following:

var dt = ev.originalEvent.dataTransfer;
switch (ev.target.id) {
    case 'effectdrop0': dt.dropEffect = 'copy'; break;
    case 'effectdrop1': dt.dropEffect = 'move'; break;
    case 'effectdrop2': dt.dropEffect = 'link'; break;
    case 'effectdrop3': dt.dropEffect = 'all'; break;
    case 'effectdrop4': dt.dropEffect = 'none'; break;
}

Although the OS itself can provide some feedback, you can also use these properties to update your own visible feedback, both on the dragged element and on the drop zone itself.

Conclusion

The new first-class drag and drop events in HTML5 and Firefox make supporting this form of UI interaction simple, concise, and powerful in the browser. But beyond the new simplicity of these events, the ability to transfer content between applications opens brand new avenues for web-based applications and collaboration with desktop software in general.

Archived Comments

David Semeria
2009-07-16T17:56:52

Wow - thanks for a great introduction to D+D in HTML 5, I'm really looking forward to giving it a thorough road test.

The key part of the implementation are the target listeners, because, as I'm sure you're aware, onmouseover would historically never fire over the target element because the dragged item would always 'cover it up'. BTW, my proposed solution was to add an 'event transparency' property, which would have made the dragged item invisible from the point of view of selected events, eg onmouseover.

This implementation is going to take a lot of pain away. You have no idea how many hoops I had to jump through to get a fully generic D+D implementation working without these new calls.

If you're interested, you can see it working here: http://lmframework.com/page.php?id=vd_twig_short_2
Alex
2009-07-17T00:13:23

Hey,

Looks great and useful article. For dragging from external applications (such as the Desktop) is a normal webpage able to get the data from the drop? I was just playing with it and seem to always get what seems to be a security error with getData or mozGetDataAt. However, the documentation seems to state that on drop that data should be made available to the page. Likewise, in the same way an image can be dragged off the page onto the desktop, can an arbitrary element be dragged onto the desktop with whatever file content to be saved? Thanks for the help.

Alex
Paul Tarjan
2009-07-17T04:52:54

Very nice, I didn't know that FF 3.5 actually had this implemented. Time to start playing :)

Also, how did you do your code posts? Was it just pasting in HTML or do you have a better setup?
Animal
2009-07-17T14:35:21

Seems pretty pointless pressing ahead with this when you just can't write a web app to use it. There's no support. All a bunch of whizzy fun I'm sure, but sod-all use.
l.m.orchard
2009-07-17T16:23:10

@Animal: Why can't you write a web app to use it? It's in the HTML5 spec, works in Fx3.5 and Safari 4 / WebKit. It doesn't have universal support yet, or course, but new standards never do.
paul
2009-07-21T08:08:26

very nice indeed! i like this, it has definitely opened up new avenues for interaction without using flash. but it will take time before people upgrade to the new browser. a lot of people who browse the web wont realise what the technology will bring and will use their current browser because it does the job!
Joel
2009-07-21T14:36:04

Excellent demos of the new features.

When using a feedback image on a zoomed in page, should the image not also be scaled up automatically?
RenegadeX
2009-07-22T04:13:46

Dragging & dropping is as you say, a "one of the most fundamental interations" that users of computer graphical interfaces have come to know, and expect.

It therefore completes dumbfounds me how it is that we still can not natively drag & scroll (& then drop) in Firefox. We can only drag & drop an item within the currently visible portion of a Firefox webpage or tab, no further up, no further down.

That is should be limited makes absolutely no sense. Microsoft built the function into their Windows 3.x (file) Explorer, and then when visual browsers came along, naturally carried it through into Internet Explorer.

Firefox Bug 41708, "Should be able to scroll in the viewport while dragging" was opened in June, 2000 (yes, 9 years ago!) and remains open, and is disregarded by Firefox developers as little more than a trivial little annoyance, and therefore is and should be treated as a low-importance 'enhancement' rather than as the 'standard feature' it should really be.

If it weren't for the extension 'DragToScroll', which has been around for 3-1/2 years now (not updated in a long time but still works if you override version compatibility), I would have dumped Firefox and switched to a different browser (Maxthon 3, most likely).

So, I'm wondering (and hoping) -- does the new HTML5 drag and drop specification include anything that if implemented properly would make it possible to scroll & drag?
l.m.orchard
2009-07-22T13:57:42

@RenegadeX: Ugh. I don't work on the browser itself, but I noticed that unexpected behavior on drag & drop. The window really should scroll when you get toward the top or bottom while dragging - and, in fact, most pre-HTML5 JS frameworks do that in their own drag & drop abstractions.

Until / unless Firefox gets this fixed, the same sort of auto-scrolling could be hacked in to HTML5 drag and drop. Not perfect, but it's doable. That is, in the drag event, you can check if the mouse is near the upper or lower edge of the viewport - and if so, start scrolling appropriately. That's pretty much how the JS frameworks do it with old-school D&D
CopperBot
2009-07-23T21:41:07

Very cool article! Thanks for sharing. HTML5 really is going to change everything about how we use and develop for the web. Pretty awesome!
Alexos
2009-07-26T05:05:44

Hi, would DD work for uploading files? How can I display a progress bar? Thanks! I've been looking for that several days, and found only applets which I can't use in my proyect :-)
Sam
2009-07-28T13:54:26

Looks fantastic, going to try it out now (once I have upgraded my FF)
Dan Kantor
2009-07-30T03:10:25

Looks like effectAllowed and dropEffect are not working for FF 3.5 on the Mac. I see effects on Windows and Safari 4 on Mac. I've been playing around with adding borders to the drop target, but the + icon for copy really helps a lot. Hopefully Mozilla will fix this next update.
Ryan
2009-09-01T02:33:06

Great article it's good to see developments in this area. I wrote an article on the further extensions Firefox 3.6 has done with the dataTransfer method by adding the file attribute so you can access local files and upload them without the need for file inputs. http://www.thecssninja.com/javascript/drag-and-drop-upload
phil swenson
2009-09-03T22:58:36

Why not a first class WYSIWYG rich text editor with copy from clipboard image support?

NO ONE has built a decent text editor in JavaScript - At least not that I've seen. And browsers don't allow image paste for reasons I don't understand.

And by decent I mean just like you'd get in textmate or another editor. Have the standard text editor features everyone expects: tab, select indent/select unindent, resize image, home, end, duplicate line, delete line, styling, etc.

would push the browser to the next level in killing the desktop. non-techies hate wikis. They want a real editor. I do too actually.... would be great for forums, bug tracking system (imagine pasting screen shots in line with your bug desc), email, etc.
Jean-Denis
2009-09-04T00:23:39

Francisco Tomalski wrote up a nice piece on HTML 5 drag'n drop at http://www.alertdebugging.com/2009/08/16/on-html-5-drag-and-drop/

where he uncovers that the proposed standard is partially broken. Any comment on his piece?
Lisa
2009-09-09T07:33:08

This is a great move forward no more relying on 3rd party apps and extensions to play video or audio, i have been reading up on HTML 5 at the HTML 5 Tutorials website, i am now playing around with one of the free templates and was wondering how to embed audio, so thanks a lot, great information, lets hope more people lean towards HTML 5 and SOON!!!

The drag and drop feature i did not notice was already working in FF 3.5, i was told to get Safari to see HTML 5 in action. Thanks for a great post
Zoltan Hawryluk
2010-01-11T15:14:48

A million thank yous! This article was great introduction to HTML5 D+D. With it, I was able to extend it to other browsers. It was a little painful at first because the browser implementations diverge in significant, but manageable ways.

If you are interested, check out my article at http://www.useragentman.com/blog/2010/01/10/cross-browser-html5-drag-and-drop/ if you are interested in my results.

I noticed you haven't blogged in a while - I hope you haven't stopped totally and continue to share with the webdev community.
Raul
2011-04-26T23:56:02

Hi, Leslie. I've just come across a problem that's driving me nuts. I'm not fully comfortable with D&D, but managed to move a crosshairs image over a map to very precisely controlled positions. It worked great on FF 3.6 and FF 4. After a couple of days of successful testing, the image suddenly refused to de dropped after being dragged (it rather flew back to its previous position). Do you know if there is a bug in FF that might cause this?

BTW, during the programming process I updated Firebug, which also is getting a little cranky.

Thanks for your prompt answer, Raul
Alexander Plutov
2011-10-01T11:51:12

Good post about Drag & Drop http://plutov.by/post/html5_drag_and_drop

I (used to) like rev="canonical"

2009-04-13T05:05:50+00:00

Update 4/14: So, I liked rev="canonical", but I like the notion of pages offering sets of alternative URLs better. There are enough cracks in the case for rev="canonical" to stop caring about it and instead focus on the notion behind it. However it's expressed—is it rel="shortlink" now?—the final remaining things I'd like to see are:

An more generalized scope for alternate URL choices asserted by publishers, not just URL shortening. Other criteria beyond character length include ease of entry on mobile devices (eg. short, but also simple, maybe mostly numeric), ease of verbal mention (eg. billboards, postcards, etc).
HTTP headers are great where available—hooray for HEAD—but it still needs to be in the page for publishers who can't set custom headers.
Microformats are great, but I'd rather not parse a whole page to the footer to lift out the desired URLs.
Don't panic. Have fun.

And with that, I'm going to try coming up with other things to write about so this blog doesn't stay dormant. The rest of this entry remains unedited below...

~~So, like it says up there: I like rev="canonical".~~

Basically, it's a way of saying—instead of using that 3rd-party service you like for munging my URLs, use one of these pre-munged URLs I've provided. Ideally, the URLs I provide will be shorter for your needs, but my URLs won't be subject to potentially distasteful things I object to with respect to a service you might pick.

I like keeping control of URL spaces in the hands of their publishers. This also opens the field for more individual choice in URL shortening services, allowing more people to try their hand at building a better mouse trap—or allowing publishers to opt out of the mess altogether.

And, apropos of that, I tend to like rev="canonical" as the means of expressing this choice.

There's a lot more background on this whole shebang, but I'll just stick to my $0.02 on the discussion so far.

It's too alpha geeky to get adopted—also: NERDS!

Alpha geeks write plugins for web publishing systems used by less-alpha geeks, who eventually install web publishing systems for those even further down the geek scale. It's not unheard of for something incredibly nerdy to become relatively common, if it proves useful.

Feed auto-discovery is in blogging software and browsers now—it was once considered a somewhat arcane usage of the rel attribute on <link> tags in HTML <head>.

Everyone needs to implement their own URL shortener.

No, but everyone gets to pick a shortener for their own URLs—which could coincidentally be self-hosted or third-party. Isn't the web great?

The `rev` attribute is too hard to understand, people will get it wrong.

I hadn't paid much attention to the rev attribute until a week or so ago. I read the HTML spec on links again. Within 10 minutes of reading the spec, my understanding became this:

rel="{X}" — "this link means {X} in relation to the current page".
rev="{X}" — "this current page means {X} in relation to this link".

I may have misunderstood it—if so, explain to me how and I'll admit it's harder to understand than I think. There are harder concepts in the HTML 4 spec.

The `rev` attribute is removed in HTML5

That's too bad for HTML5.

The rationale given for the removal seems sensible enough. But, I'd say the rationale for removal is weakened if someone finds a use for the attribute and uses it correctly.

`rel="shorter"` / `rel="short"` is better and more explicit.

I like this idea in general, and I don't really care strongly enough about rev="canonical" vs rel="short" to make much noise beyond this blog post.

But, I prefer the semantics of rev="canonical", I don't think rel="short(er)" is better, and actually I think the less explicit assertion is a feature. I've not yet been convinced otherwise—but realistically, if either catches on, I'll probably use the most popular.

`rev="canonical"` doesn't mean "shorter URL"

That's okay—what if I don't want you to have shortened versions of my URLs? Doesn't fit in your tweet? Screw you. I didn't want you to link to me that badly anyway. Bah.

That, of course, is self-defeating. Conventional use of the attribute will probably yield shorter URLs out of self-interest. Furthermore, I could even offer a half-dozen URL variations, and you could use the string length function in the language of your choice to pick one on the basis of shortness. There's a lot of choice to go around here.

Also, feed auto-discovery links don't always lead to truly alternate versions of the current page—but instead to a useful set of items, many of which may currently appear somewhere on the page. So, even the "clear" semantics have some play in them as used in the wild.

A `rev="canonical"` HTTP header is better.

Hmm, maybe. It would certainly allow for possibly lighter-weight HEAD requests in determining the alternative URL for a given URL. But, I would expect it to fall apart in collections of static HTML pages baked by an offline script where web server configuration might not make setting custom headers easy.

Not everyone has the luxury / inclination to have a dynamic language capable of setting headers running in their web server.

`rev="canonical"` should appear on hyperlinks in the body of the page

I like microformats, in general. But, I don't really want to potentially parse a whole page to find the shorter URL that might be in the footer. With rev="canonical" in the <head>, I only have to parse so far before I find it or give up.

There's a lot of prior art on this with relation to feed autodiscovery—we used to mainly look for RSS feed URLs inside the page, too. It sucked.

Claiming to be canonical for another URL is trouble.

Okay, how about requiring a reciprocal rel="canonical" on the same page with rev="canonical".

Trust but verify—and only accept rev="canonical" where rel="canonical" matches the current URL and rev="canonical" yields a 301 redirect to the rel="canonical".

I can't see this verification process being necessarily more burdensome than using a 3rd-party shortener API—and that's if you're paranoid and building an index that needs some measure of resistance to gaming.

A single character slip from `rev="canonical"` to `rel="canonical"` could wreck your Googlejuice!

So, don't do that.

Maybe Googlejuice should be more resilient. Maybe it actually is—has anyone actually soured their juice yet with a mistake like this and lived to tell the tale?

Either way, characters mean things in computer languages, so make sure you use the right ones in the right order.

You guys are moving on this stuff too fast!

Welcome to 2002, when lots of us had more spare time than employment and we deployed new crap like this on our blogs and sites daily.

Back in those olden days, we manipulated raw HTML with our bare hands and deployed radioactive code using our teeth—all without benefit of protective change control. We probably all have cancer of the access logs now, but it was all in the name of Web Science!

Too long; didn't read

So, yeah. I like the idea behind rev="canonical" and I prefer its expression as rev="canonical" best—but the idea is the important thing. Let's get over the expression part fast and spend more time exploring the nuts and bolts and implications of the concept itself.

Archived Comments

http://pigsonthewing.org.uk/
2009-04-13T14:00:04

Forget rel="short" or rel="shorter". Try rel="shortcut".
l.m.orchard
2009-04-13T14:22:53

...or I could try rev="canonical" like I said up there. An actual rationale might be more productive here. :)
Dave Dash
2009-04-13T14:37:21

So yes, you are understanding "rev" correctly per the spec. The problem I have is this:

rel=canonical was introduced so when people went to various urls that a search engine would know that the url specified in rel=canonical was the real one. Let's do this to make life easier:

http://foo.com/A and http://foo.com/B

both go to the exact same page, and the publisher can't for some reason redirect to the canonicalized page (maybe it's amazon and they have stupid large urls with affiliate codes).

so if the former is canonical then this is present:

whether you go to /A or /B.

if they use rev="canonical":

Should only be shown if they are visitng /A not /B otherwise that rev statement implies that /A or /B are the canonical forms of "http://f.us/A".

I think that's why the rev attribute isn't a very good one. Honestly I liked the suggestion (you made it maybe) for using rel="shortcut alternative" or something.

Am I seeing potential ambiguity when there isn't?

-d
l.m.orchard
2009-04-13T15:36:42

@Dave: I think you may have used some markup in your comment that got filtered out - I'm not exactly sure what you mean. :/

If we have this URL:

http://example.com/this/url/is/too/long

With these links in the head:

rel="canonical" href="http://example.com/this/url/is/too/long" rev="canonical" href="http://ex.pl/ab"

Then, the URL http://ex.pl/ab will ideally lead to a 301 redirect. But, if not, it should have the same content as http://example.com/this/url/is/too/long — and therefore list the same reciprocal "canonical" relationship that leads back to the original page.

That seems disambiguated to me—even more so than rel="shorter" or whatnot. If the URL for a rel="shorter" link didn't yield a redirect, we'd never know what was the original page without a corresponding rel="canonical" anyway.
John Manoogian III (jm3)
2009-04-14T01:59:04

LOL at "soured their juice." Well written and well reasoned as usual, Les.
samj
2009-04-14T02:39:30

I've comprehensively rebutted your argument over there so I'm not going to do it again here, except to say that given you concede that rel=short* is just as good as rev=canonical, only it's a> not deprecated and b> cannot possibly make sites drop off the face of the Internet, why do you even care enough to write this blog post?

Sam
samj
2009-04-14T02:42:35

Oh, and let's not forget that you can only use this "solution" on the canonical URL itself since it implies that the page it is attached to is itself canonical. That means you're going to have to find another solution for a potentially infinite number of other links to the same content, or risk leaking google juice all over the place.

Sam
l.m.orchard
2009-04-14T03:51:48

why do you even care enough to write this blog post?

Because someone is wrong on the Internet! And I haven't blogged in awhile, so I needed an excuse.
Dave Dash
2009-04-14T04:15:57

Les,

I not good with words today. I was trying to say what Sam said:

"this “solution” on the canonical URL itself since it implies that the page it is attached to is itself canonical"

Hence, I'd stick with a solution that uses rel - so long as we can just pick an attribute "shorter alternative" or whatever that we can agree on :)

-d
Erik Vold
2009-04-21T03:47:01

Nice rebuttal to all of this rev=canonical opposition, but you gave in man wth?!

Let me try to back you up a bit, maybe you'll agree with your gut again.

First I must start with the removal of @rev in html 5. That was an amazingly dumb move, luckily it is a draft still is all I can say. Here is why I believe this: 1) 1 char diff argument: shut up you nub programmers 2) misuse: it's night vs day, black vs white, there is no gray area except that which your confused minds have created. The confusion -I think- usually comes in to play when considering a value, which is easy if you truly understand the diff between @rel and @rev. 3) @rev values can be represented as @rel values: sure I agree but this can not be true without adding more rel values then are necessary with a @rev which is already in the html 4 spec for a reason. If you remove something with use you cause confusion on multiple fronts: first people have to become aware of the change, then they need to understand why the change was made, then they need to make the changes to their code base, and finally when they want to express rev=canonical with a @rel value, debates over what best @rel value represents the equivalent of another @rel value if it were used in @rev occur..

I submit the debate which the opposition to rev=canonical are having over rel=short* as evidence that @rev is required. A link with rev=canonical that is short is a short url of the canonical url by definition, thus rel=short* is redundant for the use case of discovering a short url for the canonical url of some document. Albeit, I think rel=shorturl (or whatever is finally decided on) could be used by the publisher to indicate a preferred short url(s), if say there were >1 rev=canonical.

In an algorithm to discover a short url for a document one method could be to scan all links and find the shortest rev=canonical, or if a rel=shorturl is provided then use that and stop scanning immediately. My point is that rev=canonical should be used. Also rel=shorturl adds a marginal benefit to rev=canonical.

Another point, a rel=shorturl is rev=canonical by definition, thus if you were to use rel=shorturl and @rev were alive and well as it should be, you should automatically add rev=canonical (even though it is implied).

Another argument the opposition make to using rev=canonical is that the number of rev=canonical's are possibly infinite, so I say only list the rev=canonical's which a user may find of interest for the use cases you can imagine.

Another case for using rev=canonical is using rev=canonical with rel=mobile perhaps? so that a user can scan rev=canonical's for a rel=mobile url which is short enough to fit in the newest microblog gadget. Where a rel=mobile without rev=canonical could be the mobile site's homepage, even better for this link would be rel="mobile home" without rev=canonical.
Erik Vold
2009-04-21T05:17:14

The reasons why I like rev=canonical: http://erikvold.com/blog/index.cfm/2009/4/21/rev_canonical_good
l.m.orchard
2009-04-28T05:31:35

@Erik: Well, I "gave in" mainly because I wasn't really looking for a fight and had a vacation to attend to.

After that, the amount of hyperbolic stop energy being slung around rel=short{foo} has exhausted my care for the subject.

Enter the LizardFeeder

2009-01-06T00:01:54+00:00

[caption id="attachment_1582" align="alignright" width="247" caption="The Mozilla Tree"][/caption]

Behind Firefox is Mozilla, and behind Mozilla is a community. And the Mozilla community acts a lot like an ecosystem, which can be visualized as a kind of living tree—not to confused with the mozilla-central tree. Oh yeah, and Mozilla is the name of both a Foundation and a Corporation.

Confused yet? If not, then we should talk so you can explain it to me, because it all looks pretty tangly and intertwingled to me. Nonetheless, it seems to work, and produces a good chunk of my favorite software and technologies.

There are many efforts to track what's going on—including planets and newsletters and bugzillas and wikis and repositories and tinderboxen. Some of these resources report on, or are driven by, the activity occurring in the others. Some are automated, and others are carefully stitched together by hand. None offer a full picture of what's going on in the Mozilla galaxy in a way that's casually comprehensible by a sane human being.

Of course, that's not a slight against any of these sites or the people maintaining them—extracting an overview from such an organic phenomenon is neither easy nor straightforward. But, it might be fun to try.

As an infovore and avid practitioner of continuous partial attention, my first impulse is to reach for a firehose and stick my head into the stream. Relax, defocus, and try to let my pattern recognizers do their thing—sometimes those pattern recognizers are in my head, and sometimes they're written in Python.

[caption id="attachment_1585" align="alignright" width="225" caption="Firefox Victory!"][/caption]

But, for Mozilla, I couldn't find a stream of sufficient volume or completeness to satisfy me or my robots. Happily, though, my feeding urge found itself aligned with a project to discover the patterns of contribution in the Mozilla community and to find a way to thank the contributors responsible.

So, while we're still working on the thank-you angle, allow me to introduce you to the Lizardfeeder. The LizardFeeder is a feed aggregator, whose source code is built atop Sam Ruby's Planet Venus. The LizardFeeder pulls together and archives activity streams from a wide variety of Mozilla community sources. Beyond the usual human-readable pages produced by a blog-gathering Planet, the LizardFeeder accumulates statistical and historical data meant for consumption and analysis by robots.

At present, the only robot navigating the LizardFeeder archives is an AJAX-ified user interface that animates the firehose as a near real-time or time-lapsed stream of events scrolling by.

This is just meant as a conversation starter, though. I'm hoping to gather feedback and find more sources, as well as to entice creative community members to come up with more sophisticated visualizations of this data.

So, take a look, check it out, and let me know what you think!

Archived Comments

dria
2009-01-06T01:26:18

Is there a list of what sources are already being read by the LizardFeeder anywhere? I scanned through the various links here but didn't see anything obvious :)
l.m.orchard
2009-01-06T01:37:15

Yeah, that part could use some improvement. There's a monster list here:

http://feeds.mozilla.com/sources.opml
Brian King
2009-01-06T10:11:46

Excellent stuff. I was going to ask about access to the list of sources for each category, but Dria beat me to it.
l.m.orchard
2009-01-06T15:26:50

Actually, it occurs to me that this config file might work as a more readable version of the list of sources:

https://svn.mozilla.org/projects/lizardfeeder/trunk/conf/config.ini-dist

There's also this, which is where most of the previous list came from:

https://svn.mozilla.org/projects/lizardfeeder/trunk/conf/hg-feeds.opml-dist
Austin King
2009-01-06T20:55:52

Thanks for the background surrounding lizard feeder. Great post!

+1 Dria and Brian

Maybe make the title of the link to the OPML more descriptive than just "Feeds" and/or link to it in the body of the UI too and write something around it to encourage other's visualizations.

Awesome work Les.
Sanjay Parekh
2009-02-04T20:38:04

Is the AJAX UI available anywhere for download? I'd like to hack it for another use altogether. Great visualization and great application. Good job.
Deen Seth.
2009-09-16T18:21:02

This is a very good idea. Can we accomplish the same result using Yahoo Pipe? Do you plan to add events from Bugzilla, and mailing list to the feed?

I am more interested in development activities. There aren't much in code category.

I am interested in analyzing development related events.

An unnecessary Template Attribute Language

2008-11-01T19:23:23+00:00

A funny thing happened on the way to building a delayed real-time feed display: I got temporarily obsessed with implementing a template language in JavaScript that, as it turned out later, I didn't need. About the feed project itself, I hope to write more soon—but for now I want to get this extra template language thing out of my system and see if anyone else might have a use for it.

See, I had a notion it would be keen if I had access to the same template language on the client as on the server. I needed to render a number of list items statically on the server with feed entries, then update that list with new entries on the client as they became available through JSON feed polling. It'd be a pain in the butt to maintain two separate list item templates for client and server, so I reached for a shared template language.

Never mind that I'd just gotten done writing a small book on Dojo, and was already aware of the existence of the DojoX Django Template Language. This might've worked, since the server end of things was written in Python (though not with Django). That the JavaScript side already used jQuery wasn't too tall a hurdle. Also, I'm sure there are a handful of other JavaScript/Python template language match-ups to be found.

But, let's be honest here: I've always been a fan of Zope's Template Attribute Language for their Page Templates, and have long wondered how hard it would be to implement. The concept seems so much cleaner to me than most string-formatting template languages, and the workflow from mockup-to-template and back again has always been appealing to me when it works. So, when my first few experimental steps in trying my hand at it actually started working, I couldn't stop coding.

And now, the thing is mostly done. It has no tests, has features left undone, and probably yields plenty of bugs—but I finished it enough to use it practically, and that was long enough to convince me it wasn't the right tool for the job.

Still, though, I can't help thinking it might be the right tool for some job. That could be because I spent a lot of time on it, or that I'm unreasonably fond of TAL, but it still feels like a decent little plugin to have on hand. Maybe someone reading this will have a similar conclusion.

Oh and by the way, plain jQuery turned out to be a better tool for the job in question. This seems to nicely balance the duplicate effort between server and client, demanding only that I stick with semantically significant CSS class names in the server template—something I should be doing anyway:

        // Clone and populate a new entry.
        var new_item = $('#feed-items .entry:last')
            .clone()
            .attr('class', entry_classes.join(' ')) 
            .find('.group span')
                .text(tags['group'])
            .end()
            .find('.title')
                .find('.favicon')
                    .attr('src', favicon)
                .end()
                .find('.link')
                    .attr('href', entry.link)
                    .text(entry.title)
                .end()
            .end()
            .find('.updated')
                .find('.timeago')
                    .attr('title', entry.updated)
                    .text(entry_updated.strftime('%+'))
                    .timeago()
                .end()
                .find('.time')
                    .text(entry_updated.strftime('%I:%M %p'))
                .end()
            .end()
            .find('.author')
                .text(entry.author || 'n/a')
            .end()
            .prependTo('#feed-items')
            .hide();

Of course, plain is a relative term here.

Archived Comments

brad clements
2008-11-03T21:26:42

Groan,

Too bad I haven't had a chance to finish documenting (and tweaking) the ATALi project.

It's a collection of Alternative TAL Implementations.

It currently has TAL for xslt (server side using libxslt directly or via lxml with metal support)

and TAL for javascript (no libraries needed, though it does recognize mochikit iterators, no metal)

https://launchpad.net/atali
John Manoogian III (jm3)
2008-11-24T18:51:17

Have you seen "trimpath"?

http://code.google.com/p/trimpath/wiki/JavaScriptTemplates

Jelly Stains and Web Masons

2008-10-29T14:42:23+00:00

From Mark Bernstein's entry on Practical Prototype and script.aculo.us:

When chemists consult a volume about professional chemical technique, or when surgeons reach for the latest update on neuroanatomy, they can usually find a book that isn't couched in terms of silly examples and jokes. So can poets, mathematicians, and geologists. For some reason, though, it has become the accepted practice that language manuals should spend lots of time with silly, self-deprecating jokes, and that their example applications should be breakfast loggers and fantasy football leagues (or, conversely, payroll programs).

As an tech author with just a few books under my belt, Mark's take on Practical Prototype and script.aculo.us struck a bit of a sour note for me, because I'd like to work on making my tech writing more entertaining than not. I think that's a good thing, but I'm willing to be convinced otherwise.

I think the issue is that there are different meanings for "professional" when it comes to the web. There are web scientists and there are web masons. Web scientists pursue fundamentals and disambiguations, while web masons are busy building the next micro-site for the new product release from the almighty client. Many web scientists are also computer scientists and many web masons are also web scientists—but most web masons I've known come from creative liberal or fine arts backgrounds.

Though, for what it's worth, even amongst computer scientists there's still a tradition of leaving room for jelly stains and other oddities. This seems to be the sort of thing Mark acknowledges with dismay. ("It’s not fair to blame Mr. DuPont for the general vice.")

Is playfulness in literature just a computer science thing? I'm not a chemist; maybe chemists just don't like being funny in writing, or maybe their jokes are more subtle.

In any case, I think the "practical" genre of tech books is aimed at people who want to get something done, aren't interested in or have little time for context or background, yet wouldn't mind being entertained during the course of weekend tinkering and self-education.

So, a good book. But take out the jokes, trim back the sample code (or dispatch it to the Web site where it makes more sense), and give us to professional perspective, and everyone is going to be much happier.

The guidelines with which I'm familiar for tech books in the "practical" or similar genres include advice such as "show, don't tell". They also suggest that, although sample code should be made available online, the author should compose the book assuming that it's a standalone product. Web sites and CDROMs with code often vanish, but a bound book remains stable—which is especially useful on a cross-country flight without net access. Professional perspective is of course a desirable thing to work into the prose, but job #1 is to illustrate the right way to do things through running code.

How does Prototype+Javascript relate to other languages — C++/STL, say, or SELF? What, precisely, are the semantics of the key methods? I don't need the inevitable chapter 1 pitch for the wonderfulness of Javascript and the badness of MSIE, but it might be a good place for a quick summary for the pros. Call by reference • no pointers • primitives are objects • everything has a prototype slot • parens() do this, braces {} do that, brackets [] do something else • single and double quotes are different. Kernighan & Ritchie did this so well in C, and it’s not like we’re not familiar with their example.

I'd posit that most in the audience for "practical" tech books are entirely unfamiliar with Kernigan & Ritchie and haven't touched a line of C source code. Most of these readers have probably tumbled down the slope from HTML to CSS and finally to JavaScript in order to get something interesting to happen in a browser—and usually while under an unreasonable deadline.

I'd really be surprised if many readers have heard of Self or C++/STL or have much of a grounding at all in computer science or programming language fundamentals. Having these fundamentals would of course help web masons get a deeper understanding of the technologies that make the job possible, but the pragmatic rewards tend not to make up for the effort involved.

So, to sum up, the purpose for this entry isn't to beat up on Mark Bernstein. He's written a great deal of prose and code that I respect, so his opinion is interesting to me. Rather, I've tried to express my own understanding of this writing market, and hoping I've aimed at the right goals.

Archived Comments

Mark Bernstein
2008-10-29T16:51:59

I'd settle for a better class of joke!

On tech content, doesn't it annoy you that it's hard (or impossible) to find a book about a Web framework or language or facility that takes advantage of what you already know? One that doesn't assume you've never seen a language before?

I understand that publishers want the book to appeal to everyone. But it's not that the reader needs to know Kernighan: you know, right? So you know that it's entirely practical to tell a colleague pretty much everything she needs to know about a (famously tricky) new language in one chapter, starting from Kernighan's example 1 -- the original "Hello, world".
Aristotle Pagaltzis
2008-10-29T20:28:27

I think part of the difference is in the subjects themselves. Programming deals with pure thought stuff, yet at the same time attempts to model concepts from the physical reality. In contrast, science textbooks deal with intrinsically tangible stuff, while mathematics textbooks deal with the abstract for abstraction’s sake. The temptation to offset the potential dullness of pure thought stuff by using it to model goofy examples is obviously high.

Corroborating my theory would be the fact that elementary, low-abstraction maths books tend to bring up oddball examples too – eg. consider the sorts of things used to illustrate the rule of proportion.
Dave Dash
2008-10-30T00:53:54

Most books about web technology are meant to have a shelf life of a few years. They aren't reference guides, and they aren't meant to stand the test of time.

PHP, Python, Prototype, etc will all change.

Because of this, they can employ jokes and real-world examples. Jokes make reading the content enjoyable. Examples are step by step instructions to creating something useful.

If those aren't needed there's always reference guides.

I don't publish books, but I do write tutorials on my blog on occasion. One pattern I employ in my blog the "executive summary" pattern. I start with the take-away, the code snippet, the answer if you will. Then I spend the rest of the post (behind the fold, if you will) explaining the answer and interjecting my style.

I like doing this, because it's what I would want. I want to know the answer, and then I can read on for some backstory... rather than having to separate the wheat from the chaff.
Andrew Dupont
2008-10-31T03:30:21

I'm quite familiar with the trend Mark's talking about. And I agree that it shouldn't count against me or my book, but perhaps I'm a bit biased there.

Three things.

First: my book reads the way it reads because that's how I write. The process of writing the thing was torturous enough that I doubt I'd have finished it if I subjected every word to the scrutiny of a hypothetical skeptical reader. I'm not good enough at the craft for that. I think that each author has to find his best angle on the material and write that way. Because there are many books and many publishers, it tends to balance out.

Second: my book covers two JavaScript frameworks built to simplify common tasks. Basic familiarity with JavaScript is assumed. Right away that's a step away from the theoretical and toward the practical. I think tech books are best when they stick to one or the other, rather than try to drift between them.

Third: the book Mark wants has already been written: JavaScript: The Good Parts, by Doug Crockford. Doug's style is dry and academic, but also direct and substantive. If I tried to write in that style it'd be only a pale imitation.

Anyway... the "quirky" trend, I agree, can be tiring. As it applies to my book, the sticking point seems to be whether I'm actually funny. Readers of this weblog will simply have to buy the book to find out. It's several hundred pages long and has a fresh, earthy "new book" smell.
l.m.orchard
2008-10-31T03:47:37

@Mark: Yeah, these days my first impulse is not to reach for the latest book on a new technology like a web framework or new language. I usually head straight for the source code or start soaking in whatever documentation may be lying around on the web.

I suppose I could be a macho geek and say that's why I'm writing the books now—but to be honest I don't tend to get much out of practical programming books. They seem to sell, though, so I keep wondering if mine shouldn't be lighter or more fun, should I get the opportunity to produce more of them.

My first book was about half-and-half prose and code, with barely any screen shots to be found. I got to about equal parts code, prose, and screen grabs in my second—I thought that might make it easier reading.

The latest one doesn't have any jokes, but I tried to get straight to the core framework features and spotlight a few obscure magical bits. I worry if it might be a bit too dry.

But yeah, I do wish there were more (though maybe shorter) books that started from the assumption that I've got a CS degree and have tangled with a few languages and libraries so far. On the other hand, I do like having a sense for the author in a book—but that might be thanks to the fact that I read more blogs than books these days and enjoy the personal touches.
l.m.orchard
2008-10-31T04:05:01

@Andrew: Hey, thanks for dropping in for a comment! I can completely sympathize with needing to find a groove to handle how grueling producing one of these books can be—especially if you're more programmer than writer. (That's me, right now, anyway.) You need to get the best thing out there in the shortest amount of time, and find a way to get yourself to finish it.

At any rate, congrats on the new book, and I hope the cross-blog chatter sells a few more copies for you!

Writing a Delicious command for Ubiquity

2008-09-01T04:37:03+00:00

In my last post, I got all fluffy about how cool Ubiquity is but didn't share any code to prove the point. As it happens, I have come up with at least one useful command that I'm starting to use habitually in posting bookmarks to Delicious. You can subscribe to my command or check out the full source—this post will serve as a dissection of the thing. Since this will be fairly lengthy, follow along after the jump.

Oh, and it's been awhile since I posted something this in-depth around here, so feel free to let me know how this first draft works. And, bug reports and patches are of course welcome.

To begin, consider the following code starting off the command source code:

/**
 * share-on-delicious - an Ubiquity command for sharing bookmarks on
 * delicious.com
 *
 * l.m.orchard@pobox.com
 * http://decafbad.com/
 * Share and Enjoy!
 */
var uext = Application.extensions.get('ubiquity@labs.mozilla.com');

var cookie_mgr = Components.classes["@mozilla.org/cookiemanager;1"]
    .getService(Components.interfaces.nsICookieManager);

The first thing to note here is that a short header comment introduces the command. This isn't required, but it's a good idea. It's also something you can't really do with bookmarklets. On the other hand, Greasemonkey user scripts expect metadata about the script to be provided here, but Ubiquity doesn't use this convention.

Second, notice that the code accesses some chrome-level resources. Again, this is something unavailable to bookmarklets and Greasemonkey user scripts. Just take a look at the FUEL library documentation to get a quick sense of what's available using that simplified API, not to mention what's available using the lower-level browser APIs.

Now, check out this next chunk of code, which begins the construction of an Ubiquity command:

CmdUtils.CreateCommand({
    
    name:        
        'share-on-delicious',
    icon:
        'http://delicious.com/favicon.ico',
    description: 
        'Share the current page as a bookmark on delicious.com',
    help:        
        'Select text on the page to use as notes, or enter your own ' + 
        'text after the command word.  You can also assign tags to the '+ 
        'bookmark with the "tagged" modifier, and alter the bookmark ' + 
        'default page title with the "entitled" modifier.  Note that ' + 
        'you must also already be logged in at delicious.com to use ' +
        'this command.',

    homepage:   
        'http://decafbad.com',
    author: { 
        name: 'Leslie Michael Orchard', 
        email: 'l.m.orchard@pobox.com' 
    },
    license:
        'MPL/GPL/LGPL',

Whereas Greasemonkey scripts support metadata in the header comment, the Ubiquity command script API works a little differently.

The CmdUtils module provided by Ubiquity offers a CreateCommand function, which expects an object as a parameter. The object literal whose construction is begun in the code above serves as a self-contained package for the command, bearing metadata describing the command as well as containing all the code necessary to implement it.

So, in the above code block, you can see the machine-readable description of the command—including a command name, display icon, home page URL, author information, and license. The command name (share-on-delicious) will be used by the Ubiquity command parser, but the rest of the description will also be used in the list of commands available to the user, invoked by the command-list command, like so:

Moving along, this next chunk of code introduces the first functional bits of the command:

    takes: { notes: noun_arb_text },
    modifiers: { 
        tagged:  noun_arb_text,
        entitled: noun_arb_text
    },

Like smart keyword shortcut bookmarks, Ubiquity commands accept user-supplied input. But, what's unique to Ubiquity is that it employs a parser whose goal is to support something approximating natural language. At present, this results in commands that support a single primary argument—declared above with the takes property—and any number of additional keyword modifiers—declared above by the modifiers property.

For the command under construction here, this establishes a pattern something like the following:

share-on-delicious {notes} [tagged {tags} entitled {title}]

Content for the {notes} argument can either be typed directly by hand, or it can be supplied by text highlighted on the page. To use highlighted text, you can either issue the command alone, or use the word this for the {notes} argument before including further modifiers.

The modifiers tagged and entitled are optional, and can be used in any order. Each of these keywords signifies the start of a different argument—which unfortunately can collide with the literal data supplied for notes, which will hopefully be a rare occurrence.

All of this adds up command invocations including the following:

share-on-delicious
share-on-delicious I really like this page tagged nifty amusing
share-on-delicious this entitled This bookmark has no tags
sh this tagged osx software apple entitled This is good OS X software

That last example is important—since I have no other commands starting with "sh", I can abbreviate the full command. Ubiquity only requires enough of a command name to disambiguate it within your collection of commands.

Another thing to note is the use of the constant value noun_arb_text, which declares that these arguments should expect any arbitrary text as input.

This facility is not exploited for the present command, but Ubiquity defines noun types. These include concepts such as plain text, dates, address book contacts, browser tabs, bookmark tags, and more. You can define your own noun types, as well as implement suggestion schemes that help guide the user toward constructing useful input values in the command interface. You can read more about this in the official author tutorial.

Next up is a quick bit of command-specific configuration:

    /**
     * Command configuration settings.
     */
    _config: {
        // Base URL for the delicious v1 API
        api_base:      'https://api.del.icio.us',

        // Domain and name of the delicious login session cookie.
        cookie_domain: '.delicious.com',
        cookie_name:   '_user'
    },

Since this command will be posting to Delicious via the V1 API, it's handy to declare the base URL for the API in an easily changed spot. That way, you could change this value later on to point the command at another implementation of the API.

Additionally, this command will employ a little-known authentication trick supported by the Delicious API that accepts the user's login cookie set by the Delicious website—this "cookie god" auth is used by the official Delicious addon for Firefox. It's handy for piggybacking on the website login and removing the need to ask the user for their username and password again and possibly storing it in an insecure manner.

In fact, this next chunk of code defines a utility method to rummage through the cookie jar:

    /**
     * Dig up the Delicious login session cookie.
     */
    _getUserCookie: function() {
        var iter = cookie_mgr.enumerator;
        while (iter.hasMoreElements()) {
            var cookie = iter.getNext();
            if( cookie instanceof Components.interfaces.nsICookie && 
                cookie.host.indexOf(this._config.cookie_domain) != -1 && 
                cookie.name == this._config.cookie_name) {
                return decodeURIComponent(cookie.value);
            }
        }
    },

The method defined above, ._getUserCookie(), uses the browser's cookie manager and the values defined in the previous configuration section to find the login session cookie set for Delicious. Take note that this is far beyond the allowed capabilities of bookmarklets and Greasemoney user scripts—this is digging straight into the browser itself, skipping past the usual content-space security restrictions.

In other words: In Ubiquity, the gun is loaded and you should be careful.

Moving along, consider this next utility method:

    /**
     * Given input data and modifiers, attempt to assemble data necessary to
     * post a bookmark.
     */
    _extractBookmarkData: function(input_obj, mods) {
        return {
            _user:
                this._getUserCookie(),
            url:
                context.focusedWindow.location,
            description:
                mods.entitled.text || context.focusedWindow.document.title,
            extended: 
                input_obj.text,
            tags:
                mods.tagged.text
        };
    },

Named ._extractBookmarkData(), this utility method accepts the results of Ubiquity's parser interpreting the primary argument and modifier arguments supplied by the user. Using these data structures, it attempts to build a structure representing the fields of a Delicious bookmark.

The _user field is used for authentication via the site login cookie. The url is set from the location bar of the current page. The description, or title, field of the bookmark is taken from either the entitled modifier or the title of the current page. The tags, if any, come from the tagged modifier. And, finally, the extended notes for the bookmark are taken from the primary input argument of the command.

As you'll see shortly, this utility method will be used in both the preview and the execution of the command.

Next, there's one more utility method to cover:

    /**
     * Given an object, build a URL query string
     */
    _buildQueryString: function(data) {
        var qs = [];
        for (k in data) if (data[k]) 
            qs.push( encodeURIComponent(k) + '=' + 
                encodeURIComponent(data[k]) );
        return qs.join('&');
    },

In anticipation of using the Delicious V1 API, the ._buildQueryString() method accepts an object and constructs a URL query string from the encoded properties of the object. This will be paired with the ._extractBookmarkData() method to supply data for API calls.

Moving along, it's time to start digging into the meat of this Ubiquity command:

    /**
     * Present a preview of the bookmark under construction during the course
     * of composing the command.
     */
    preview: function(pblock, input_obj, mods) {

        var bm          = this._extractBookmarkData(input_obj, mods);
        var user_cookie = this._getUserCookie();
        var user_name   = (user_cookie) ? user_cookie.split(' ')[0] : '';

        var ns = { user_name: user_name, bm: bm };
        var tmpl;

With this code, the implementation of command method .preview() has begun. This method is used by Ubiquity to generate a live preview of the command. Called with a DOM node (pblock) and partially completed command input (input_obj and mods), this method is expected to build a representation of the command's results in the DOM node. As the user types, this method will be called over and over again, ideally offering feedback as the user composes a command.

Continuing on, consider this next chunk of code checking the validity of command input:

        if (!user_name) {

            // If there's no user name, there's no login, so this command won't work. 
            tmpl = [ 
                'No active user found - log in at ', 
                ' ',
                'delicious.com ', 
                'to use this command.'
            ].join('');

        } else if (!bm.description) {

            // If there's no title, then this is an error too.
            tmpl = [ 
                'A title is required for bookmarks on ', 
                ' ',
                'delicious.com ', 
                ''
            ].join('');

This chunk of code first checks for a user name, which can be extracted from a valid Delicious login cookie, if one was found. If not found, the command will fail—so the preview built here will instruct the user to login at Delicious before going further.

The second precondition for using the command is that the bookmark has been given a title. By default, this is the title of the current page—but, some pages don't offer titles. So, an error needs to be flagged if the user hasn't manually supplied a title in this case.

Finally, notice in both of these error cases, a string of HTML is composed in the variable tmpl. This will be used at the end of the method to populate the DOM node passed in as pblock.

Now, assuming that all the command's prerequisites have been met, it's time to try constructing a proper preview for the results of this command:

        } else {

            // Attempt to construct a vaguely delicious-esque preview of a bookmark.
            tmpl = [ 
                '',
                '',
                    'Share a bookmark at  ',
                        'delicious.com/${user_name}:',
                    '',
                        '',
                        '${bm.description}',
                        '${bm.url}',
                        bm.extended ? 
                            '${bm.extended}' : '',
                        bm.tags ?
                            'tags: ${bm.tags}' : '',
                    '',
                ''
            ].join("\n");

        }

        pblock.innerHTML = CmdUtils.renderTemplate(tmpl, ns);
    },

Building on the notion that previews are built in a DOM node, the code above uses both CSS and HTML to assemble a quick-and-dirty facsimile of a Delicious bookmark—which will be rendered like this:

Also note that Ubiquity provides a template engine for use in generating content—namely the JavaScript Templates engine from the TrimPath project. This engine may eventually be replaced with another, but the notion is that Ubiquity will provide tools to more easily generate previews and more.

The conclusion of the .preview() method uses the template engine with a call to CmdUtils.renderTemplate() to inject content into the preview element by way of the .innerHTML property.

Now that the preview is out of the way, it's time to get down to implementing the execution of the command:

    
    /**
     * Attempt to use the delicious v1 API to post a bookmark using the 
     * command input
     */
    execute: function(input_obj, mods) {
        var bm          = this._extractBookmarkData(input_obj, mods);
        var user_cookie = this._getUserCookie();
        var user_name   = (user_cookie) ? user_cookie.split(' ')[0] : '';

        if (!user_name) {
            // If there's no user name, there's no login, so this command won't work. 
            displayMessage('No active user found - log in at delicious.com ' +
                'to use this command.');
            return false;
        }

        if (!bm.description) {
            // If there's no title, somehow, then this is an error too.
            displayMessage("A title is required for bookmarks at delicious.com");
            return false;
        }

Mirroring the .preview() method, the .execute() method first checks for validity of the arguments given by the user. A missing user name or title result in a notification that the command has failed.

But, if the arguments are all valid, it's time to actually issue a request to the Delicious V1 API:

        var path = '/v1/posts/add';
        var url  = this._config.api_base + path;

        var req = Components.classes["@mozilla.org/xmlextras/xmlhttprequest;1"].
            createInstance();

        req.open('POST', url, true);

        req.onload = function(ev) { 
            displayMessage('Bookmark "' + bm.description + '" ' + 
                'shared at delicious.com/' + user_name);
        }

        req.onerror = function(ev) { 
            displayMessage('ERROR: Bookmark "' + bm.description + '" ' + 
                ' NOT shared on delicious.com/' + user_name);
        }

Using the base URL for the Delicious API declared earlier in the configuration section, the .execute() method constructs an API URL for the /v1/posts/add method. Then, it creates an instance of XMLHttpRequest from the browser to be used in sending the API request. Event handlers are registered with the object to present notifications to the user indicating whether or not the API request was successful.

At long last, it's time to wrap up this method and make the API request:

        req.setRequestHeader('Authorization', 'Basic Y29va2llOmNvb2tpZQ=='); // btoa('cookie:cookie')

        var mediator = Components.classes["@mozilla.org/appshell/window-mediator;1"].
            getService(Components.interfaces.nsIWindowMediator);
        var win = mediator.getMostRecentWindow(null);
        var user_agent = win.navigator.userAgent + ";Ubiquity-share-on-delicious";

        req.setRequestHeader("User-Agent", user_agent);      

        req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
        req.send(this._buildQueryString(bm));
    },

    EOF:null // I hate trailing commas
});

The login cookie authentication supported by the Delicious V1 API is triggered by supplying a user name / password pair of cookie, which is done by setting the Authorization request header. The login cookie is then expected to be passed in as the POST variable _user, which is done in the ._extractBookmarkData() utility method.

Another bit here that shows more access of browser resources is the construction of a unique User-Agent header for this API call based on the browser's own User-Agent string, something that's suggested in the guidelines for using the Delicious API.

Finally, the .execute() method—and the command itself—is wrapped up with by sending off the bookmark data encoded as POST form variables with the ._buildQueryString() utility method.

And, that's it—a command-driven Delicious browser extension in a little over 200 lines of code. There's still more to be done to really make this thing full-featured, but I think this shows off the basic features of Ubiquity. I'm hoping to dig in deeper and explore further, taking a look at running Greasemonkey-style code at browser startup and page load, as well as playing with some more browser chrome features.

Archived Comments

Jason Clark
2008-09-01T14:47:35
First off- fantastic post. Great to see a lengthy post here again, although I'm one to talk. This is an excellent introduction to Ubiquity command development, and tres useful to boot.

I'm wondering why you chose to construct and post the XMLHttpRequest manually instead of using jQuery, which is included with Ubiquity. I don't know that there's any benefit other than some simplicity, but I took a crack at converting your code to use jQuery, which works nicely. In the 'execute' function, replace everything after "var url = this._config.api_base + path;" with this (hope code blocks work in comments):

var win = context.focusedWindow; var user_agent = win.navigator.userAgent + ";Ubiquity-share-on-delicious";
```
    jQuery.ajax({
      type: "POST",
      url: url,
      data: this._buildQueryString(bm),
      username: "cookie",
      password: "cookie",
      beforeSend: function( req ) {
        req.setRequestHeader("User-Agent", user_agent); 
      },
      error: function() {
        displayMessage('ERROR: Bookmark "' + bm.description + '" ' + 
            ' NOT shared on delicious.com/' + user_name);
      },
      success: function() {
        displayMessage('Bookmark "' + bm.description + '" ' + 
            'shared at delicious.com/' + user_name);
      },
    });
```
Also, with both versions of the code, I'm seeing some unexpected behavior around authentication. Assume I'm logged in to delicious, with "stay logged in" checked, and I restart my browser. Trying to post with the command fails with a 401 unauthorized, even though I can see the cookie was sent (via Live HTTP Headers extension). Going to delicious.com shows me logged in, and once I've viewed the site, the command works. Except that now I can't reproduce; but I know it happened because I've got the headers. At any rate, it is working nicely, but the previous failure is bugging me... feel like I'm overlooking something.

Thanks again for an awesome post. Hope to see more of the same.
Ryan Scott Scheel
2008-09-01T15:07:01

You should be helping with the documentation, if you aren't already. Very nice job with this article;
Aza Raskin
2008-09-02T01:44:13

Hi Leslie,

This is a beautiful tutorial on writing a Ubiquity command. We'd love your help in making Ubiquity's documentation better (especially dev facing). You should totally link to this from the Ubiquity Wiki -- or even add the content in someway.

Anyway, just wanted to say thanks.

-- Aza
seth
2008-09-08T17:30:16

Am I doing something wrong?

When I enter in "sh this tagged tag1 tag2 entitled This is a title" everything after the tagged is added as a tag. It's not seeing the entitled part.
http://spyced.blogspot.com/
2008-09-15T19:12:58

Did something break? I'm getting a 404 accessing http://decafbad.com/hg/UbiquityCommands/file/tip/delicious.ubiq.js
l.m.orchard
2008-09-15T23:07:42

Yeah, looks like I had a small snafu with switching back from Lighttpd to Apache. Left out a rewrite rule - doh!
Tony
2008-10-22T04:56:03

Great article. This is replacing my delicious bookmarklet. Thanks!
dgtlchlk
2009-04-14T01:06:57

Great article and command. Wish it worked correctly with the latest 0.1.8 release though. No matter what text you put in it adds everything as the notes. The tagged and entitled modifiers don't work.
Nolan
2009-04-16T03:10:07

I second that comment. Delicious is actually saying the link given was "chrome://browser/content/browser.xul" and marking it as harmful inside delicious!
Justin
2009-07-06T12:39:39

Hi Leslie,

Thanks for the delicious ubiquity command. Unfortunately, as one of the commenters above mentions, the tagged modifier doesn't seem to work. I'm using Ubiquity 0.5 pre and typing the phrase:

share tagged foo

Adds the bookmark to Delicious with the note text "tagged foo"

Cheers, Justin
Eric
2009-07-20T22:43:17

Echoing others.. I'm using the latest ubiquity (err, next to latest.. 0.1.8?), and doing "share-on-delicious this is a note tagged testing" saves the link with notes "this is a note tagged testing" and no tags. :( Then again, the other delicious command I've found (by someone else) is having the same issue.

Ubiquity cracks open personal mashup tinkering

2008-08-31T04:07:22+00:00

When I was a wee hacker, I lived my digital life though a Commodore 64. I played games on it, did homework, talked to people far away—you know, all the stuff they showed in the pictures on the box. I also took things apart—both the machine itself and software running on it. I grew up learning that my digital environment was ultimately understandable, susceptible to tinkering, and open to being bent to my own purposes.

From the Commodore 64, I graduated eventually to terminals and text editors, opening portals mostly onto computers elsewhere via powerful UNIX command shells. And, of course, over the past decade, this has largely given way to life in a browser.

Yet, for a little while, particularly in the first few years of browsers, freedom to tinker seemed cramped. JavaScript had yet to arrive, and was a little messy when it did. There was no relatively easy addon development. And, though the portals opened by a browser were richer than those provided by terminals, the paths of navigation defined by links controlled by site owners offered less freedom of movement than UNIX commands. I could create my own pages, but I couldn't do much to others' pages.

But then, javascript: URLs came around, dots were connected, and bookmarklets were born. Suddenly, it was possible to customize my browsing environment with arbitrary JavaScript code having access to the current page—no matter whose page it was. And, through the various tricks of the AJAX trade, bookmarklets have only gotten more capable throughout the years.

Smart keyword shortcuts came around a little later, allowing quick access to bookmarks via simple keywords typed into the location bar. The smart part, though, came in the form of bookmarked URLs with placeholders and keywords given arguments to fill the placeholders—allowing not only quick access to bookmarked pages but also search engine forms bookmarked with late-bound fields.

Bookmarklets inherited the benefits of smart keyword shortcuts. The same placeholder in http: URLs can be inserted into the code of a javascript: URL, thus parameterizing the JavaScript code and incidentally turning the location bar into a kind of primitive command line. For example, one of my most heavily used "addresslets" is based on John Resig's original "Super-Fast Delicious Bookmarklet".

Another leap in prying open the browser tinkering space came in the form of Greasemonkey—an addon-powered environment created explicitly for the purpose of end-user scripting applied to others' pages. Greasemonkey user scripts can do more than bookmarklets, and with a much better development environment to boot. And, though a user script can't do quite as much as a proper browser addon, they're much easier to hack on and distribute.

Now, consider one of Mozilla Labs' newest projects, named Ubiquity. This rough and experimental addon for Firefox combines and improves upon everything I've described so far:

Ubiquity is a hackable command line environment, better than bookmarklets and smart keyword shortcuts;
Ubiquity enables persistent customization of others' pages, not unlike Greasemonkey;
Ubiquity facilitates live in-browser creation and web-based subscription to user commands and scripts;
Ubiquity gives access to browser chrome resources without a need for frequent restarts;

So far, most of the commands I see popping up since the 0.1 release have not accomplished much more than smart keyword shortcuts in the location bar could. But, it's early yet, and Ubiquity is far from limited to these commands.

Once the basics have been well-explored, I expect to see more people taking a crack at the broader capabilities offered by Ubiquity. Bookmarklets and Greasemonkey can't access browser chrome—but Ubiquity can. Ubiquity also offers a user interface that's so much more promising than keyword shortcuts, including command previews and typed parameters with suggestions.

Ubiquity promises web-wide mashups directed by a conversational command interface. All in all, the potential of this makes me feel like my digital environment—browser and web as a whole—is getting even more intimately, personally hackable.

It'll be very interesting to see where this project goes.

Archived Comments

Raul
2008-08-31T10:20:58

Hi, was using the original delicious command linked from the Ubiquity wiki, just tried yours and its definitely more polished and functional. Great job with the preview and the extra functionality. Only thing is 'share-to-delicious' is too much to type so I unsubscribed the previous command and changed the namespace in yours. This is clearly going to become a problems as the commands proliferate.
l.m.orchard
2008-08-31T14:43:39

@Raul: "share-to-delicious" is long, but keep in mind is that you only need to type enough of the command to disambiguate it. That is, all I type is "sh this tagged osx software apple" because I have no other commands starting with "sh". Watch the list of commands in the preview as you type. Those tell you what the parser thinks of what you're typing as you type. It's like automatic tab-completion.

Also, I think there's work planned to put some usage based sorting into the command parser, preferring the commands you use most in order of disambiguation.
Abi
2008-08-31T15:33:24

Nice post. I share your sentiment with regards to Ubiquity commands. A lot of commands that I see are just plain simple searches. I hope developers will work on more interesting things. For example, even things like auto-form filling for this comment (possibly even on page load without having to type a command) could be done by Ubiquity. There's still a lot more room for experimentation.
Abi
2008-08-31T15:45:59

But otherwise, I really like your command especially the preivew. We should include it as a builtin command, if you don't mind. :)
Abi
2008-08-31T15:51:20

I noticed a bug in your command. If I select some text in the awesomebar, the bookmark url is chrome. You should use something the command utils to get the url, instead.
l.m.orchard
2008-09-01T06:17:37

@Abi: Making this a built-in command is totally fine by me! It can use more work, though, for sure.

Also, I can reproduce that bug. Ugh. I can't find any methods in the CmdUtils to get the current page URL, though. I'll keep poking a bit though.
Abi
2008-09-01T06:50:23

Your blog seems to be rejecting code (that's why I had so many posts in the first place). You can get the current page url using (with dots):

CmdUtils getDocumentInsecure() location href
Blair McBride
2008-09-02T02:50:12

I recommend against using getDocumentInsecure() - its got "Insecure" in its name for a reason! Instead, you should use:

Application.activeWindow.activeTab.uri.spec

date-based pagination

2008-07-17T18:16:01+00:00

Here's a small idea I've not yet had the chance to try out on a large scale: Time-based pagination in lieu of page-number-based pagination for personal content - ie. blogs, bookmarks, status updates, etc. (You know, User Generated Content except I dislike the term.)

Page numbers change over time, while time-based URLs are stable. Most people don't generate an unreasonable amount of stuff in a day, so a page-per-day might not be so bad. But, if there's too much stuff, degenerate to a page-per-12-hours or whatnot.

Delicious used to have something like this in the web UI, back in the mists of 2003 or so, but Joshua got rid of it after a few design iterations. The Delicious API is still somewhat based on it, which causes some confusion—but I tend to like it, thus this post.

Archived Comments

fluffy
2008-07-17T20:22:38

You mean, like the date-based archives that Movable Type has had for ages?

Yeah, I like those.
Neil Kandalgaonkar
2008-07-18T19:44:42

I've had the same idea for Upcoming, for obvious reasons... our SRP pagination would then be permalinks.
Anil Dash
2008-07-25T17:11:40

Actually, yeah, we built the default pagination for archives in MT4 to be category-by-month, for exactly this reason. I regularly see "?page=2" in URLs and it just seems like such an obvious, avoidable mistake for so many to have embraced. SEO aside, permalinks that are destined to expire over time seem pretty reader-hostile, too.

Queue everything and delight everyone

2008-07-04T19:17:18+00:00

This is a blog post I've had simmering in my brainmeats for well over a year or two. I'm suddenly inspired to break blog-radio-silence and get it out of my head.

From Let the microblogs bloom - RussellBeattie.com:

Once this is widely accepted (and I'm sure there are many that would argue with me), the thing that will separate these types of services won't be whether they stay up (ala Twitter), but how fast your subscription messages are updated. Some services might be smaller or offer more features but not update as quickly whereas others will pride themselves on being as close to real-time as possible. The key is that it's all about messaging, not publishing. (Oh, and this also facilitates federation as well, but that's another topic).

One of the problems it seems most modern web apps face is the tendency to want to do everything all at once, and all in the same code that responds directly to a user. Because, while you're in there building a user interface, it's easy to implement everything else that needs to happen in that same UI module or library.

Someone wants to post a bookmark? Someone wants to post a message? Well, of course you want the system to cross-reference and deliver that new piece of User Generated Content through every permutation of tag, recipient, keyword, and notification channel supported by your system.

But, do you really have to do everything all at once—while the person who generated that content is tapping his or her foot, waiting for the web interface to respond with feedback? Are all of these things immediately vital to the person watching the browser spin, right now?

No. Your user wants to get on with things. He or she wants to see the submitted content get accepted and, as feedback and confirmation, see it reflected in a personal view immediately. Does it matter—to this person, at this moment—whether it shows up simultaneously in a friend's inbox, the public timeline, a global tag page, or even an RSS or Atom feed?

Again, no, simultaneity doesn't really matter—because no human beings actually appreciate it. Instead, imagine a ripple effect of concentric social and attention contexts with associated people spreading out from the original submission. (This probably rates the creation of a diagram someday.)

To make the person who's submitting something happy, offer feedback visible in their own personal context in under 50-200 milliseconds. (That is, less than half-a-second at worst, in people terms.)
The next person to delight is someone following the first person's published content—and humanly speaking, delays of tens of thousands of milliseconds can be acceptable here. (That is, 1-10 seconds at worst, in people terms.)
Finally, you can start worrying about strangers, allowing the content to propagate to tag pages, keyword tracking pages, and other public views—and I'd assert that delays of hundreds of thousands of milliseconds are acceptable here. (That is, 1-2 minutes at worst, in people terms.)

The idea here is that the social structure can help you scale, while still delighting people. Even with these delays, the system is still better at getting the word out than the original content creator would be at notifying all the others involved with an out-of-band system like IM or email. And that's at worst—on most good days, all the delays should tend to be on the order of seconds or less.

And how do you do all of this? Use queues. Sure, the original submission of content can and should be done all at once—just enough to get the content into the user's collection. Then, queue a job for further processing and get out of the way. In fact, just queue one job from the user interface—the processor of that queue can then queue further jobs for all the other individual processing tasks that are likely susceptible to plenty of parallel processing and horizontal scaling.

Meanwhile, the original user creating content has been thanked for their submission and life goes on. In fact, their life may include going on to submit many more pieces of content in rapid succession, thanks to your delightfully responsive web user interface.

And, in the end, that's really the purpose of a web-based content creation interface—accepting something as quickly as possible to make the user happy enough to continue submitting more. The other part of the user interface, retrieval, serves simply to get the original content distributed as fast as can be reasonably expected.

Now, preparing for fast retrieval is another story. The flip side to processing queues are message inboxes—expect content duplicated everywhere and fetched simply, rather than using cleverly expressed SQL joins that bring a system to its knees. But, that's another post altogether. :)

Archived Comments

shyam
2008-07-04T21:12:46

Interesting bits of conversation happening all over the place regarding queues and the irony of it all - the much maligned Java has had workqueues since the early days.

What everyone will learn, rather painfully in in cases like Twitter, is that all data is not created, consumed or processed equally. If you write your system which treats data equally you'll wind up with many Twitters all over the place.
l.m.orchard
2008-07-04T21:37:30

Oh, definitely. Work queues are not a new thing at all. It's just that I think there're a lot of modern web app builders who skipped Java "enterprise" software—skipped, or hoped to run away—and are rediscovering the whole set of problems. Maybe the solutions will be less over-engineered this time.
Mike Warot
2008-07-04T21:40:15

The thing that drives me nuts about twitter is that the core data rate is only about 30k/second... yet it kept going down. It's easy to spit out a broadcast to a subnet and never even miss a packet if there are only 100 of them per second or so. There's no reason on god's green earth that twitter should be anywhere near overloaded.

Bad architecture, on the other hand, is the work of Satan. ;-)
Simon Willison
2008-07-04T21:40:39

Thank you! I've been trying to put my thumb on why queues are so interesting for months; this expresses it perfectly.
Gabe Wachob
2008-07-04T22:54:10

Usually I would have something serious to say in agreement with you, because I do so much agree with you.

But I have just one comment:

DUH!!!
Don Park
2008-07-06T05:16:32

Good suggestions. For social services like Twitter, I would also add one more item:

Prioritize by Relationship

For example, two-way Twitter relationships (mutual-follow or recent @ or direct message exchange) should be refreshed before one-way. One can go further by placing higher priority on users whom poster sent messages to or received from within past X-hours.
alexis
2008-07-10T12:35:25

er... eventually consistent social graphs anyone?
Robert Schultz
2008-07-10T13:36:22

I agree with everything you've said. Especially the last part, duplicating data in the format it will be retrieved in rather than using complicated and CPU intensive SQL queries. This is especially true for any sort of statistics or reporting. I learned this by seeing my website's statistics growing slower and slower to retrieve as more and more traffic caused the database to become larger and larger and all of a sudden those queries that ran nearly instantly, even with good indexing were taking several seconds to return.
citric
2008-07-10T21:49:19

Web apps doing things while the user waits unnecessarily is an old phenomenon. I think it's often a matter of developers not wanting to (and/or being politically unable to) venture into what they consider the sysadmin's domain. Take the way-too-common case of apps that make the client wait while it does housekeeping. Why isn't this in a cron job? One reason is maybe this is KewlOSSBlogWikiPackage and it's simpler to say "just untar the package under htdocs and you're done" instead of saying "also, unpack these scripts in a non-servable area and set them up to run hourly, but not all at the same time; stagger them a little. And run them with the same UID your web server is running as". But we end up with a lot of apps that (badly) reimplement basic tools their OS ships with in the first place.
Bill Seitz
2008-07-28T21:23:06

I wonder if you're still setting the bar too high for low-priority connections. I mean, microblogging isn't really messaging, and maybe isn't (shouldn't-be?) conversation.

So why wouldn't 10-15min be good enough?

What % of "messages" are read instantly after they hit an inbox?
Jay Yukes
2008-11-23T10:35:55

I had to solve a similar problem. Needed the fastest possible response, so had to rule out interacting with the Database directly from the web app. Used PHP message queue Dropr to defer all DB work. It is very fast, easily over 1000 messages/second
jmxz
2009-02-13T20:00:13

Wow those comments make me feel old. I remember when these java queues everyone's referring to reminded me of how I had a VAX dedicated to queuing and scheduling batch jobs for a Cray.
Chris Henry
2009-06-11T20:09:47

Handling load is probably one of the biggest problems facing websites today. Queueing is definitely the way to go, but like you said, sites need the type of architecture where it's easy to deploy services to different machines. Usually by the time the site is under load, it's too late...
maht
2010-05-04T14:50:05

I use Inotify as my queue messaging system

http://maht0x0r.blogspot.com/2009/06/serialising-multiple-writers-in-shell_20.html

Inotify can wait on MOVED_TO or CLOSE_WRITE events so that you can add them to the queue when the upload has finished.

It should also be noted that this is a mnethod of load balacing too. Instead of 1000 parallel thumbnails being produced all context switching away, you can determine how many processes get spawned, use the OS' resource managing features etc.

OPML reading lists in FeedMagick2

2007-10-17T07:22:47+00:00

For anyone who's interested: I've been hacking a little bit on FeedMagick2 again, with the latest addition being an OPML reading list feed blender.

What's an OPML reading list? Basically, it's the same as as OPML export of a feed reader's subscription list - only rather than doing a one-time import into another program, the OPML is itself treated as a live feed. A feed reader that supports OPML reading lists will continually check the list for updates and sync RSS/Atom feed subscriptions with its contents, maybe in a special sub-folder.

Here's a quick demo:

The itch I mean to eventually scratch is to replace the front page of decafbad.com with a live updating aggregation of the stuff I create and capture daily on the web. It'll be basically a self-assembling tumblelog pulled from many different services across the web. It'll also replace the footer of accumulated crud I've got on this very blog - which I thought was a good idea at one point, but now consider NeatLikeDigitalWatches.

With that in mind, the next thing I plan to develop is an hAtom module or XSL transform. This will turn the blended feed into an XHTML page. Maybe someday, hAtomic will launch, and I'll have a nice pretty style for the page too. Some time after that, I might work up a module that stows away dated historical archives of the feed and pages. I have further plans and ideas, but I'm trying to focus on the itchy spots first so that I might actually get something done in this round of serial enthusiasm.

0xDECAFBAD - Tag: webdev

Putting clouds in boxes for webdevs at Mozilla

Sighting storm clouds on the horizon

Sharing recipes for fluffy clouds

No assembly required

So, now what?

Moved to Jekyll and Disqus

Archived Comments

Firefox Sync server on Google App Engine

Archived Comments

Case Study: Building a Bookmark Management UI for Mozilla's BYOB

Using Tinderbox to write articles for the web

HTML5 drag and drop in Firefox 3.5

The New Drag and Drop Events

Using dataTransfer

Using Drag Feedback Images

Using Drop Effects

Conclusion

Archived Comments

I (used to) like rev="canonical"

It's too alpha geeky to get adopted—also: NERDS!

Everyone needs to implement their own URL shortener.

The rev attribute is too hard to understand, people will get it wrong.

The rev attribute is removed in HTML5

rel="shorter" / rel="short" is better and more explicit.

rev="canonical" doesn't mean "shorter URL"

A rev="canonical" HTTP header is better.

rev="canonical" should appear on hyperlinks in the body of the page

Claiming to be canonical for another URL is trouble.

A single character slip from rev="canonical" to rel="canonical" could wreck your Googlejuice!

You guys are moving on this stuff too fast!

Too long; didn't read

Archived Comments

Enter the LizardFeeder

Archived Comments

An unnecessary Template Attribute Language

Archived Comments

Jelly Stains and Web Masons

Archived Comments

Writing a Delicious command for Ubiquity

Archived Comments

Ubiquity cracks open personal mashup tinkering

Archived Comments

date-based pagination

Archived Comments

Queue everything and delight everyone

Archived Comments

OPML reading lists in FeedMagick2

The `rev` attribute is too hard to understand, people will get it wrong.

The `rev` attribute is removed in HTML5

`rel="shorter"` / `rel="short"` is better and more explicit.

`rev="canonical"` doesn't mean "shorter URL"

A `rev="canonical"` HTTP header is better.

`rev="canonical"` should appear on hyperlinks in the body of the page

A single character slip from `rev="canonical"` to `rel="canonical"` could wreck your Googlejuice!