0xDECAFBAD - Tag: delicious

Let a million bookmarks bloom

2010-12-19T21:26:38+00:00

TL;DR: Don't depend on Delicious; host your own, pay for it elsewhere, or hope for the best. Use real-time feeds to stitch the bookmarking diaspora back together into topical aggregate indexes.

In the last entry, I wrote about why my use of Delicious has dropped over time, and what I've been missing as a result. In short, I think there's place for something like Delicious, and I'd like to get back to it.

Update: See also, "We can save Delicious, but probably not in the way you think" from Stephen Hood, one of my former Delicious co-workers. He covers some more good angles I didn't get around to here.

Active thoughts

So, it's true Yahoo! announced that they're "actively thinking about the future of Delicious" and that "we believe there is a home outside the company". But, none of that says they've found a home or that it will be easy to transplant if they do. In the meantime, there's a lot of uncertainty and probably only an overworked skeleton crew left onboard.

Much of Delicious now depends on home-grown technology that is shared by many other Yahoo! properties. This should not be a surprise: Beyond the money, that was kind of the point of selling to Yahoo! in particular. As a major search company in 2005, they presumably had the indexing mojo help scale it up. So, transferring the site to a new technology stack will definitely figure into the value proposition of anyone looking to take over Delicious.

In short, any buyers of Delicious will have their hands full—probably more so than Yahoo! did when they first bought it. Don't hold your breath for a bright future.

Who cares?

What it comes down to this:

If you use Delicious regularly, you care about your bookmarks.
If you're part of an interest group, you care about their bookmarks and they about yours.
If you're a random web surfer, you care about finding things relevant to your interests.
If you're Yahoo!—well, I would have thought you cared about building a human-powered index of the web, but who knows where your collective heads are at these days.

The important thing is that no owners of Delicious will care in particular about #1 and #2—that is, your bookmarks and those of your colleagues. If they care about anything, it's the overall value of a well-indexed set of web resources to #3 and how they can monetize that.

In other words: If they can't figure out how to monetize the aggregate index, don't expect them to start caring about your data. See also: If you're not paying for it; you're the product.

What now?

Find someone else to host your bookmarks, maybe you yourself. Think about helping move and/or hosting your colleagues' collections. In particular, look for someone whose motivations are aligned with your own—by way of either money or common interest.

There are plenty of services stepping in to take over from Delicious. I like pinboard.in, because it's very close to the original Delicious, will capture links from other modern services, and it charges you money for valuable features.

There are also a few self-hosted substitutes, and there will more in the weeks to come. Some take these weekend projects as fodder to discount Delicious in general, but they're missing the point: The basic features of Delicious were never particularly complex—it's scaling those features up to web scale that's hard in a single service.

But, you may not need web scale. Also, hosting is getting cheaper every day. Gamers host servers for themselves and their friends, so why not start thinking about hosting a social bookmarking server for interesting people by invite? If you don't have the technical chops yourself, you probably know someone who does.

If this kerfluffle with Delicious does nothing else, it should really get us thinking about hosting our own stuff again. Cheap hosting is not just for startups.

And after that?

I'd love to see a million social bookmarking sites bloom, both as new public services and self-hosted sites for friends. What this diaspora would lack, though, is the awesomeness of seeing everything tagged "webdev" and who else bookmarks things you like. What made Delicious hard is what made it interesting, and it's possible that no one service will ever again reach a critical mass in social bookmarking.

But, I don't think Delicious could have scaled up forever and stayed interesting. To address this, I advocated breaking Delicious up into semi-porous cells around loosely defined interest groups. Basically, a lot like what Reddit does with subreddits and what the Stack Exchange network does with Q&A topics.

As it was, Delicious had grown up around an implicit interest group of webdevs, programmers, tech heads, and the like. So, I figured that spawning more cells would be more interesting than just shoving endless people into the existing one. I also thought it could help scale the site in terms of server architecture, since cells of relevance could be fairly independent.

The problem with a delicious diaspora, though, is the opposite: Scattered sites will offer fragments of larger communities that may not find each other on their own. So, we might like to find a way to piece them back together.

Luckily, I think we have the technology to rebuild Delicious at web scale.

A tasty future, maybe?

Assume that lots of people have tagged bookmark collections in lots of places. What if those sites all supported PubSubHubBub and/or RSS Cloud across feeds by dimensions of tag, user, and URL?

I think you'd have the makings of a loosely coupled and distributed Delicious:

The motivation to maintain individual and group bookmark collections could match up better with those who care about them. You could use pinboard.in. You could host your own bookmarks. You could be a minor hero and host those of the people you find interesting.

Then, imagine a layer of search engines, each specializing in collecting a topically-interesting mix of real-time feeds from an array of social bookmarking sites. A future Google could subscribe to hubs pings from all the above and build something interesting and valuable without hosting the bookmarks of pesky, demanding users.

I don't think this is super-science rocket surgery. Oddly enough, this is close to how I hoped we'd restructure Delicious back in the 2.0 days. It's what I was getting at with my "Queue everything, delight everyone" post. I wanted to see us treat individual users' collections as isolated from the aggregate indexes, connected only by messages and queues. I'm not sure we ended up with something as de-coupled as that, but I didn't have enough visibility into the deep backend to say for sure. Either way, I think this could be an opportunity to address things at web scale without a single company running the show.

In conclusion

Use the web. Host your own, pay for it, or find someone who values your data.

You don't need a central blog server to have a Google. And you don't need a Delicious to have social bookmarking. It seems like we're drifting away from this as services like Facebook and Twitter rise to swallow whole categories of web application in monolithic silos.

Seriously, hosting is getting cheaper and easier. Use it and host your own. Yeah, cheap is not free, but is what you're doing worthless to you? There are interesting alternatives that don't involve handing everything over to someone who doesn't actually care about you and your stuff.

And, if it turns out that social bookmarking settles down into a small but highly involved niche of curators, so much the better that they keep their own lights on.

Archived Comments

Rogers Cadenhead
2010-12-20T05:44:29

Hosting is cheap.

Running web applications on your own server securely and reliably is expensive, assuming you value your own time.
JB
2010-12-20T10:05:58

I really like this article. There should indeed be a system where you can self-host your bookmarks with some CMS and a centralized site for less techy people. All of these bookmarks would be somehow merged to produce intelligent data. Kind of what Automattic does with WordPress.org and WordPress.com.

Aside, this new Delicious system should have a more "sharing" side with ability for networkers to comment on each bookmark, "redeli" the bookmark (like retweet), etc... Unite bookmark and twitter functions - What Yahoo! should have try to developp during these past five years and failed to. I already tried to find some link posted in the past in my Twitter stream, and it made me really angry and frustrated. And I didn't succeed. Twitter is nice but still has a too bad signal-to-noise ratio : Tweets are way too much passing, impossible to organized, and replies are not enough differentiated from normal tweet.

What could be an ideal way to go is a WordPress plugin. Based on already there custom post type system, tags system, comments sytem... WordPress is already a successful product, widely used. It would make easier the process of leaving current doomed delicious to a decentralized system.

And Automattic should integrate this plugin to their WordPress.com blogs. They even would not have to buy Delicious to Yahoo!, just spread the word and invite people to subscribe to their service and import their delicious data.

Finally, imagine that Automattic creates a subdomain of wordpress.com where all the data (bookmarks and data) would be gathered. The plugin on self hosted blogs would propose to the owner to share his personal bookmarks data with this website via an API. A network could be based on wordpress.com optional subscription.

I don't have any personal or business interest in Automattic (just a WordPress user) but it seems the perfect way to go.
l.m.orchard
2010-12-20T16:49:26

@Rogers: Yeah, I totally glossed over the horrors of sysadmin work and security. I'm kind of bipolar that way. I'm very optimistic about the stuff you can do with all this cheap capacity. But, then, I get my 5th XSS bug of the day to fix and want to change careers.

I tend to think that cheap/disposable VMs and things like Google App Engine can make it easier, though. You can pay for & run just enough to run your app, hopefully keep the maintenance needs down and minimize the attack surface.

I don't think it's that bad for small self-hosted sites like I'm talking about, but then I have a high threshold for crap.
Bill Seitz
2010-12-20T20:09:41
I'm not feeling that optimistic about things like GoogleAppEngine these days.
1. How much is GAE mattering to anyone these days? What does that imply for its long-term survival?
2. WikiLeaks/TOS etc.
l.m.orchard
2010-12-21T02:26:18

@Bill: Well, GAE is mattering to me insofar as it's a decent place to deploy quick python web apps. Of course, that might be bad, since quick python apps aren't really taking advantage of the whole BigTable thing.

As for wikileaks, well... Somebody can always cut you off
Evan Prodromou
2010-12-22T17:55:13

I'd love to get your comments on the Bookmark extension we're working on for StatusNet. It uses OStatus (http://ostatus.org/) for distributing bookmarks (that's PuSH + Activity Streams + Salmon) and the Bookmark object type for AS.

It's shaping up nicely; I'm going to set up a sample site for testing v v soon.

Tags do work (for me, at least)

2009-01-18T06:29:34+00:00

For the "too long; didn't read" crowd:

I've been using a lot of tags on Delicious over a relatively long time, so they seem very useful to me.
Delicious encourages the use of tags through UI convention and tool usage patterns, whereas Flickr presents no particular bias toward collecting tags from users.
Since title and description attract more contribution effort from users on Flickr than on Delicious, it's natural that search over those fields will be more productive than for tags.
Search on Delicious doesn't have access to the complete text of the bookmarked resource, and often tags will contain information missing from the supplied title or description.
All told, tags on Delicious are more essential than tags on Flickr.
In conclusion, I think Do Tags Work? misses the value of tags, as I know them, by focusing on Flickr.

Of course, I don't really care what this means for folksonomy and the rest of Web 2.0—tags work for me on Delicious. So, I suspect this means I'm not entirely opposed to the sentiment in Do Tags Work?, because I don't think tags work everywhere their use is attempted.

The rest of this entry elaborates on the above.

From Cathy Marshall's Do Tags Work?:

Have I convinced you that tags aren't all they've cracked up to be? I hope I have, but nonetheless there's a lingering fascination. Surely there's something to be done about tags: we don't want to just turn up our noses at Mr. Weinberger's argument.

From Mark Bernstein's In TEKKA: Tags:

The study could be repeated on more images, and in additional contexts. It might be bad luck. But if it’s a repeatable result — and I think we all know it is — then we’re going to have to rethink a lot of our Web 2.0 rhetoric. Folksonomy is an illusion.

From my bookmark on Delicious:

If you can get past the rambling paragraphs of awkward fun-poking at tags interspersed with library science / web 2.0 / cultural references—as well as a discovery of what, you know, Flickr is all about—there's a well-embellished and obsessively-assembled statistical analysis of tags vs title vs notes in finding photos featuring tourist heel-spinning on the testicles of a bull mosaic in Milan. My impression is that she's missed the point of tags, but I'm having trouble reducing the impression to a critique.

It might be that I'm intimidated by the writing in Do Tags Work? or by the fact that a lot of work was done to produce numbers I can't dispute. Either way, I have neither the time to produce a counter-study nor the wherewithal to extract and refute points in that article with precision. So, I'm just going to throw my impressions out there and hope they coalesce.

Tags are extremely useful to me.
- I've been using tags on Delicious since shortly after the site was created.
- To date, I've saved 11825 bookmarks and spawned 6069 tags. (More on that ratio later.)
- Tags have worked very well for me, both in later finding my own saved resources and in finding aggregates of others'.
Flickr is not the poster child of tagging—Delicious is.
- On Flickr, I've found tags occasionally interesting, but not incredibly so. More a curiosity than a killer feature, really. (More on that in a bit, too.)
- To be honest, I've not found a great deal of use for tags around Web 2.0 properties beyond Delicious, folksonomy hype notwithstanding.
- Fun fact: When I worked on Delicious, we disliked the term "folksonomy" and tried never to use it in serious discussion.
The places where I've found tags useful are text-heavy, where the full text is either not indexed or not displayed in results when using search.
- Consider magazine articles or blog posts bookmarked on Delicious.
- Those bookmarks are subject to the character limits of title or description fields.
- Far more often than on Flickr, there is information in tags not present in a bookmark's title or description.
On Delicious, the use of bookmarklets and extensions is the primary source of new bookmarks.
- These tools tend to produce a title based on the original bookmarked page and, when present, a description summarized from the page contents by way of highlighted text.
- With this in mind, consider that tags often make up the sole intentional contribution made to the bookmark by the person saving it.
  - Beyond the choice to save the bookmark in the first place, that is.
- Whereas on Flickr, the effort is spent on title and, sometimes, the description. Beyond that point, tags are an afterthought, at best.
The user interface of tagging is near effortless versus, say, formal taxonomies or nested folders.
- Even the much-debated choice of the big, fat space bar as a delimiter supports minimal effort.
- Minimal effort lowers mental cost, which promotes more noise, but also encourages more input (or any input) and thus more signal overall.
- People have limited attention span budgets from which useful metadata can be solicited. Tagging lowers the budget requirement.
The reason why I have generated a ratio of 1 new tag for every 2 bookmarks is that I use tags in a way akin to free association, made possible by the low cost of tagging.
- This is a feature, not a bug.
- I have many, many tags used only once and never again—some of them are, in fact, jokes.
- But, I have generated some tags that I've come to adopt more heavily, most of which proved surprisingly useful over months or years of recurrence.
- As for those tags I've never used again: They cost nothing, can be hidden below visibility thresholds, and can be merged later into more appropriate tags if I care to do so.

And, there's where I run out of steam. For the conclusion, go back to the beginning of this entry. Biased though I might be, I think Delicious is the place to study tagging, and there they're of great use. This is where I think Do Tags Work? misses the value, despite the volume of writing and data.

Archived Comments

kellan
2009-01-18T07:40:08

In addition Flickr's tagging implementation has always been non-optimal because there is a 1-to-1-to-1 relationship of object (photo) - tag - tagger. To call tagging on Flickr a folksonomy is to fundamental miss why the term was coined to describe del.icio.us (we can argue about whether we like the term). Tags in del.icio.us act as votes (among other things), much like links in PageRank, and the mish-mash of curatorial activity on Flickr (faving/commenting/groups).

Which is a long way of saying I agree! (and we're thinking about how to fix tags on Flickr, but thats been true as long as I've been there)
Mark Bernstein
2009-01-18T14:17:50

It seems to me that it ought to be straightforward to repeat this study of delicious.

If I follow you, you find delicious tags useful primarily as a personal mnemonic -- a way for finding your own stuff. That's an valuable application, but not the one addressed here (or by folksonomists); Marshall is exploring whether tags are useful for helping the community organize a collection for collaborative use. I haven't heard much about using delicious this way, but perhaps it could be done.
karl
2009-01-18T18:30:05

note that she is saying that tags are good at finding your own personal stuff, and that you confirm this in your blog post. So at least you are in agreement on this.

But let's dig a bit further what is she saying and what you are saying. I have too tagged a lot of stuff with another service similar to del.icio.us. The result is that no, it is not useful to find something quickly, and I would even go further that I don't even find my own stuff. What I do though on a regular basis is to reduce my own stuff by changing the tags for a set a of pages for example if there are économie, economie, economy, business, I might finally reduce it to one tag "économie". What I would really wish is that I could do a hierarchy of these. For example sometimes I tag Japan and Tokyo, what I wish is that in my own hierarchy when I tag Tokyo, the system assumed Japan.

On Flickr it really depends on what we are looking for, but I'm using flickr for finding images for specific location and for specific creative commons licenses. In both cases, these metadata depends on an ontology (taxonomy, structured schemas, $ORGANIZATIONSYSTEM)

geolocation… the ontology is the map, a well defined system where you can pinpoint an image somewhere and get the real lat/long values.

licenses… a controlled vocabulary of different type of licenses.

The debate is not about "do tags suck" but do they work? And indeed they are not really efficient compared to other methods. It's cool they exist, but they should not be here to justify that ontology sucks either. Every community which is in a need of achieving a specific repetitive task comes one day or the other to structured data organization.
Ethan
2009-01-19T02:33:33

Very often, the suggested tags on a new Delicious post inspire my personal informational ontology. Folksonomy is real and valuable, but like any public resource, it requires some effort on my end to make it worth my while. Flickr's tagging system is valuable to me, but not as crucial to the overall experience. This is in part because Flickr's thumbnail images are a faster way to free-associatively group like concepts together than verbal tags ever could be.

Your readers might enjoy an analysis of how I use Delicious.
l.m.orchard
2009-01-19T02:37:53

@Kellan: That's definitely another thing - Flickr has plenty of explicit curatorial features that suck up the energy otherwise directed toward tagging—the only game in town on Delicious.

That's not a bad thing - those are good features that wouldn't necessarily work for Delicious. And if they would work for Delicious, someone somewhere is probably emulating them with specially-formed tags to varying degrees of success. Eventually we might have discovered those uses and wrapped a feature around them. (see also: for:* and system:media:*)

Actually, one area where I think Flickr has done better with tagging is in machine tags. There was some disagreement about that even on the Delicious team—pitting it against simplicity of tags—but I thought there was something there worth exploring.
l.m.orchard
2009-01-19T02:41:29

@Mark: Not primarily as a personal mnemonic—I probably should have expanded more on the social aspects. I find tags on Delicious just as useful for finding others' bookmarks, too.

I assume that many others use tags much like I do—especially the people bookmarking things I find interesting. On Delicious, something akin to lightweight communities emerge based around bottom-up tags.

I am no library scientist—but, if you've got a community with a collection to organize, I don't think you really need tags. That community probably already has a notion of categories worth extracting in a more formal top-down process.

As for repeating the study, I probably won't be the person to do it. But, for a quick example that I'd suggest is par for the course, take a look at the bookmarks for the article in question.

Of the 18 bookmarks, 10 have notes. That page only shows the most common title, but I'd be surprised if any of the 18 have a title other than "Do Tags Work?" That leaves the tags. Of the top 10 tags, only one appears in the title ("tags"). The word "folksonomy" appears in none of the descriptions. There's also "socialsoftware" used 3 times, which appears neither in title or description, nor in the original article itself. I think this shows better use of tags than the example used from Flickr.
l.m.orchard
2009-01-19T03:12:10

@karl: I wrote that tags are useful for finding both mine and others' bookmarks. Meaning tags work as intended, at least for me on Delicious. I suspect they work for others too, since they keep tagging things in ways that I keep finding them.

You wrote about another service, but I didn't—I'm writing about tags on Delicious. I don't know what the thing they call tags looks like on that other unnamed service.

This is part of the reason behind my mild disdain for the term "folksonomy": Delicious didn't introduce tagging to start a best practice for Web 2.0 at large—tagging was introduced as a site feature for Delicious. Attempts to clone that feature elsewhere have been uneven in results, despite having had a term coined and a body of hype spun around it.

Tagging is like a salt water fish that lots of people thought was pretty and they started trying to stick in fresh water tanks. I don't think it thrives everywhere people have tried to stick it and not everyone who's tried to clone tagging has gotten all the important parts right.

On Flickr, I've asserted that tags aren't that useful. Machine tags like geo tags are a big exception though. Otherwise, they have a wealth of other tools for finding things beyond tags, whereas on Delicious they're the only game in town.

I guess my main point is this: In my experience, tags work. On Delicious. Where they're largely agreed to have been cloned from. Elsewhere, I think there have been transcription errors and misinterpretations that give the concept a dubious reputation.

I'm not sure what you mean about tags being not really efficient compared to other methods. Tags are a way to get people to provide metadata useful in emergent classification that would otherwise have not been supplied at all. I think they're efficient at that.

Many people just punt when faced with sorting a new resource into a formal category or folder structure. I suppose that filters out all but the conscientious, leaving better quality results. Is that what you mean?
Daniel Tunkelang
2009-01-19T06:07:24

My colleagues and I at Endeca learned a lot from out experience with the ACM Digital Library, a collection of author-tagged computer science articles. As is, the author-supplied tags weren't all that useful. But we were able to derive a vocabulary from them that we then used for automatic tagging. We also applied a variant of this approach to the website for a leading sports programming network.

You can learn more details about our approach in the proceedings of the 2008 Workshop on Human Computer Information Retrieval:

http://research.microsoft.com/en-us/um/people/ryenw/hcir2008/
Britta
2009-01-19T06:35:48

Heh, so much of this depends on personal use. Like we always tell people, there's no wrong way to tag, and that makes it really really hard to generalize accurately about What Tags Mean.

I tend to use Delicious tags to associate a bookmark with a thread of my thinking (which is very useful), and I use a combination of search and tags to find things. But on Flickr, I like looking through tagged photos, and I like looking through groups even more. I search to find old photos, often finding a picture because of the tags on the photo. Both Flickr and Delicious work very well for me.

Tags are interface; the "meaning" of each tag is de-emphasized. Compared to hierarchical structures, they're easier, messier, more-used, more fun. They're flexible - they're personal.

Tags on Delicious are collective; tags on Flickr are collective (except when you tag another person's photos); folksonomy implies collaborative.
Britta
2009-01-19T06:39:57

P.S. I rewrote pretty much the entire Wikipedia article on tags a few months ago - http://en.wikipedia.org/wiki/Tag_(metadata) - which was fun. If you disagree with me about tagging, just look at the official article! Ho ho ho.
Paul Mison
2009-01-19T13:17:34

Just to throw a minor spanner in Les's argument in comment #5, I do have a slightly different title: I put the site name after a pipe for every bookmark I make, partly because it might make it easier to distinguish posts from different sources at some point (without recourse to URL or tag) and partly because it looked good when Dan Hill did it (I nicked the idea, as with so many things.) I don't think this really invalidates his point, though: tags are the place for (most) people to put their mark on a link. Us description-editors are odd.

(Annoyingly, I can't find a link that shows that well. There's my bookmarks tagged 'tagging', I suppose. I've only just noticed this has related tags- that's a nice touch I'd never really spotted.)

On a side note, I love machine tags in Flickr, mainly because they're hidden from the user by default. (They've got a lot more useful now that there are specialised methods in the API, too.) I mused on Twitter last week that neither Delicious nor a third party has done anything with the loose convention of via: or cite: tags that some people use. One day...
Karl
2009-01-19T17:52:28
I said tags were not very useful to me in the sense they don't help me to find interesting stuff (even on delicious) :) Your mileage may vary. Understood.

About categories and structured hierarchies, I never mentionned to have to pick up one unique category. My goal is to be able to have hierarchieS. You could imagine a system where one tags "chinese"
1. meant a human chinese person
2. or meant the written language.
Structures are useful. As I said, I'm not against tags, I'm just for a better use of tags. I believe that plugging SKOS would be a very easy way to leverage the power of tags by letting people doing their own personal ontologies if they need it. :)

So the summary instead of a flat system, I want a hierarchical system, a graph without destroying the flat system.
l.m.orchard
2009-01-19T18:13:37

@karl: I'd say once you introduce intentional structure or hierarchy, you no longer have tags. I think you really want categories. You can emulate categories using tags, but you'll be going against the grain.

Tags are about throwing things at the wall and letting analysis and algorithms help surface implicit structures or relations.

And, yes, the semantic issue of "chinese" the language vs "chinese" the food vs "chinese" the people is a long standing one. Single tags don't solve this and tagging in general kind of shrugs on the issue—though tag intersections involving many tags help disambiguate intended meaning. (ie. chinese+food, chinese+language)
My tag cloud
2009-01-19T20:31:25

I didn't thoroughly read this post or the link, but I'm going to guess that the value of tagging depends on the tags you use. For instance, I wouldn't tag this with "tags" but instead with "tagging debate", "tagging meta", or similar. That would allow a visitor to this site to find every similar discussion. The "tags" tag might include a lot of things that someone wasn't looking for, such as a comparison of tagging services.

This could also be tagged with the name of the person who wrote the link so the visitors could find discussions involving that person.

For how I do things, my tag cloud is at my name's link.

Note the tags with [name topic], such as "obama cabinet". Anyone who sees that should realize right away what it involves. And, someone looking for specific Obama information can scan the tags starting with his name for subtopics. Note also the tags about themes, such as "mean spirited", which refers to an "argument" some people make. And, click a tag like "bill richardson" to see how I make some sense of large numbers of posts: many of the lists with a large number of posts are preceded by a summary linking to individual posts.

(Note: that doesn't represent all the content at my site because I switched from MT to Drupal a couple months ago and only a fraction of thousands of posts is tagged. There are also a few hundred tags that aren't shown; I need to figure out how to show lots of tags.)

Improving my Delicious command for Ubiquity

2008-09-07T05:20:20+00:00

After writing up my first stab at a Delicious command for Ubiquity, I planned to continue revising it based on feedback and to work on exploring more of what Ubiquity enables. I started looking into writing my own nouns for tag suggestions, as well as playing with page load and browser startup hooks. And, I also started poking at a little bit of deeper extension development, which took up most of my time today.

I've updated my UbiquityCommands page and checked in my latest revision of the Delicious command.

The main new feature is a status bar item reporting bookmarks for the current page:

As you can see above, the command now comes with a status bar panel powered by the Delicious URL info JSON feed, providing bookmarking info on every page visited. It shows a bookmark count, a tooltip with further information, and sends the user to the URL info page on Delicious when clicked. It mostly works, but it could use some looking at. This is my first time really cracking open the hood on Firefox and XUL, and so I'm feeling around in the dark.

Specifically, I'm using Ubiquity's page load hook—but I'm also trying to augment that by tracking tab selection events, in order to keep the status bar info updated for the active tab. But then, that leads me to trying to track new windows, to attach the tab selection event handler for every newly opened window. Or I could just be barking up the wrong tree entirely. At any rate, the code is probably brain-dead dumb, so I hope someone can clue me into a better way.

Archived Comments

Gen Kanai
2008-09-07T08:10:39

Les, you may want to let R/RW know about your new command.

http://www.readwriteweb.com/archives/the_ultimate_list_of_custom_ubiquity_verbs.php
Abimanyu Raja
2008-09-07T08:55:52

I disagree... Your command doesn't fit on that list. It's way too good for that list. :)
Jeton
2008-09-07T21:56:43

Well, i was having problems understanding the "tagged" and "entitled". Didn't knew it was without brackets....

But i gotta say, it actually replaces the delicious firefox extension for its easy way of posting links on delicious.

Well done, and seeing that it's only the begining i don't doubt that it will improve much more!

Oh, and i had a WTF when i first saw that delicous icon on the status bar of Firefox. Didn't knew it was from your command ;)

Thanks for making this.
Jeton
2008-09-07T22:20:09

One question: Is there a chance to remove the quotation marks (i meant the same on the above comment as well, not brackets) when saving a note about a bookmark?

Everytime i bookmark a page via this command, it quotes the note.
l.m.orchard
2008-09-07T22:41:20
@Jeton: I guess the quoted thing is something I do by habit, to discriminate between what's my comment and what's summary straight from the page. One way you can force it unquoted is with the "noted" modifier. So, select some text on the page, and you can do something like:
```
sha noted this tagged foo bar baz
```
That should use the selected text without quotes. Also, you can just type arbitrary stuff:
```
sha noted These are notes entitled A title goes here tagged foo bar baz
```
l.m.orchard
2008-09-07T22:44:59

@Jeton: Actually, I just made a check-in that flips it around. There's now a "quoted" modifier to wrap the notes in quotes, leaving it unquoted otherwise. I might be the weirdo here in my quote usage :)
Jack Mottram
2008-09-09T10:04:12

I'm a weirdo too, and now I can't quite work out how to get quotes around stuff I'm quoting from a page. I assumed this would work:

sha this quoted

but it just adds the word 'quoted' to the end of the quoted text, without quotes.

Similarly:

sha quoted this tagged test

adds the words 'tagged' and 'test' to the end of the quoted text, though this time everything _is_ in quotes. (So it seems I can get quotes, but lose the ability to tag in the process.)

sha this quoted text

puts the word 'text' in quotes, but drops the selected text...

So, I give up - what's the secret to getting quotes around quoted text?!
Jack Mottram
2008-09-09T10:11:59

And of course I worked it out immediately after leaving that comment:

sha tagged test monkey whatever quoted this

works just fine (but feels the wrong way around to be tagging before quoting). And now I can't work out how to quote and note, so to speak. I tried adding

noted These are my comments on what I just quoted

but it doesn't work, it puts 'noted These are my &c.' inside the quotes...

(I should say, despite all this moaning, I really have been enjoying using the command - it's very slick compared to the official add-on.)
l.m.orchard
2008-09-09T14:30:06
Huh, weird - that's exactly how I use the command:
```
sha quoted this tagged foo bar baz
```
One major difference, though, is that I'm using a bleeding edge checkout of the Ubiquity extension. I wonder if there's a bug in the parser for the last release. :(

The definition of the modifiers is pretty simple, so there's not a whole lot of debugging I can do in the command itself. Ugh.

Oh, and also: RIght now it's either plain notes or quoted notes. Presence of the quoted modifier clobbers plain notes. But, how that I think about it I wonder if I should make those work together.
Frank
2008-09-09T17:40:39

Hello,

I was wondering if there was any chance you could put together a post annotating/explaining the changes/additions to your improved Delicious command. In particular, some explanations of the code dealing with the statusbar would be helpful for us beginners. Thanks.

Frank
Tony
2008-10-22T05:12:52

Not a big fan of things adding stuff my status bar. :/ Ubiquity commands should just focus on Ubiquity, not XUL changes in Firefox.
l.m.orchard
2008-10-29T16:04:28

@Tony: Actually, commands are just one of the goals of Ubiquity. Easier XUL hacking, ala Greasemonkey, is another goal, albeit under-explored so far.
http://dy-verse.blogspot.com/
2009-04-27T14:50:43

Here is a share-to-delicious ubiquity command that automatically generates tags using YAHOO pipes. It also uses the text selected on the page as notes and the page title as the title of the article.

http://gist.github.com/26425
l.m.orchard
2009-04-28T05:25:58

Auto-generating tags via Yahoo Pipes is a neat trick, but it's unfortunately very much frowned upon by delicious. Not that anything negative will happen to you, but it defeats the purpose of tagging altogether.

Try searching for "Lazy Sheep Bookmarklet" and possibly read:

http://www.mail-archive.com/ydn-delicious@yahoogroups.com/msg00853.html

In particular - "People add metadata so that 1) they can find things and 2) other people can find things. You are removing the step in which people add the tag metadata, thus making the system less valuable to themselves and others."

The key word repeated there is "people"

Writing a Delicious command for Ubiquity

2008-09-01T04:37:03+00:00

In my last post, I got all fluffy about how cool Ubiquity is but didn't share any code to prove the point. As it happens, I have come up with at least one useful command that I'm starting to use habitually in posting bookmarks to Delicious. You can subscribe to my command or check out the full source—this post will serve as a dissection of the thing. Since this will be fairly lengthy, follow along after the jump.

Oh, and it's been awhile since I posted something this in-depth around here, so feel free to let me know how this first draft works. And, bug reports and patches are of course welcome.

To begin, consider the following code starting off the command source code:

/**
 * share-on-delicious - an Ubiquity command for sharing bookmarks on
 * delicious.com
 *
 * l.m.orchard@pobox.com
 * http://decafbad.com/
 * Share and Enjoy!
 */
var uext = Application.extensions.get('ubiquity@labs.mozilla.com');

var cookie_mgr = Components.classes["@mozilla.org/cookiemanager;1"]
    .getService(Components.interfaces.nsICookieManager);

The first thing to note here is that a short header comment introduces the command. This isn't required, but it's a good idea. It's also something you can't really do with bookmarklets. On the other hand, Greasemonkey user scripts expect metadata about the script to be provided here, but Ubiquity doesn't use this convention.

Second, notice that the code accesses some chrome-level resources. Again, this is something unavailable to bookmarklets and Greasemonkey user scripts. Just take a look at the FUEL library documentation to get a quick sense of what's available using that simplified API, not to mention what's available using the lower-level browser APIs.

Now, check out this next chunk of code, which begins the construction of an Ubiquity command:

CmdUtils.CreateCommand({
    
    name:        
        'share-on-delicious',
    icon:
        'http://delicious.com/favicon.ico',
    description: 
        'Share the current page as a bookmark on delicious.com',
    help:        
        'Select text on the page to use as notes, or enter your own ' + 
        'text after the command word.  You can also assign tags to the '+ 
        'bookmark with the "tagged" modifier, and alter the bookmark ' + 
        'default page title with the "entitled" modifier.  Note that ' + 
        'you must also already be logged in at delicious.com to use ' +
        'this command.',

    homepage:   
        'http://decafbad.com',
    author: { 
        name: 'Leslie Michael Orchard', 
        email: 'l.m.orchard@pobox.com' 
    },
    license:
        'MPL/GPL/LGPL',

Whereas Greasemonkey scripts support metadata in the header comment, the Ubiquity command script API works a little differently.

The CmdUtils module provided by Ubiquity offers a CreateCommand function, which expects an object as a parameter. The object literal whose construction is begun in the code above serves as a self-contained package for the command, bearing metadata describing the command as well as containing all the code necessary to implement it.

So, in the above code block, you can see the machine-readable description of the command—including a command name, display icon, home page URL, author information, and license. The command name (share-on-delicious) will be used by the Ubiquity command parser, but the rest of the description will also be used in the list of commands available to the user, invoked by the command-list command, like so:

Moving along, this next chunk of code introduces the first functional bits of the command:

    takes: { notes: noun_arb_text },
    modifiers: { 
        tagged:  noun_arb_text,
        entitled: noun_arb_text
    },

Like smart keyword shortcut bookmarks, Ubiquity commands accept user-supplied input. But, what's unique to Ubiquity is that it employs a parser whose goal is to support something approximating natural language. At present, this results in commands that support a single primary argument—declared above with the takes property—and any number of additional keyword modifiers—declared above by the modifiers property.

For the command under construction here, this establishes a pattern something like the following:

share-on-delicious {notes} [tagged {tags} entitled {title}]

Content for the {notes} argument can either be typed directly by hand, or it can be supplied by text highlighted on the page. To use highlighted text, you can either issue the command alone, or use the word this for the {notes} argument before including further modifiers.

The modifiers tagged and entitled are optional, and can be used in any order. Each of these keywords signifies the start of a different argument—which unfortunately can collide with the literal data supplied for notes, which will hopefully be a rare occurrence.

All of this adds up command invocations including the following:

share-on-delicious
share-on-delicious I really like this page tagged nifty amusing
share-on-delicious this entitled This bookmark has no tags
sh this tagged osx software apple entitled This is good OS X software

That last example is important—since I have no other commands starting with "sh", I can abbreviate the full command. Ubiquity only requires enough of a command name to disambiguate it within your collection of commands.

Another thing to note is the use of the constant value noun_arb_text, which declares that these arguments should expect any arbitrary text as input.

This facility is not exploited for the present command, but Ubiquity defines noun types. These include concepts such as plain text, dates, address book contacts, browser tabs, bookmark tags, and more. You can define your own noun types, as well as implement suggestion schemes that help guide the user toward constructing useful input values in the command interface. You can read more about this in the official author tutorial.

Next up is a quick bit of command-specific configuration:

    /**
     * Command configuration settings.
     */
    _config: {
        // Base URL for the delicious v1 API
        api_base:      'https://api.del.icio.us',

        // Domain and name of the delicious login session cookie.
        cookie_domain: '.delicious.com',
        cookie_name:   '_user'
    },

Since this command will be posting to Delicious via the V1 API, it's handy to declare the base URL for the API in an easily changed spot. That way, you could change this value later on to point the command at another implementation of the API.

Additionally, this command will employ a little-known authentication trick supported by the Delicious API that accepts the user's login cookie set by the Delicious website—this "cookie god" auth is used by the official Delicious addon for Firefox. It's handy for piggybacking on the website login and removing the need to ask the user for their username and password again and possibly storing it in an insecure manner.

In fact, this next chunk of code defines a utility method to rummage through the cookie jar:

    /**
     * Dig up the Delicious login session cookie.
     */
    _getUserCookie: function() {
        var iter = cookie_mgr.enumerator;
        while (iter.hasMoreElements()) {
            var cookie = iter.getNext();
            if( cookie instanceof Components.interfaces.nsICookie && 
                cookie.host.indexOf(this._config.cookie_domain) != -1 && 
                cookie.name == this._config.cookie_name) {
                return decodeURIComponent(cookie.value);
            }
        }
    },

The method defined above, ._getUserCookie(), uses the browser's cookie manager and the values defined in the previous configuration section to find the login session cookie set for Delicious. Take note that this is far beyond the allowed capabilities of bookmarklets and Greasemoney user scripts—this is digging straight into the browser itself, skipping past the usual content-space security restrictions.

In other words: In Ubiquity, the gun is loaded and you should be careful.

Moving along, consider this next utility method:

    /**
     * Given input data and modifiers, attempt to assemble data necessary to
     * post a bookmark.
     */
    _extractBookmarkData: function(input_obj, mods) {
        return {
            _user:
                this._getUserCookie(),
            url:
                context.focusedWindow.location,
            description:
                mods.entitled.text || context.focusedWindow.document.title,
            extended: 
                input_obj.text,
            tags:
                mods.tagged.text
        };
    },

Named ._extractBookmarkData(), this utility method accepts the results of Ubiquity's parser interpreting the primary argument and modifier arguments supplied by the user. Using these data structures, it attempts to build a structure representing the fields of a Delicious bookmark.

The _user field is used for authentication via the site login cookie. The url is set from the location bar of the current page. The description, or title, field of the bookmark is taken from either the entitled modifier or the title of the current page. The tags, if any, come from the tagged modifier. And, finally, the extended notes for the bookmark are taken from the primary input argument of the command.

As you'll see shortly, this utility method will be used in both the preview and the execution of the command.

Next, there's one more utility method to cover:

    /**
     * Given an object, build a URL query string
     */
    _buildQueryString: function(data) {
        var qs = [];
        for (k in data) if (data[k]) 
            qs.push( encodeURIComponent(k) + '=' + 
                encodeURIComponent(data[k]) );
        return qs.join('&');
    },

In anticipation of using the Delicious V1 API, the ._buildQueryString() method accepts an object and constructs a URL query string from the encoded properties of the object. This will be paired with the ._extractBookmarkData() method to supply data for API calls.

Moving along, it's time to start digging into the meat of this Ubiquity command:

    /**
     * Present a preview of the bookmark under construction during the course
     * of composing the command.
     */
    preview: function(pblock, input_obj, mods) {

        var bm          = this._extractBookmarkData(input_obj, mods);
        var user_cookie = this._getUserCookie();
        var user_name   = (user_cookie) ? user_cookie.split(' ')[0] : '';

        var ns = { user_name: user_name, bm: bm };
        var tmpl;

With this code, the implementation of command method .preview() has begun. This method is used by Ubiquity to generate a live preview of the command. Called with a DOM node (pblock) and partially completed command input (input_obj and mods), this method is expected to build a representation of the command's results in the DOM node. As the user types, this method will be called over and over again, ideally offering feedback as the user composes a command.

Continuing on, consider this next chunk of code checking the validity of command input:

        if (!user_name) {

            // If there's no user name, there's no login, so this command won't work. 
            tmpl = [ 
                'No active user found - log in at ', 
                ' ',
                'delicious.com ', 
                'to use this command.'
            ].join('');

        } else if (!bm.description) {

            // If there's no title, then this is an error too.
            tmpl = [ 
                'A title is required for bookmarks on ', 
                ' ',
                'delicious.com ', 
                ''
            ].join('');

This chunk of code first checks for a user name, which can be extracted from a valid Delicious login cookie, if one was found. If not found, the command will fail—so the preview built here will instruct the user to login at Delicious before going further.

The second precondition for using the command is that the bookmark has been given a title. By default, this is the title of the current page—but, some pages don't offer titles. So, an error needs to be flagged if the user hasn't manually supplied a title in this case.

Finally, notice in both of these error cases, a string of HTML is composed in the variable tmpl. This will be used at the end of the method to populate the DOM node passed in as pblock.

Now, assuming that all the command's prerequisites have been met, it's time to try constructing a proper preview for the results of this command:

        } else {

            // Attempt to construct a vaguely delicious-esque preview of a bookmark.
            tmpl = [ 
                '',
                '',
                    'Share a bookmark at  ',
                        'delicious.com/${user_name}:',
                    '',
                        '',
                        '${bm.description}',
                        '${bm.url}',
                        bm.extended ? 
                            '${bm.extended}' : '',
                        bm.tags ?
                            'tags: ${bm.tags}' : '',
                    '',
                ''
            ].join("\n");

        }

        pblock.innerHTML = CmdUtils.renderTemplate(tmpl, ns);
    },

Building on the notion that previews are built in a DOM node, the code above uses both CSS and HTML to assemble a quick-and-dirty facsimile of a Delicious bookmark—which will be rendered like this:

Also note that Ubiquity provides a template engine for use in generating content—namely the JavaScript Templates engine from the TrimPath project. This engine may eventually be replaced with another, but the notion is that Ubiquity will provide tools to more easily generate previews and more.

The conclusion of the .preview() method uses the template engine with a call to CmdUtils.renderTemplate() to inject content into the preview element by way of the .innerHTML property.

Now that the preview is out of the way, it's time to get down to implementing the execution of the command:

    
    /**
     * Attempt to use the delicious v1 API to post a bookmark using the 
     * command input
     */
    execute: function(input_obj, mods) {
        var bm          = this._extractBookmarkData(input_obj, mods);
        var user_cookie = this._getUserCookie();
        var user_name   = (user_cookie) ? user_cookie.split(' ')[0] : '';

        if (!user_name) {
            // If there's no user name, there's no login, so this command won't work. 
            displayMessage('No active user found - log in at delicious.com ' +
                'to use this command.');
            return false;
        }

        if (!bm.description) {
            // If there's no title, somehow, then this is an error too.
            displayMessage("A title is required for bookmarks at delicious.com");
            return false;
        }

Mirroring the .preview() method, the .execute() method first checks for validity of the arguments given by the user. A missing user name or title result in a notification that the command has failed.

But, if the arguments are all valid, it's time to actually issue a request to the Delicious V1 API:

        var path = '/v1/posts/add';
        var url  = this._config.api_base + path;

        var req = Components.classes["@mozilla.org/xmlextras/xmlhttprequest;1"].
            createInstance();

        req.open('POST', url, true);

        req.onload = function(ev) { 
            displayMessage('Bookmark "' + bm.description + '" ' + 
                'shared at delicious.com/' + user_name);
        }

        req.onerror = function(ev) { 
            displayMessage('ERROR: Bookmark "' + bm.description + '" ' + 
                ' NOT shared on delicious.com/' + user_name);
        }

Using the base URL for the Delicious API declared earlier in the configuration section, the .execute() method constructs an API URL for the /v1/posts/add method. Then, it creates an instance of XMLHttpRequest from the browser to be used in sending the API request. Event handlers are registered with the object to present notifications to the user indicating whether or not the API request was successful.

At long last, it's time to wrap up this method and make the API request:

        req.setRequestHeader('Authorization', 'Basic Y29va2llOmNvb2tpZQ=='); // btoa('cookie:cookie')

        var mediator = Components.classes["@mozilla.org/appshell/window-mediator;1"].
            getService(Components.interfaces.nsIWindowMediator);
        var win = mediator.getMostRecentWindow(null);
        var user_agent = win.navigator.userAgent + ";Ubiquity-share-on-delicious";

        req.setRequestHeader("User-Agent", user_agent);      

        req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
        req.send(this._buildQueryString(bm));
    },

    EOF:null // I hate trailing commas
});

The login cookie authentication supported by the Delicious V1 API is triggered by supplying a user name / password pair of cookie, which is done by setting the Authorization request header. The login cookie is then expected to be passed in as the POST variable _user, which is done in the ._extractBookmarkData() utility method.

Another bit here that shows more access of browser resources is the construction of a unique User-Agent header for this API call based on the browser's own User-Agent string, something that's suggested in the guidelines for using the Delicious API.

Finally, the .execute() method—and the command itself—is wrapped up with by sending off the bookmark data encoded as POST form variables with the ._buildQueryString() utility method.

And, that's it—a command-driven Delicious browser extension in a little over 200 lines of code. There's still more to be done to really make this thing full-featured, but I think this shows off the basic features of Ubiquity. I'm hoping to dig in deeper and explore further, taking a look at running Greasemonkey-style code at browser startup and page load, as well as playing with some more browser chrome features.

Archived Comments

Jason Clark
2008-09-01T14:47:35
First off- fantastic post. Great to see a lengthy post here again, although I'm one to talk. This is an excellent introduction to Ubiquity command development, and tres useful to boot.

I'm wondering why you chose to construct and post the XMLHttpRequest manually instead of using jQuery, which is included with Ubiquity. I don't know that there's any benefit other than some simplicity, but I took a crack at converting your code to use jQuery, which works nicely. In the 'execute' function, replace everything after "var url = this._config.api_base + path;" with this (hope code blocks work in comments):

var win = context.focusedWindow; var user_agent = win.navigator.userAgent + ";Ubiquity-share-on-delicious";
```
    jQuery.ajax({
      type: "POST",
      url: url,
      data: this._buildQueryString(bm),
      username: "cookie",
      password: "cookie",
      beforeSend: function( req ) {
        req.setRequestHeader("User-Agent", user_agent); 
      },
      error: function() {
        displayMessage('ERROR: Bookmark "' + bm.description + '" ' + 
            ' NOT shared on delicious.com/' + user_name);
      },
      success: function() {
        displayMessage('Bookmark "' + bm.description + '" ' + 
            'shared at delicious.com/' + user_name);
      },
    });
```
Also, with both versions of the code, I'm seeing some unexpected behavior around authentication. Assume I'm logged in to delicious, with "stay logged in" checked, and I restart my browser. Trying to post with the command fails with a 401 unauthorized, even though I can see the cookie was sent (via Live HTTP Headers extension). Going to delicious.com shows me logged in, and once I've viewed the site, the command works. Except that now I can't reproduce; but I know it happened because I've got the headers. At any rate, it is working nicely, but the previous failure is bugging me... feel like I'm overlooking something.

Thanks again for an awesome post. Hope to see more of the same.
Ryan Scott Scheel
2008-09-01T15:07:01

You should be helping with the documentation, if you aren't already. Very nice job with this article;
Aza Raskin
2008-09-02T01:44:13

Hi Leslie,

This is a beautiful tutorial on writing a Ubiquity command. We'd love your help in making Ubiquity's documentation better (especially dev facing). You should totally link to this from the Ubiquity Wiki -- or even add the content in someway.

Anyway, just wanted to say thanks.

-- Aza
seth
2008-09-08T17:30:16

Am I doing something wrong?

When I enter in "sh this tagged tag1 tag2 entitled This is a title" everything after the tagged is added as a tag. It's not seeing the entitled part.
http://spyced.blogspot.com/
2008-09-15T19:12:58

Did something break? I'm getting a 404 accessing http://decafbad.com/hg/UbiquityCommands/file/tip/delicious.ubiq.js
l.m.orchard
2008-09-15T23:07:42

Yeah, looks like I had a small snafu with switching back from Lighttpd to Apache. Left out a rewrite rule - doh!
Tony
2008-10-22T04:56:03

Great article. This is replacing my delicious bookmarklet. Thanks!
dgtlchlk
2009-04-14T01:06:57

Great article and command. Wish it worked correctly with the latest 0.1.8 release though. No matter what text you put in it adds everything as the notes. The tagged and entitled modifiers don't work.
Nolan
2009-04-16T03:10:07

I second that comment. Delicious is actually saying the link given was "chrome://browser/content/browser.xul" and marking it as harmful inside delicious!
Justin
2009-07-06T12:39:39

Hi Leslie,

Thanks for the delicious ubiquity command. Unfortunately, as one of the commenters above mentions, the tagged modifier doesn't seem to work. I'm using Ubiquity 0.5 pre and typing the phrase:

share tagged foo

Adds the bookmark to Delicious with the note text "tagged foo"

Cheers, Justin
Eric
2009-07-20T22:43:17

Echoing others.. I'm using the latest ubiquity (err, next to latest.. 0.1.8?), and doing "share-on-delicious this is a note tagged testing" saves the link with notes "this is a note tagged testing" and no tags. :( Then again, the other delicious command I've found (by someone else) is having the same issue.

Delicious 2.0 is more than a pretty new face

2008-08-04T17:21:55+00:00

I'm no longer at Yahoo! and I no longer work on Delicious, but I'm still a huge supporter. And, since I'm pretty sure everyone over there is either burnt out or still insanely busy at the moment, it might be awhile before anyone tells the full story of what this relaunch offers. As it is, even I only know a bit of what all went into this effort.

So, off the top of my head, I thought I might point out just a few of the easily-missed improvements the new site offers beyond the great new love-it-or-hate-it visual redesign that seems to have occupied most of the discussion I've seen so far:

Search works and is incredibly fast. In fact, search may really be the real star of the redesign show here—especially since del.icio.us often took 30 seconds or more for a simple search, rendering it all but useless. Today, though, it's at ludicrous speed in comparison—and so finally, the real power of search applied to social bookmarking might start to shine with the critical mass of content found by real people using Delicious.

To scratch my own itch, I've created an unofficial OpenSearch search engine plugin for Delicious on Mycroft. Though I think it comes along with the browser extensions, I've yet to find this for autodiscovery from the site itself.

The notes field in bookmarks has been expanded from 255 to 1000 characters. Yes, this means that you can now include long running quotes from pages, or complete paragraphs of rambling discourse.

The feeds have all been overhauled and reorganized. An attempt at backward compatibility was made, but the old feed URLs are all deprecated. Replacing these, there's now a common and consistent URL namespace for feeds across formats.

Almost all RSS feeds have JSONP counterparts, and further feed formats could be considered. Additionally, the old mix of RSS 1.0 and 2.0 has been dropped in favor of RSS 2.0 format across the board to support podcast and media enclosure elements consistently. The linkrolls, tagrolls, network badges, and tagometers now all use the new JSONP feeds—and the widgets can be examined as example code in using the feeds.

Tag bundles can now be viewed as combined bookmark views, complete with feeds. This augments bundles from a simple visual organization tool to a more powerful content aggregation function. Personally, I never had much use for tag bundles until now, but since they can actually be used to partition tags and bookmarks I might actually take the time to use them. Check out these example URLs:

Bundles now work for tags, network contacts, and subscriptions. This means, for example, that you can partition your network contacts into topical groups. From there, you can subscribe to those partitioned bookmarks in different folders in readers, or use the bundled bookmark views in mashups through JSON feeds. Check out these example URLs:

Network member and fan feeds now include the date when the contact was added. Pro tip: Subscribe to fan feeds to see when new people have started following your bookmarks.

Several previously undocumented parts of the V1 API have now been documented. In particular, the following new parameter combinations have been used with the browser extensions for primitive bookmark sync:

https://api.del.icio.us/v1/posts/all?hashes
https://api.del.icio.us/v1/posts/all?meta=yes
https://api.del.icio.us/v1/posts/all?fromdt={DT}&todt={DT}
https://api.del.icio.us/v1/posts/all?start={##}&results={##}

The posts/all API URL works again for users with large collections. With my 11k+ bookmarks, del.icio.us was keeling over with the attempt to assemble and deliver my entire collection with a posts/all call. Now, however, Delicious 2.0 appears able to handle this call for my account.

Easter eggs have been rotated and recalibrated. No, I'm not going to tell you what or where or how many they are.

Archived Comments

Aristotle Pagaltzis
2008-08-05T02:31:52

Since I’m pretty sure everyone over there is either burnt out or still insanely busy at the moment, it might be awhile before anyone tells the full story of what this relaunch offers.

Admit it, Les, you just got impatient after building a bunch of shiny toys that no one at Yahoo! seemed to be coming forth to tell the world about them. :) (I would be miffed too.)
Yining
2008-08-05T06:51:44

one thing in the new version that disappoints me is the tab-to-complete feature is gone. It was replaced with a drop-down list which not only costs me more finger/hand movement but also breaks my daily usage habit of del.icio.us. Oh, maybe I can write a GMscript to scratch my own itch....

Btw, regarding the search, I find using a keyworded bookmark is much faster. For example, I could setup keyword 'del' and just type 'del webdev+openid' to search all my bookmarks tagged with these two keywords. One plus point, it works on SeaMonkey too (which doesn't support opensearch yet).
Nick
2008-08-05T06:56:05

Yay! So glad you wrote this, Les!
Paul D. Waite
2008-08-05T17:07:03

one thing in the new version that disappoints me is the tab-to-complete feature is gone. It was replaced with a drop-down list which not only costs me more finger/hand movement but also breaks my daily usage habit of del.icio.us

Amen.

Also,the option to delete a bookmark from its own page seems to have disappeared. I have to find it in my list of bookmarks, which isn’t always easy when it’s been there for a couple of weeks.
Paul D. Waite
2008-08-05T17:09:25

Sorry, that sounded a bit negative. I do love how autocomplete now only offers me tags that I’ve used. And it is kinda neat to see how many times I’ve used each one, but I’d give that up to have the old tab-to-autocomplete back. Way too much thinking involved with the menu.
Niv Singer
2008-08-05T18:52:38

I couldn't find the "Network member and fan feeds" - where are they??
l.m.orchard
2008-08-05T19:03:00

@Niv: Not all feeds are linked in from pages - check out the feeds help page on delicious.com. For example:

http://feeds.delicious.com/v2/rss/networkmembers/nivsin

http://feeds.delicious.com/v2/rss/networkfans/nivsin
Philip Dorrell
2008-08-06T02:01:52

It seems that the posting API's have all stopped accepting a "tags" parameter, which partly breaks my Firefox "deli" search keyword.

e.g. "deli toread math" used to fill in tags "toread" and "math", now it doesn't.

Also, does delicious have any equivalent to Google's site: prefix? That would be really good for a "most popular pages on my website" option.
l.m.orchard
2008-08-06T21:41:06

@philip Yup, that's broken for me too. See also: http://decafbad.com/blog/2008/08/02/delicious-20-legacy-bookmarklet-fix

As for something like site:, I think host: used to work but it seems no longer with the new search implementation.
Jon Williams
2008-08-22T18:38:43

Network member and fan feeds now include the date when the contact was added. Pro tip: Subscribe to fan feeds to see when new people have started following your bookmarks.

Where is this? (I feel silly not being able to find fan feeds)

delicious 2.0 legacy bookmarklet fix

2008-08-02T22:34:28+00:00

As you've probably seen by now, Delicious 2.0 has launched. It's an all new design and the whole thing has been rewritten from the ground up. Most of the gripes I've seem like general dislike of change—which actually attests to the gargantuan effort put forth to reimplement the original from scratch in a whole new language and architecture.

That said, I found at least one little bug that stops my usual bookmarklet flow. And, what's really annoying is that, it's probably a bug in code I wrote at one point. As it turns out, the original URL parameters for the bookmark posting form don't seem to be accepted anymore, so legacy bookmarklets may be broken. I swore I tested that, since I've got some personal investment in it.

But, although I can't contribute code to the project anymore, I've at least still got Greasemonkey. And, through this quick & dirty user script, I'm back to using my legacy bookmarking bookmarklets. In case you're interested, here's my favorite Firefox keyword shortcut bookmarklet—just select some text, type the keyword in the URL bar with some tags, and hit return.

Archived Comments

Peter Beardsley
2008-08-03T13:52:14

Just curious-- what language was del.icio.us originally implemented in, and what language was it re-implemented in for 2.0?
l.m.orchard
2008-08-03T18:30:58

@peter: The original del.icio.us was built with Mason in Perl, backed by MySQL. The new Delicious 2 is built with Symfony in PHP for the web frontend, with all the out-of-box Symfony database abstraction replaced by a Yahoo!-developed data management and business logic layer written mostly in C++ for the backend. I'm pretty sure I've read all of that in other public releases, so I don't think I'm letting any cats out of bags there :)

date-based pagination

2008-07-17T18:16:01+00:00

Here's a small idea I've not yet had the chance to try out on a large scale: Time-based pagination in lieu of page-number-based pagination for personal content - ie. blogs, bookmarks, status updates, etc. (You know, User Generated Content except I dislike the term.)

Page numbers change over time, while time-based URLs are stable. Most people don't generate an unreasonable amount of stuff in a day, so a page-per-day might not be so bad. But, if there's too much stuff, degenerate to a page-per-12-hours or whatnot.

Delicious used to have something like this in the web UI, back in the mists of 2003 or so, but Joshua got rid of it after a few design iterations. The Delicious API is still somewhat based on it, which causes some confusion—but I tend to like it, thus this post.

Archived Comments

fluffy
2008-07-17T20:22:38

You mean, like the date-based archives that Movable Type has had for ages?

Yeah, I like those.
Neil Kandalgaonkar
2008-07-18T19:44:42

I've had the same idea for Upcoming, for obvious reasons... our SRP pagination would then be permalinks.
Anil Dash
2008-07-25T17:11:40

Actually, yeah, we built the default pagination for archives in MT4 to be category-by-month, for exactly this reason. I regularly see "?page=2" in URLs and it just seems like such an obvious, avoidable mistake for so many to have embraced. SEO aside, permalinks that are destined to expire over time seem pretty reader-hostile, too.

Queue everything and delight everyone

2008-07-04T19:17:18+00:00

This is a blog post I've had simmering in my brainmeats for well over a year or two. I'm suddenly inspired to break blog-radio-silence and get it out of my head.

From Let the microblogs bloom - RussellBeattie.com:

Once this is widely accepted (and I'm sure there are many that would argue with me), the thing that will separate these types of services won't be whether they stay up (ala Twitter), but how fast your subscription messages are updated. Some services might be smaller or offer more features but not update as quickly whereas others will pride themselves on being as close to real-time as possible. The key is that it's all about messaging, not publishing. (Oh, and this also facilitates federation as well, but that's another topic).

One of the problems it seems most modern web apps face is the tendency to want to do everything all at once, and all in the same code that responds directly to a user. Because, while you're in there building a user interface, it's easy to implement everything else that needs to happen in that same UI module or library.

Someone wants to post a bookmark? Someone wants to post a message? Well, of course you want the system to cross-reference and deliver that new piece of User Generated Content through every permutation of tag, recipient, keyword, and notification channel supported by your system.

But, do you really have to do everything all at once—while the person who generated that content is tapping his or her foot, waiting for the web interface to respond with feedback? Are all of these things immediately vital to the person watching the browser spin, right now?

No. Your user wants to get on with things. He or she wants to see the submitted content get accepted and, as feedback and confirmation, see it reflected in a personal view immediately. Does it matter—to this person, at this moment—whether it shows up simultaneously in a friend's inbox, the public timeline, a global tag page, or even an RSS or Atom feed?

Again, no, simultaneity doesn't really matter—because no human beings actually appreciate it. Instead, imagine a ripple effect of concentric social and attention contexts with associated people spreading out from the original submission. (This probably rates the creation of a diagram someday.)

To make the person who's submitting something happy, offer feedback visible in their own personal context in under 50-200 milliseconds. (That is, less than half-a-second at worst, in people terms.)
The next person to delight is someone following the first person's published content—and humanly speaking, delays of tens of thousands of milliseconds can be acceptable here. (That is, 1-10 seconds at worst, in people terms.)
Finally, you can start worrying about strangers, allowing the content to propagate to tag pages, keyword tracking pages, and other public views—and I'd assert that delays of hundreds of thousands of milliseconds are acceptable here. (That is, 1-2 minutes at worst, in people terms.)

The idea here is that the social structure can help you scale, while still delighting people. Even with these delays, the system is still better at getting the word out than the original content creator would be at notifying all the others involved with an out-of-band system like IM or email. And that's at worst—on most good days, all the delays should tend to be on the order of seconds or less.

And how do you do all of this? Use queues. Sure, the original submission of content can and should be done all at once—just enough to get the content into the user's collection. Then, queue a job for further processing and get out of the way. In fact, just queue one job from the user interface—the processor of that queue can then queue further jobs for all the other individual processing tasks that are likely susceptible to plenty of parallel processing and horizontal scaling.

Meanwhile, the original user creating content has been thanked for their submission and life goes on. In fact, their life may include going on to submit many more pieces of content in rapid succession, thanks to your delightfully responsive web user interface.

And, in the end, that's really the purpose of a web-based content creation interface—accepting something as quickly as possible to make the user happy enough to continue submitting more. The other part of the user interface, retrieval, serves simply to get the original content distributed as fast as can be reasonably expected.

Now, preparing for fast retrieval is another story. The flip side to processing queues are message inboxes—expect content duplicated everywhere and fetched simply, rather than using cleverly expressed SQL joins that bring a system to its knees. But, that's another post altogether. :)

Archived Comments

shyam
2008-07-04T21:12:46

Interesting bits of conversation happening all over the place regarding queues and the irony of it all - the much maligned Java has had workqueues since the early days.

What everyone will learn, rather painfully in in cases like Twitter, is that all data is not created, consumed or processed equally. If you write your system which treats data equally you'll wind up with many Twitters all over the place.
l.m.orchard
2008-07-04T21:37:30

Oh, definitely. Work queues are not a new thing at all. It's just that I think there're a lot of modern web app builders who skipped Java "enterprise" software—skipped, or hoped to run away—and are rediscovering the whole set of problems. Maybe the solutions will be less over-engineered this time.
Mike Warot
2008-07-04T21:40:15

The thing that drives me nuts about twitter is that the core data rate is only about 30k/second... yet it kept going down. It's easy to spit out a broadcast to a subnet and never even miss a packet if there are only 100 of them per second or so. There's no reason on god's green earth that twitter should be anywhere near overloaded.

Bad architecture, on the other hand, is the work of Satan. ;-)
Simon Willison
2008-07-04T21:40:39

Thank you! I've been trying to put my thumb on why queues are so interesting for months; this expresses it perfectly.
Gabe Wachob
2008-07-04T22:54:10

Usually I would have something serious to say in agreement with you, because I do so much agree with you.

But I have just one comment:

DUH!!!
Don Park
2008-07-06T05:16:32

Good suggestions. For social services like Twitter, I would also add one more item:

Prioritize by Relationship

For example, two-way Twitter relationships (mutual-follow or recent @ or direct message exchange) should be refreshed before one-way. One can go further by placing higher priority on users whom poster sent messages to or received from within past X-hours.
alexis
2008-07-10T12:35:25

er... eventually consistent social graphs anyone?
Robert Schultz
2008-07-10T13:36:22

I agree with everything you've said. Especially the last part, duplicating data in the format it will be retrieved in rather than using complicated and CPU intensive SQL queries. This is especially true for any sort of statistics or reporting. I learned this by seeing my website's statistics growing slower and slower to retrieve as more and more traffic caused the database to become larger and larger and all of a sudden those queries that ran nearly instantly, even with good indexing were taking several seconds to return.
citric
2008-07-10T21:49:19

Web apps doing things while the user waits unnecessarily is an old phenomenon. I think it's often a matter of developers not wanting to (and/or being politically unable to) venture into what they consider the sysadmin's domain. Take the way-too-common case of apps that make the client wait while it does housekeeping. Why isn't this in a cron job? One reason is maybe this is KewlOSSBlogWikiPackage and it's simpler to say "just untar the package under htdocs and you're done" instead of saying "also, unpack these scripts in a non-servable area and set them up to run hourly, but not all at the same time; stagger them a little. And run them with the same UID your web server is running as". But we end up with a lot of apps that (badly) reimplement basic tools their OS ships with in the first place.
Bill Seitz
2008-07-28T21:23:06

I wonder if you're still setting the bar too high for low-priority connections. I mean, microblogging isn't really messaging, and maybe isn't (shouldn't-be?) conversation.

So why wouldn't 10-15min be good enough?

What % of "messages" are read instantly after they hit an inbox?
Jay Yukes
2008-11-23T10:35:55

I had to solve a similar problem. Needed the fastest possible response, so had to rule out interacting with the Database directly from the web app. Used PHP message queue Dropr to defer all DB work. It is very fast, easily over 1000 messages/second
jmxz
2009-02-13T20:00:13

Wow those comments make me feel old. I remember when these java queues everyone's referring to reminded me of how I had a VAX dedicated to queuing and scheduling batch jobs for a Cray.
Chris Henry
2009-06-11T20:09:47

Handling load is probably one of the biggest problems facing websites today. Queueing is definitely the way to go, but like you said, sites need the type of architecture where it's easy to deploy services to different machines. Usually by the time the site is under load, it's too late...
maht
2010-05-04T14:50:05

I use Inotify as my queue messaging system

http://maht0x0r.blogspot.com/2009/06/serialising-multiple-writers-in-shell_20.html

Inotify can wait on MOVED_TO or CLOSE_WRITE events so that you can add them to the queue when the upload has finished.

It should also be noted that this is a mnethod of load balacing too. Instead of 1000 parallel thumbnails being produced all context switching away, you can determine how many processes get spawned, use the OS' resource managing features etc.

wow, eve, and delicious

2007-11-21T04:20:14+00:00

Come see me try to write big words on my new blog about what I hope is not a new MMORPG addiction, while I work through the aftermath of this thing having been hacked.

bookmarking outlier

2007-10-09T16:18:56+00:00

I've amassed 9681 bookmarks on del.icio.us as of right now. Quick napkin calculation: I post an average of 7 bookmarks a day, over my lifetime of 1353 days using the site. Within the next 45 days or so, I should reach 10000 bookmarks at this rate. I wonder if I should throw a party?

Invites to the delicious preview

2007-09-28T18:12:05+00:00

Update: Oh! And since even I missed this link in the original blog post, you can request to be invited. Not sure if that results in an email, either, though.

Update #2: Also, we're trying out Vanilla forums as an alternative to the delicious mailing list. If you're invited to the preview, your login should work there too. We might even launch public forums once del 2 is out.

In case anyone's forgotten: I work on delicious. Since I've had a steady stream of friends and acquaintances asking me about how they can get into the invite, I thought I'd mention this:

You may already have been invited! :)

Unlike most Web 2.0 sites that open up a preview or invite-only thing with a mysterious sign-up form, we can't fire off an email when we create an account for someone on our preview. It's thanks to our privacy policy - although we have the email addresses of most everyone we've been adding to the preview, we're not allowed to send anything other than password recovery emails. Weird, I know, but sometimes organizations try to follow things like that.

So, the problem is that we have many more people invited to the preview than have actually started using it. As briefly mentioned in the blog post announcing the preview, you need to check your "Links for You" to see if you've been sent a link to the preview.

But, if your invite has possibly been buried in other links from friends - you might just try signing into the preview with the username and password from your existing delicious account! It might just work.

Archived Comments

chryss
2007-09-28T19:58:43

Sniff. Cry. I'm not.
slackorama
2007-09-28T21:12:20

Wah. Nor am I. We are network buds and everything? What's up with that?
Kevin Severud
2007-09-28T22:27:55

So if I haven't gotten my invite yet and I log in trough the preview link (which I did) should my bookmarks still have been imported? Currently I have none of my bookmarks from the original site.
D. Archibald Smart
2008-03-05T03:56:24

if i promise to use the preview, can i get an invite, please?

correlation is not causation

2006-12-21T02:01:38+00:00

By the by: Correlating the new del.icio.us JSON URL feed with Google's recent deprecation of the SOAP Search API is about as effective as noticing how often the clock reads 12:34 when you just happen to be looking at it.

Archived Comments

Joe Grossberg
2006-12-21T13:10:16

Because ... you say so? :)

Also, I think you misuse the word "correlation".

And I think there is a correlation -- both are indicative of a trend in "web services" today.

JSON and YAML are getting more popular in that arena and XML (especially SOAP) is something that a lot of programmers dislike workign with.

Tagometers and Travesties

2006-12-21T01:19:56+00:00

You know you've truly arrived in Silicon Valley when the darling new feature you just helped launch on a Web 2.0 site gets shredded in a rant by Dave Winer. :)

My day, having started with me in a foul mood, has been made. Pre-launch, the tooth marks in my tongue have gotten deep as I watch the comments pile up. It's exciting to have so many people debating something you've made. (Update: Seriously. I'm in a better mood now. It's made me laugh.)

Actually, it wasn't the feature itself that got shredded, so much as the new JSON data feed revealed before the intended star of the show made it on-stage, by way of a screencast preview on Y!DN by Matt McAlister and an amplification by Niall Kennedy.

The feed itself is mainly intended to power our new Tagometer page widget include thingy. Beyond that, leaving the feed decoupled from the include itself made sense to me, both for our own flexibility's sake and as a nod toward our mashup friends in the blogosphere and lazyweb.

You see, I like JSON. It was convenient in this use case of creating a remote blog widget fed by cross-domain sourced data. I, personally, didn't even have to reinvent anything that wasn't already implemented in assorted off-the-web tools. JSON worked for this purpose - it might not work for your purpose. ~~If it doesn't, please accept my insincere apology, but yours wasn't my primary goal. Attempts to string me up for this travesty will be met with giggles and pointing.~~ (Update: I realize I'm not the to-be-strung-up party responsible for the JSON travesty - I just couldn't resist being a smart ass for its own sake. I really do giggle and point a lot. My actual apologies for that.)

It's funny, laugh.

JSON URL feed is not really an API. JSON URL feed is not a bold declaration of side-taking in the grand war of web service specifications. Do not base business models on JSON URL feed. Caution: JSON URL feed may suddenly accelerate to dangerous speeds or stop altogether. JSON URL feed contains a liquid core, which, if exposed due to rupture, should not be touched, inhaled, or looked at. Ingredients of JSON URL feed include an unknown glowing substance which fell to Earth, presumably from outer space. If JSON URL feed begins to smoke, get away immediately. Seek shelter and cover head.

Do not taunt JSON URL feed.

Archived Comments

Dave Winer
2006-12-21T02:00:51

Not sure if I helped your mood or made it worse, but the rant wasn't about you.

http://www.json.org/xml.html

Tell me what you think of that bit of hyperbole. Someone there is trying to undermine a lot of work a lot of people did over a long period of time to get software to interoperate. When you say that's just my issue, you miss the point. It should be everyone's.
Geof F. Morris
2006-12-21T02:28:32

Aw, man. It should've been "Do not taunt happy JSON URL feed."
l.m.orchard
2006-12-21T02:47:44

Honestly, it brightened my mood. The smiley was neither ironic nor sarcastic. Something big I helped launch has been stirring an interesting discussion, however tangential my blame in the making.

Vanity and Carly Simon aside, I know I'm not the main perpetrator of JSON. I just couldn't resist an opportunity to be a smart ass. For that I have a sincere apology. I should probably strike it or delete it now that the giddiness of a major launch and a lot of attention has subsided.

But seriously, what I meant above is not that this is your issue, per se. This post is really more directed at some comments on your blog post, in bulk.

Our particular intentions behind using JSON here do not necessarily map to anyone else's expectation from a web service or standards-driven perspective. It seems like a lot of people are looking at this JSON data since its early release, divorced from it's main consumer: The Tagometer JS include. It works for that purpose - and incidentally might help others - but we're not working very hard to support other uses.

Apropos of that, I don't particularly buy the hyperbole about JSON being XML's superior in the general case. JSON's a brilliant hack, but it's got issues. And seriously, if anyone thinks that adding and removing script tags on the fly in a browser DOM is the best overall way to go about things... they should be strung up. It just so happens that right now, it's the only way to do some things.
timb
2006-12-21T11:22:52

hooray, i'm glad to see the json url feed.

however, i shouldn't have to generate the md5s of the urls i am interested in... i should just be able to give it the actual urls.
John Manoogian III (jm3)
2006-12-30T02:28:08

mmmmm, swilling the liquid core...!

painfully metalicious

2006-11-29T00:05:22+00:00

I find this sequence of URLs almost as funny as the blonde joke:

I'm also happy that del.icio.us uses MD5 hashes to normalize this situation.

Update: I also see that Kottke beat me to the meta.

Back in the Saddle Again

2006-10-02T21:55:42+00:00

When I got up this morning, this was my sad state of affairs:

But, I'm back at the office now, albeit at reduced capacity for zooming around - unless said zooming may be done on crutches. And, let me tell you, zooming around on crutches is both hazardous and exhausting in an office environment. Also? I want my hands back. It's hard to carry hot coffee or oatmeal with no hands.

I haven't been posting much around here, thanks to various moments in a drugged up haze or just plain wallowing in convalescent marathon WoW sessions. The only real productive work I've gotten done was for del.icio.us. Over the past few weeks, I just haven't felt like doing much or hacking anything at all. However, I am pretty sad that I missed Yahoo! Hack Day.

On the bright side, I'm planning to be propped up in a corner at least for the del.icio.us birthday bash. Come sign the cast, or plaster me with a sticker while I'm too immobile to do much about it.