0xDECAFBAD - Tag: outliners

HTML5 drag and drop in Firefox 3.5

2009-07-15T15:26:40+00:00

Oh hey, look! It's another blog post—and this one is cross-posted on hacks.mozilla.com. I won't say this is the start of a renewed blogging habit, but let's see what happens.

Drag and drop is one of the most fundamental interactions afforded by graphical user interfaces. In one gesture, it allows users to pair the selection of an object with the execution of an action, often including a second object in the operation. It's a simple yet powerful UI concept used to support copying, list reordering, deletion (ala the Trash / Recycle Bin), and even the creation of link relationships.

Since it's so fundamental, offering drag and drop in web applications has been a no-brainer ever since browsers first offered mouse events in DHTML. But, although mousedown, mousemove, and mouseup made it possible, the implementation has been limited to the bounds of the browser window. Additionally, since these events refer only to the object being dragged, there's a challenge to find the subject of the drop when the interaction is completed.

Of course, that doesn't prevent most modern JavaScript frameworks from abstracting away most of the problems and throwing in some flourishes while they're at it. But, wouldn't it be nice if browsers offered first-class support for drag and drop, and maybe even extended it beyond the window sandbox?

As it turns out, this very wish is answered by the HTML 5 specification section on new drag-and-drop events, and Firefox 3.5 includes an implementation of those events.

If you want to jump straight to the code, I've put together some simple demos of the new events.

I've even scratched an itch of my own and built the beginnings of an outline editor, where every draggable element is also a drop target—of which there could be dozens to hundreds in a complex document, something that gave me some minor hair-tearing moments in the past while trying to make do with plain old mouse events.

And, all the above can be downloaded or cloned from a GitHub repository I've created especially for this article—which continues after the jump.

The New Drag and Drop Events

So, with no further ado, here are the new drag and drop events, in roughly the order you might expect to see them fired:

dragstart: A drag has been initiated, with the dragged element as the event target.
drag: The mouse has moved, with the dragged element as the event target.
dragenter: The dragged element has been moved into a drop listener, with the drop listener element as the event target.
dragover: The dragged element has been moved over a drop listener, with the drop listener element as the event target. Since the default behavior is to cancel drops, returning false or calling preventDefault() in the event handler indicates that a drop is allowed here.
dragleave: The dragged element has been moved out of a drop listener, with the drop listener element as the event target.
drop: The dragged element has been successfully dropped on a drop listener, with the drop listener element as the event target.
dragend: A drag has been ended, successfully or not, with the dragged element as the event target.

Like the mouse events of yore, listeners can be attached to elements using addEventListener() directly or by way of your favorite JS library.

Consider the following example using jQuery, also available as a live demo:


    Drag me!
    Drop here!

Thanks to the new events and jQuery, this example is both short and simple—but it packs in a lot of functionality, as the rest of this article will explain.

Before moving on, there are at least three things about the above code that are worth mentioning:

Drop targets are enabled by virtue of having listeners for drop events. But, per the HTML 5 spec, draggable elements need an attribute of draggable="true", set either in markup or in JavaScript.

Thus, $('#newschool .dragme').attr('draggable', 'true').
The original DOM event (as opposed to jQuery's event wrapper) offers a property called dataTransfer. Beyond just manipulating elements, the new drag and drop events accomodate the transmission of user-definable data during the course of the interaction.
Since these are first-class events, you can apply the technique of Event Delegation.
What's that? Well, imagine you have a list of 1000 list items—as part of a deeply-nested outline document, for instance. Rather than needing to attach listeners or otherwise fiddle with all 1000 items, simply attach a listener to the parent node (eg. the

Using dataTransfer

As mentioned in the last section, the new drag and drop events let you send data along with a dragged element. But, it's even better than that: Your drop targets can receive data transferred by content objects dragged into the window from other browser windows, and even other applications.

Since the example is a bit longer, check out the live demo and associated code to get an idea of what's possible with dataTransfer.

In a nutshell, the stars of this show are the setData() and getData() methods of the dataTransfer property exposed by the Event object.

The setData() method is typically called in the dragstart listener, loading dataTransfer up with one or more strings of content with associated recommended content types.

For illustration, here's a quick snippet from the example code:

var dt = ev.originalEvent.dataTransfer;    
dt.setData('text/plain', $('#logo').parent().text());
dt.setData('text/html', $('#logo').parent().html());
dt.setData('text/uri-list', $('#logo')[0].src);

On the other end, getData() allows you to query for content by type (eg. text/html followed by text/plain). This, in turn, allows you to decide on acceptable content types at the time of the drop event or even during dragover to offer feedback for unacceptable types during the drag.

Here's another example from the receiving end of the example code:

var dt = ev.originalEvent.dataTransfer;    
$('.content_url .content').text(dt.getData('text/uri-list'));
$('.content_text .content').text(dt.getData('text/plain'));
$('.content_html .content').html(dt.getData('text/html'));

Where dataTransfer really shines, though, is that it allows your drop targets to receive content from sources outside your defined draggable elements and even from outside the browser altogether. Firefox accepts such drags, and attempts to populate dataTransfer with appropriate content types extracted from the external object.

Thus, you could select some text in a word processor window and drop it into one of your elements, and at least expect to find it available as text/plain content.

You can also select content in another browser window, and expect to see text/html appear in your events. Check out the outline editing demo and see what happens when you try dragging various elements (eg. images, tables, and lists) and highlighted content from other windows onto the items there.

Using Drag Feedback Images

An important aspect of the drag and drop interaction is a representation of the thing being dragged. By default in Firefox, this is a "ghost" image of the dragged element itself. But, the dataTransfer property of the original Event object exposes the method setDragImage() for use in customizing this representation.

There's a live demo of this feature, as well as associated JS code available. The gist, however, is sketched out in these code snippets:

var dt = ev.originalEvent.dataTransfer;    

dt.setDragImage( $('#feedback_image h2')[0], 0, 0);

dt.setDragImage( $('#logo')[0], 32, 32); 

var canvas = document.createElement("canvas");
canvas.width = canvas.height = 50;

var ctx = canvas.getContext("2d");
ctx.lineWidth = 8;
ctx.moveTo(25,0);
ctx.lineTo(50, 50);
ctx.lineTo(0, 50);
ctx.lineTo(25, 0);
ctx.stroke();

dt.setDragImage(canvas, 25, 25);

You can supply a DOM node as the first parameter to setDragImage(), which includes everything from text to images to canvas elements. The second two parameters indicate at what left and top offset the mouse should appear in the image while dragging.

For example, since the #logo image is 64x64, the parameters in the second setDragImage() method places the mouse right in the center of the image. On the other hand, the first call positions the feedback image such that the mouse rests in the upper left corner.

Using Drop Effects

As mentioned at the start of this article, the drag and drop interaction has been used to support actions such as copying, moving, and linking. Accordingly, the HTML 5 specification accomodates these operations in the form of the effectAllowed and dropEffect properties exposed by the Event object.

For a quick fix, check out the a live demo of this feature, as well as the associated JS code.

The basic idea is that the dragstart event listener can set a value for effectAllowed like so:

var dt = ev.originalEvent.dataTransfer;
switch (ev.target.id) {
    case 'effectdrag0': dt.effectAllowed = 'copy'; break;
    case 'effectdrag1': dt.effectAllowed = 'move'; break;
    case 'effectdrag2': dt.effectAllowed = 'link'; break;
    case 'effectdrag3': dt.effectAllowed = 'all'; break;
    case 'effectdrag4': dt.effectAllowed = 'none'; break;
}

The choices available for this property include the following:

none: no operation is permitted
copy: copy only
move: move only
link: link only
copyMove: copy or move only
copyLink: copy or link only
linkMove: link or move only
all: copy, move, or link

On the other end, the dragover event listener can set the value of the dropEffect property to indicate the expected effect invoked on a successful drop. If the value does not match up with effectAllowed, the drop will be considered cancelled on completion.

In the a live demo, you should be able to see that only elements with matching effects can be dropped into the appropriate drop zones. This is accomplished with code like the following:

var dt = ev.originalEvent.dataTransfer;
switch (ev.target.id) {
    case 'effectdrop0': dt.dropEffect = 'copy'; break;
    case 'effectdrop1': dt.dropEffect = 'move'; break;
    case 'effectdrop2': dt.dropEffect = 'link'; break;
    case 'effectdrop3': dt.dropEffect = 'all'; break;
    case 'effectdrop4': dt.dropEffect = 'none'; break;
}

Although the OS itself can provide some feedback, you can also use these properties to update your own visible feedback, both on the dragged element and on the drop zone itself.

Conclusion

The new first-class drag and drop events in HTML5 and Firefox make supporting this form of UI interaction simple, concise, and powerful in the browser. But beyond the new simplicity of these events, the ability to transfer content between applications opens brand new avenues for web-based applications and collaboration with desktop software in general.

Archived Comments

David Semeria
2009-07-16T17:56:52

Wow - thanks for a great introduction to D+D in HTML 5, I'm really looking forward to giving it a thorough road test.

The key part of the implementation are the target listeners, because, as I'm sure you're aware, onmouseover would historically never fire over the target element because the dragged item would always 'cover it up'. BTW, my proposed solution was to add an 'event transparency' property, which would have made the dragged item invisible from the point of view of selected events, eg onmouseover.

This implementation is going to take a lot of pain away. You have no idea how many hoops I had to jump through to get a fully generic D+D implementation working without these new calls.

If you're interested, you can see it working here: http://lmframework.com/page.php?id=vd_twig_short_2
Alex
2009-07-17T00:13:23

Hey,

Looks great and useful article. For dragging from external applications (such as the Desktop) is a normal webpage able to get the data from the drop? I was just playing with it and seem to always get what seems to be a security error with getData or mozGetDataAt. However, the documentation seems to state that on drop that data should be made available to the page. Likewise, in the same way an image can be dragged off the page onto the desktop, can an arbitrary element be dragged onto the desktop with whatever file content to be saved? Thanks for the help.

Alex
Paul Tarjan
2009-07-17T04:52:54

Very nice, I didn't know that FF 3.5 actually had this implemented. Time to start playing :)

Also, how did you do your code posts? Was it just pasting in HTML or do you have a better setup?
Animal
2009-07-17T14:35:21

Seems pretty pointless pressing ahead with this when you just can't write a web app to use it. There's no support. All a bunch of whizzy fun I'm sure, but sod-all use.
l.m.orchard
2009-07-17T16:23:10

@Animal: Why can't you write a web app to use it? It's in the HTML5 spec, works in Fx3.5 and Safari 4 / WebKit. It doesn't have universal support yet, or course, but new standards never do.
paul
2009-07-21T08:08:26

very nice indeed! i like this, it has definitely opened up new avenues for interaction without using flash. but it will take time before people upgrade to the new browser. a lot of people who browse the web wont realise what the technology will bring and will use their current browser because it does the job!
Joel
2009-07-21T14:36:04

Excellent demos of the new features.

When using a feedback image on a zoomed in page, should the image not also be scaled up automatically?
RenegadeX
2009-07-22T04:13:46

Dragging & dropping is as you say, a "one of the most fundamental interations" that users of computer graphical interfaces have come to know, and expect.

It therefore completes dumbfounds me how it is that we still can not natively drag & scroll (& then drop) in Firefox. We can only drag & drop an item within the currently visible portion of a Firefox webpage or tab, no further up, no further down.

That is should be limited makes absolutely no sense. Microsoft built the function into their Windows 3.x (file) Explorer, and then when visual browsers came along, naturally carried it through into Internet Explorer.

Firefox Bug 41708, "Should be able to scroll in the viewport while dragging" was opened in June, 2000 (yes, 9 years ago!) and remains open, and is disregarded by Firefox developers as little more than a trivial little annoyance, and therefore is and should be treated as a low-importance 'enhancement' rather than as the 'standard feature' it should really be.

If it weren't for the extension 'DragToScroll', which has been around for 3-1/2 years now (not updated in a long time but still works if you override version compatibility), I would have dumped Firefox and switched to a different browser (Maxthon 3, most likely).

So, I'm wondering (and hoping) -- does the new HTML5 drag and drop specification include anything that if implemented properly would make it possible to scroll & drag?
l.m.orchard
2009-07-22T13:57:42

@RenegadeX: Ugh. I don't work on the browser itself, but I noticed that unexpected behavior on drag & drop. The window really should scroll when you get toward the top or bottom while dragging - and, in fact, most pre-HTML5 JS frameworks do that in their own drag & drop abstractions.

Until / unless Firefox gets this fixed, the same sort of auto-scrolling could be hacked in to HTML5 drag and drop. Not perfect, but it's doable. That is, in the drag event, you can check if the mouse is near the upper or lower edge of the viewport - and if so, start scrolling appropriately. That's pretty much how the JS frameworks do it with old-school D&D
CopperBot
2009-07-23T21:41:07

Very cool article! Thanks for sharing. HTML5 really is going to change everything about how we use and develop for the web. Pretty awesome!
Alexos
2009-07-26T05:05:44

Hi, would DD work for uploading files? How can I display a progress bar? Thanks! I've been looking for that several days, and found only applets which I can't use in my proyect :-)
Sam
2009-07-28T13:54:26

Looks fantastic, going to try it out now (once I have upgraded my FF)
Dan Kantor
2009-07-30T03:10:25

Looks like effectAllowed and dropEffect are not working for FF 3.5 on the Mac. I see effects on Windows and Safari 4 on Mac. I've been playing around with adding borders to the drop target, but the + icon for copy really helps a lot. Hopefully Mozilla will fix this next update.
Ryan
2009-09-01T02:33:06

Great article it's good to see developments in this area. I wrote an article on the further extensions Firefox 3.6 has done with the dataTransfer method by adding the file attribute so you can access local files and upload them without the need for file inputs. http://www.thecssninja.com/javascript/drag-and-drop-upload
phil swenson
2009-09-03T22:58:36

Why not a first class WYSIWYG rich text editor with copy from clipboard image support?

NO ONE has built a decent text editor in JavaScript - At least not that I've seen. And browsers don't allow image paste for reasons I don't understand.

And by decent I mean just like you'd get in textmate or another editor. Have the standard text editor features everyone expects: tab, select indent/select unindent, resize image, home, end, duplicate line, delete line, styling, etc.

would push the browser to the next level in killing the desktop. non-techies hate wikis. They want a real editor. I do too actually.... would be great for forums, bug tracking system (imagine pasting screen shots in line with your bug desc), email, etc.
Jean-Denis
2009-09-04T00:23:39

Francisco Tomalski wrote up a nice piece on HTML 5 drag'n drop at http://www.alertdebugging.com/2009/08/16/on-html-5-drag-and-drop/

where he uncovers that the proposed standard is partially broken. Any comment on his piece?
Lisa
2009-09-09T07:33:08

This is a great move forward no more relying on 3rd party apps and extensions to play video or audio, i have been reading up on HTML 5 at the HTML 5 Tutorials website, i am now playing around with one of the free templates and was wondering how to embed audio, so thanks a lot, great information, lets hope more people lean towards HTML 5 and SOON!!!

The drag and drop feature i did not notice was already working in FF 3.5, i was told to get Safari to see HTML 5 in action. Thanks for a great post
Zoltan Hawryluk
2010-01-11T15:14:48

A million thank yous! This article was great introduction to HTML5 D+D. With it, I was able to extend it to other browsers. It was a little painful at first because the browser implementations diverge in significant, but manageable ways.

If you are interested, check out my article at http://www.useragentman.com/blog/2010/01/10/cross-browser-html5-drag-and-drop/ if you are interested in my results.

I noticed you haven't blogged in a while - I hope you haven't stopped totally and continue to share with the webdev community.
Raul
2011-04-26T23:56:02

Hi, Leslie. I've just come across a problem that's driving me nuts. I'm not fully comfortable with D&D, but managed to move a crosshairs image over a map to very precisely controlled positions. It worked great on FF 3.6 and FF 4. After a couple of days of successful testing, the image suddenly refused to de dropped after being dragged (it rather flew back to its previous position). Do you know if there is a bug in FF that might cause this?

BTW, during the programming process I updated Firebug, which also is getting a little cranky.

Thanks for your prompt answer, Raul
Alexander Plutov
2011-10-01T11:51:12

Good post about Drag & Drop http://plutov.by/post/html5_drag_and_drop

XoxoOutliner and further outline addressing adventures

2006-11-15T08:07:12+00:00

Revised the addressing code a bit, adding a few new kinds of addresses and getting ready to support sub-outline updates. That is, fetch a sub-branch of an outline and then later post a change to that sub-branch using the same address. Needs more thought - ie. what happens if things move between fetch and update? - but here are a few more samples:

First is a straight linear index counting down from the top of the outline:
- http://decafbad.com/2006/11/XoxoOutliner/outlines/README;index:4?format=xoxo
Second is a navigation of outline structure, alternating numbers and letters:
- http://decafbad.com/2006/11/XoxoOutliner/outlines/README;level:3c4?format=xoxo

That's all for now. In my next round of enthusiasm, I may try stealing Tom Morris' Opath idea...

Archived Comments

Vincent D Murphy
2006-11-18T20:52:07

I think (and said as much on Tom Morris' site) that a fragment identifier would be a better solution, in which case Opath would be a fragment identifier syntax for OPML and XOXO. At least it would be the best solution from a REST/web architecture point of view..
l.m.orchard
2006-11-18T21:40:41

One reason I didn't use the #identifier URI syntax for suboutlines is because some gymnastics need to be done to get the hash through to the server from a browser. Otherwise, it gets treated as an in-page anchor. The semicolon syntax seems to work well for a set of path-segment parameters, and follows the standard (if I've read it correctly).

In either case, it works for me, and should be just fine in a REST context - the suboutline syntax here should always identify a single parent outline node as a resource, and will eventually work for GET / PUT / POST / DELETE.

Now I just need to implement a solution for the Lost Update Problem.
Michael Poremba
2008-09-18T23:45:56

Wondering if you ever completed your online outliner? Check out dynamiclist.com, a functioning but incomlete project I launched back in 2001. The editor is rich and works well. Been thinking of reviving now that all major browsers support the contentEditable tag.

XoxoOutliner rewrite coming, now with event delegation

2006-11-06T18:14:26+00:00

For what it's worth: I've answered my own question about event delegation based drag and drop. It seems to work quite well on my machine, which is quite a bit beefier than my old machine. I should try it out on there soon. But, I'll hopefully be releasing it soon in the form of a mostly-rewritten XoxoOutliner that will feature drag and drop outline items as well as keyboard navigation/control and in-place editing. And, it should actually start to become useful, since I plan to also whip up a PHP-based server side bit that will handle loading and saving outlines, as well as possibly a client-side bit that can save outlines using local browser storage. I have the technology - we'll see how far my ambitious plans take me during this hobbling sprint of development.

Archived Comments

Shawn Medero
2006-11-07T11:01:11

Great news Les! I had checked out the souce of XoxoOutliner and was thinking about playing around myself. Let me know if you'd like an additional tester. :)
Jesse Andrews
2006-11-08T18:18:11

Is there a demo of the event delegation framework you are building posted?
l.m.orchard
2006-11-08T18:23:29

Nope, nothing available or posted yet. Soon, though - hoping to check the works in before the end of the week. It's a bit in pieces right now
Brad Neuberg
2006-11-29T09:57:04

Les, very exciting that you're jumping back into the XoXoOutliner code! I can't wait to try to hook it up to HyperScope and play with it some more. It's the best open source DHTMLly outliner out there.

Two things that might help you; for local storage, I just checked in a new storage provider for Dojo Storage that will use the native storage abilities in Firefox 2 if they are available, in an automagic way; if they aren't there, then it will revert to hidden Flash like it's worked in the past. This might be useful for you; don't know what you were thinking about for local storage. There's a page up with better docs on Dojo Storage now at http://manual.dojotoolkit.org/WikiHome/DojoDotBook/Book50

The second thing is I just finished an HTML to OPML outline transformer; right now it adds in the line for the HyperScope XSLT stylesheet, but it can be used by any OPML hackers who want to play with bringing HTML into the OPML world, including for things like experimenting with the kind of OPath, deep linking, and inclusion work you've hinted about in other blog entries. A blog post on it is here: http://codinginparadise.org/weblog/2006/11/html-transformer-for-hyperscope-apply.html . Might be fun for you to use in your hacking.

We still need to grab drinks some time! How is your leg healing up? Hope you had a great thanksgiving.

Best, Brad

Event Delegation based DHTML Drag and Drop?

2006-10-31T21:46:40+00:00

So, as the urge to tinker rises again, I've started playing with YUI and cracked open some XoxoOutliner code again. One of the things that struck me like a bolt from the blue recently is the notion of event delegation. For the most part, I've treated event bubbling as a nuisance, except for where it came in handy in keyboard input handling. But avoiding the need to individually track events on every list element on the page via instantiated objects and methods is hot and forehead-slappingly obvious once you've seen it working. Just implement one set of event handlers on the outline root element (ie. UL or OL), and do the right thing as events come in from the list child elements.

Thus, while I can do outline node expand/collapse with ease, my goal is to implement an event delegation based approach to dragging outline nodes around. Apropos of that, I've been poking at how event delegation it works in YUI. While there's nothing much special about it in general, I've run into a bit of a snag when it comes to drag-and-drop.

See, YUI comes with a rather nice drag-and-drop abstraction that I'd like to use. That abstraction, however, requires the instantiation of lots of little objects - one per outline node. Definitely not delegation-based: The problem is that I'll be adding and removing outline nodes at whim throughout the lifetime of the page, which would require lots of care to ensure that I'm maintaining drag-and-drop wrapper objects. I've done that a bit in XoxoOutliner; it sucks.

So, I was able to build part of a delegation-based drag tracker from scratch. I'm sad to lose the other niceties that the YUI DnD offers, but it works. The problem now, though, is the drop part of the equation. I can drag elements around the page until my heart's content, but I can never quite tell in what context I've dropped it.

I poured through the source code for the YUI DnD implementation, as well as the implementation of a few other JS frameworks' DnD offerings. The general solution seems to be calculating and caching the page coordinate regions for each element relevant in the drop operation. In my case, that usually includes every outline node on the page. That's easy when you have lots of little objects instantiated - you just iterate through them and match up regions with the coordinates where the drag stopped. But, part of my win with event delegation was that I don't have to maintain a pile of objects.

Oh yeah, and did I mention that I don't assign IDs to all the little list items making up my outline? And that I don't particularly feel like doing so? That also throws a monkey wrench in the YUI DnD paradigm.

The only idea I have so far is that I need to at least build a cache mapping regions to outline nodes, and keep it fairly well maintained. That should be lighterweight than a fleet of DOM event wrappers, but still annoying. I could trigger a cache refresh when a drag first starts, but that will probably drag down UI response. I could trigger it whenever the outline changes, but that's just moving the burden. And then, who knows how I'll map ID-less elements to their respective regions, while ensuring I stay clear of memory leaks. That's still semi-voodoo to me, and I feel ashamed of that.

Eh, maybe it won't be as bad as I think. But, to anyone who understands what I just spewed: Any ideas?

Archived Comments

Nick
2006-11-01T00:19:27

You know, Marty and I were talking tonight about how some of your posts make NO sense to us. She even said "it's like he's talking another fucking language".

Damn Programmers!!

--nick
Gareth Andrew
2006-11-01T18:57:21

I'm sure I've missed something obvious - why can't you simply use the same event delegation approach on the mouseUp event? getTarget should return the node you've dropped on, what else do you need?
l.m.orchard
2006-11-01T19:18:52

Gareth: Yup, the issue I have with the mouseUp is that getTarget gives me what was under the mouse when I let go of the button - namely, the element I've been dragging around, not the element under the thing being dragged. In addition, for user feedback during the drag (ie. at mouseMove time), I want to know what's under the dragged element - so I can highlight the potential landing spot, etc.
Ben
2006-11-02T13:10:53

I presume the problem is that, given a point on the screen, you want to find exactly what DOM element is beneath it.

I found a solution for this in my proof-of-concept structured editor http://www.fluffy.co.uk/stediting/ -- you need to know where the mouse was clicked to insert the caret.

It might be a start anyway.
misuba
2006-11-06T02:40:19

I think the dragdrop stuff in scriptaculous would get you most of this... at least, it is good about draggable elements having handles you can specify (e.g. child elements of the thing you want to drag), and flexible about what sorts of targets you can land on.
l.m.orchard
2006-11-06T19:06:03

misuba: Nope, the Scriptaculous approach gives me problems too. The first is that I don't like Prototype, unless they've stopped messing with Object.prototype since I last checked. The second is that DnD in Scriptaculous involves juggling lots of little objects ilke most other DnD implementations I've found. When I have an outline of 100's to 1000's of items, that really blows. I think I've got an approach working now that's significantly lighter weight.
misuba
2006-11-06T22:26:55

I'm interested to see your approach. (FWIW, Proto hasn't touched Object.prototype for a couple of versions now.)
Justin Meyer
2008-05-31T07:43:52

I am working on the exact same issue for JavaScriptMVC. If you haven't seen JavaScriptMVC, check out its trunk. In the controller2 plugin, you'll find a really nice event delegation library.

I've only thought of 1 way to do event delegation drops. It's crazy, but it might work.

First, style Drags so their z-Index is LOWER than the Drops. You want your mouse to hit the droppable areas.

Once a droppable is moused over by dragging a drag to it, it creates a transparent element in the exact same position as the Droppable. It also lowers the droppable z-Index. This gives the appearance that the draggable is over the droppable.

To simulate movement, the transparent droppable's mouseovers are sent to the draggable.

On mousing out of the transparent Droppable, it sets everything back to normal.

I haven't tried this yet. The biggest issue I see is that when you grab a draggable in the middle, half of it will be behind the droppable until the mouseover happens. This isn't ideal, but maybe there is something that can be done about that.
Batiste
2009-01-20T10:06:54

A jQuery solution for Drag and Drop with event delegation :

http://batiste.dosimple.ch/blog/posts/2008-05-18-1/jquery-drag-and-drop-and-resize-event-delegation.html

Futility in alternate pasts and futures in human augmentation

2006-09-06T19:57:42+00:00

While it’s great to draw inspiration and ideas from the past, recreating the past in the hope that it becomes the future seems like a futile idea. Does anyone really want to return to a command-line interface to manipulate documents? It’s designing for a past that never happened, one where we all became computer scientists and enjoyed manipulating documents via arcane commands.
...
A better, more productive, use of time would have been to say, what inspiration can still be gained from Engelbart’s ideas? There’s still a lot to be gleaned from his 1962 (!) paper [Augmenting Human Intellect](http://www.bootstrap.org/augdocs/friedewald030402/augmentinghumanintellect/ahi62index.html). How might some of his thoughts on collaborative intelligence be implemented in our world now, in 2006, within the technology we have now? That’s the question waiting to be solved.

Source: adaptive path » blog » blog archive » The Futility of Designing for an Alternate Past

Allow me to engage in some Devil's Advocacy here - although I really am an Engelbart sympathizer:

Consider a program like Microsoft Word, with all its ribbons and toolbars and menus and animated assistance. When you first started working with it, you probably spent time navigating these visual and guided parts of the user interface to get your job done. But, after awhile, you probably discovered keyboard shortcuts and accellerators - CTRL-s to save, for instance. These have likely been invaluable in speeding up your work and helping the application get out of your way. So, having reached this point, do you ever really have a use for the "user friendly" bits anymore? Or, have you graduated to "manipulating documents via arcane commands"?

What if this application had never sugar-coated things and had instead optimized for efficiency and ergonomics in daily expert operation, trading an "intuitive interface" for an offer to incrementally train on its necessarily complex functionality? After awhile, you'll have it all down, and be ready to shed the training wheels.

What if - instead of a maze of menus and toolbar icons - your mouse just had dozens of easily-accessible buttons? You're used to only having a left and a right click from which to choose. If you've splurged, you might have the more expansive choices offered by a fancier pointing device. But, what if you had a chording keyboard under your off-mouse hand, offering an order of magnitude more mouse pointer actions?

For example, how about a "delete word" mouse button? Or a "copy sentence" button? Or maybe even a "jump to the selected link with a custom view filter" button? The important part is that these commands act immediately, just like a mouse button, upon whatever's under the pointer. There's no left-click then CTRL-x to Cut - no, you just point at a word, and say "cut that". There's a lot of power and efficiency here.

These "what if" scenarios are not just wishful thinking, though. They're what Doug Engelbart and crew implemented. These are things I picked up after being invited to try a hands-on self-guided tour of Augment. I only wish I'd had a chording keyboard to get the full experience. The interface was like a mouse-heavy VIM, with verb-object command patterns and structured document interactions. (Or, rather, VIM is like a mouse- and outline-deficient derivation of Augment.)

The basic core of this facet of Augment is this: Computers are powerful tools with great potential to augment human intellect. As such, they offer a lot of complex functionality. But, human beings are trainable, and can assimilate this functionality. Once assimilated, it's best to squeeze out all the performance you can. You don't see today's degree of computer "user friendliness" in chainsaws, tanks, jack-hammers, semi-trucks, or fighter jets. These things are necessarily complex and require training. Why should the most powerful of intelligence enhancement tools offered by computers be any different? Of course, you generally won't lose a limb to a computer, but you might be mentally impaired or lose valuable work in the process.

This is, I think, one of the still-relevant central facets of Doug Engelbart's ideas that could use some re-examination today. It could just be because I'm an übernerd who thinks it's fun to self-train on things like VIM and Augment, but I also think that there's a lot of potential to be unlocked once you clear away expectations of "intuitive interfaces" that are decidedly not nipples.

And, since I've admitted my recently acquired semi-addiction, consider World of Warcraft as an expert application. Advanced players could never succeed by navigating a complex yet "friendly" UI to invoke various spells and skills and in-game actions. Just take a look at some of the customizations and UI revisions being offered at this site. Some configurations of this game smack me as eerily similar to the principles of Augment. In fact, just this weekend, I was considering blowing the dust off this keyset controller I used to use with Everquest, years ago.

Then again, maybe it's a matter of intensity. Coordinating with a 40-player guild to slay something from the molten bowels of the earth is a slightly different activity than composing a memo or even a few-dozen-page report. On the other hand, I really would've liked to strip away most of the Word interface while writing my two books. And someday, who's to say that online interpersonal collaboration in the general case won't more closely resemble a World of Warcraft raid? Having just read Vernor Vinge's latest book Rainbows End, he makes a lot of intelligence augmentation and collaboration tasks look just like WoW.

Archived Comments

Bill Seitz
2006-09-06T21:58:26

The fighter-jet metaphor is interesting. Obviously a fighter has a narrower scope/focus than the general computer. But perhaps there's a narrower technique/practice of intelligence augmentation that warrants a more specialized/locally-optimized interface design.

But then maybe the Engelbart work isn't focused enough on a particular context and associated method-of-use?

To relate this to a similar software offering, how important are the specific Compendium features compared to the process of IBIS?

And a big factor in the ChangeFunction is how critical the problem/pain is being solved by the new offering. Can you convince people that there will be a pay-back for learning to use HyperScope that compensates for the investment, compared to other uses of your time?

Let's put it this way: if you were picking between 2 start-ups to invest in, how much weight would you associate with 1 of the teams using HyperScope? How does this compare to betting on a dogfight where 1 party has an F-15 and the other a Cessna?
phil jones
2006-09-07T22:22:40

Very nice post.

And Bill's comments too. They remind me of the LEO editor for Python which was always touted as having great productivity benefits if only your team would undergo the three month training required to use it properly. Not sure if it ever took off or could. But there's something nice about the idea that LEO empowered programmers could outperform the norm.

I'm convinced that there are certainly productivity improvements available to power-users, beyond anything currently dreamed of, once we step away from the assumption that "ease of use" equals "1-to-1 correspondance between functionality and UI objects".

As every nerd knows, real (interesting) productivity, comes from higher levels of abstraction. And maybe what's really important about the outliner tradition (from HyperScope to MORE / UserLand / OPML to LEO) is that it remains loyal to this notion. When you collapse a block of text and ideas down to a single-line, you are essentially abstracting away from that detail and working with the higher-level description.

OTOH, the Xerox Parc tradition of the GUI and direct manipulation, lost this core ideal. (At least as it was spread via Apple and Microsoft, although obviously you can probably do all sorts of powerful abstractions via a Smalltalk interface)

I'm pretty sure that this insight is general. The really interesting innovations beyond HyperScope are going to be new ways of giving the power-users yet more abstract ways of manipulating their information. Either by folding more of it together as complex aggrogates, or allowing large-scale cross-cutting processing. (Maybe style-sheets in Word are the only other surviving popular example.)
Harold
2006-09-08T22:45:31

Y'know - this is what it is like to be an Emacs user. I've come to the opinion that the set of document formats you can work with, and the set of commands you can perform on them, should be somewhat-to-completely separate from the UI.

That way, you can have a learners/beginners UI to get people up to speed, then they graduate to the intermediate UI that assumes knowledge of things like C-s, C-o, C-q etc. And of course, if an application follows strong UI design guidelines, experienced computer users might be able to start a new program in the intermediate UI. Gosh! C-o opens a file in Excel too!

Then, I think there should be a choice of expert-level UIs. For example, VIM and Emacs have both grown together (VIM started small, light and fast, and Emacs stared with everything AND the kitchen sink), so that they both represent reasonable choices for a power-user's text editor.

This is also one of the problems that web developers are working on (or working around). Google and others have started introducing intermediate level UI features in web apps (like shortcut keys), but try building a site that looks and feels like WoW...
Ben Tremblay
2006-09-12T20:35:09

What still catches me with systems like Augment (I would call Neuberg's new incarnation "hyperScope".) is precisely the mousing.

The war is long lost but I recall with fond pleasure how I blew a Word user away by using WP5.2 ... ^F6-P boom And when I rolled out the functions I'd cobbled together with WP's lovely macro language? Sonic boom. The key in that situation was that I had a large number of unique tasks and a small very number of tasks carrying a huge workload (MILSPEC change management). So it was ideal for hot-keyed macros: like shooting fish in a barrel.

For one thing, unless I'm reading passively or doing some flavour of CAD my hands are nowhere near the mouse. Or, to invert that, when I'm keyboarding I have to routinely suppress my resentment with reach, swivel, click, drag, select, click, select ... interminable menus and options bla-bla-blah, and nowhere muscle memory comes into play. But even with that aside, to have to right click and then select Delete from a menu /after/ having dragged to select a block ... I can outshoot that action stream using keystrokes anyday, if the app allowed me to.

I don't disagree with the fundamental insight ... far from it. But we've just barely begun to implement the foundational cognitive ergonomics. (I was gratified to see in one thread that Brad explicated his having moved Help to the upper right ... cuz that's where it is most often. When it works it works cuz it works. Tradition is sometimes/often arbitrary; life's like that and we should sometimes just suck it up.)

Harold's point about expert users is, I think, key. It's merely foolish to impose a system that makes good use of habituation onto a newly arrived visitor. I'm quite sure that attentive study shows a clustering or quantum of user intention and expertise ... until and unless we contrive some seamless continuum (a terrible distraction inspired by naive perfectionism) we should focus on differentiating expters from n00bs (no diss) and serve both well. "Intermediate level" sounds quite appropriate ... so long as this isn't just a maelstrum of fish/foul goat/sheep confounds.

Alternatively we can always fall back on the old TRW concept of making people think more like machines. There might be funding for that. ;-P
Ben Tremblay
2006-09-12T20:39:47

Muscle memory just jumped up and reminded me of this: in a situation where I was doing Print Preview a gazillion times a day Shift-F7 6 was as effortless as breathing. snap
Edward Vielmetti
2006-10-18T06:00:17

I can speak as a former Emacs user and coder that the only reason I gave it up for vi was that it hurt my hands too much to make all of those funky keyboard chords, and it started to hurt my head to remember all of the time-saving things I had built.

Classically there's a tradeoff between the ease of typing something and the amount of think time you have to put into remembering what to type.

One thing I am annoyed by on too many blogs is the inability to tab from the comment field to the "submit" button, which forces a mouse event and a scroll event.