0xDECAFBAD - Tag: python

Badger 2: Return of the Mustelidae

2011-07-18T15:03:13+00:00

TL;DR: A year ago, I said I wanted to build a badging service for the Mozilla community, so we could all celebrate how awesome we all are. Well, I putzed around and didn't get far, so I'm starting over with something more immediately useful.

(This is an awesome illustration by Sean Martell, used entirely without permission. I'll take it down if anyone gets mad.)

Almost a year ago, I wrote about building a badging service for all of Mozilla.

In a nutshell, I left the 2010 Mozilla Summit wanting to build a tool that empowered people to craft social objects representing achievement and gratitude, and give them to each other.

Since then, I've played around with a prototype in Django with Pinax. But, I abandoned that thing due to getting super busy with the Firefox 4 release. It also didn't help that I was a Django newb and that I got caught up in screwing around with shiny things like Pinax and Lettuce and neglected the actual feature set.

So now, finally, I'm picking the thread back up again. This time, with a little less ambition and a few more practical aims.

Make something more useful right away

While I have spent some dangerous time screwing around with shinies like Vagrant, Puppet, and Playdoh, I think I've gotten that mostly out of my system and have gotten down to work as of last night.

Rather than build a Grand Central Station for badges, I've reset my goals to a progression of practical results:

Build a reusable Django app that supports some basic use cases of creating and awarding badges to users.
Add Activity Streams narrating badge claims
Add support for PubSubHubbub with Badger activity streams
Add a REST API, maybe with OAuth
Accept badges for local users, via PuSH from external sites
See what the Open Badges Project is up to, and how Badger can fit in with their goals.

The idea here is to make Badger a practical, reusable, drop-in app for existing sites—preferably (but not necessarily) the Playdoh-flavored sites we're building at Mozilla. Then, incrementally and optionally empower a Badger-enabled site to become a badge hub by accepting reports of badge claims from other Badger-enabled sites.

Take baby steps to federation

So, for example, let's say we dropped Badger into a few sites like these:

At first, Badger would just augment those sites with badging facilities for their respective user bases. That could fill some needs right away.

Then, say we built a site that offered personal profiles to members of the Mozilla community. With the Activity Stream and PuSH augmentations, users could connect their profiles with each of the other Mozilla sites they use. That way, whenever they claim a badge out there, it gets delivered to their community profile.

Why bother federating at all?

If the only cross-site goal to build an all-in-one display case for badges on a community profile, I could just build JSON feeds and wrap them in a JS-based widget. In fact, a JSON-formatted Activity Stream will work nicely for that. Drop that into the profile page, and we're done.

There are lots of reasons to federate. But, at least one thing not supported by this solution are meta-badges. That is, badges awarded for badges.

Let's say you close 100 bugs on Bugzilla, answer 100 questions on SUMO, edit 100 pages of documentation on MDN, and author 100 add-ons. And, let's say you get badges on the respective sites for all of those activities.

Well, now, I'd like to give you a badge on your community profile for being a Mozilla Superhero. I could just do that by hand, having observed all your greatness. But, I'd rather the system just do that automatically.

If your community profile treated all your badges from your chosen affiliated sites as first-class citizens, then I could build the meta-badge trigger right there.

Of course, I could also write a program that monitors badge feeds from various sites. Then, use the community profile site's Badger API to issue a badge when the conditions are met. But, as maintainer of the monitoring program, I'd have to keep track of what feeds where count for which people, which could be a mess.

There's still some thinking to do here.

What about identity and co-opting badges?

I'm still thinking through how someone connects a profile to other sites. As in, how can Badger ensure that the owner of the community profile is also the owner of a badge claim feed on MDN? Maybe some kind of OAuth dance that results in a PuSH subscription between sites?

Consider the least-capable federation case: a Badger-compatible feed hosted as a static file on Amazon S3. And, by Badger-compatible, I mean an Activity Stream that contains badge claims. You shouldn't need my specific implementation of Badger-in-Django to produce one of those.

Although PuSH is still the ideal case (I think), occasional polling of an Activity Stream feed should still be supported. If a user connects that up to their community profile, we just have to take their word for it. We'd rely on community members to flag stinkers trying to claim badge feeds that aren't theirs.

Beyond that, there's probably some crypto-voodoo we can invoke here, for cases where badge claim validity really matters—as opposed to the fun and games I have in mind for Mozilla.

What about the Open Badges Project?

From various conversations over the past year, it sounds to me like the Open Badges Project is getting the parts in place to do even more interesting federation of badges that gives ownership of the badge hub to the user.

In that scenario, you have a backpack in your browser and can carry your badge claims with you between sites with no prior coordination between the sites themselves.

This sounds like something in which Badger should be able to participate, eventually. If not as a hub, Badger should at least offer the tools to create and award badges that work with the user-owned hub scheme.

So, what next?

Now that I've written all this out, I need to hack on it. If you're interested in this stuff, let me know. That way, it'll look like someone might actually find this stuff handy when I get something working!

Archived Comments

David Boswell
2011-07-19T01:18:24

Very cool to see an update on Badger. I'm definitely interested in this and would love to see something like this show up in the Mozillians phonebook that will be ready soonish :) David
Brian King
2011-07-19T15:12:01

Over at the Mozilla Reps program, we are discussing the possible use of badges - https://wiki.mozilla.org/ReMo
Gen Kanai
2011-07-19T15:48:22

Agreed with DBos. Would love to see Badger as an official service, especially with the new Campus Reps program that Jason Haas is rebooting as well as with the Mozilla Representatives program. Also you might get a badge for contributing to the Join Mozilla program? So many great uses for us.
interstar
2011-08-05T03:15:39

Hi Les, Very very interesting. I was just thinking that there needed to be something like the Open Badges project today. (I'd even got as far as knocking up a data-model in web2py.) And then I saw this. I quickly cloned and tried out your app with an ordinary Django project but it's blowing up without jingo and I guess other Playdoh related modules. Are you committed to Playdoh for Badger2 ?
- Les Orchard
  2011-08-05T18:05:31
  
  I'm committed to Playdoh insofar as my initial users will probably be my co-workers at Mozilla, who are all building Playdoh-based sites. But, there's no reason why it shouldn't eventually work with vanilla Django. It's using core template loaders and shortcuts, so switching over to standard stuff shouldn't be an issue for the views and models. I'm hoping at some point to put together some more Django-conventional template tags and templates toward that end, unless someone gets to it before me in a pull request (patches welcome!). Really, I think the template tags are the more important thing, since any templates out of the box will likely get overridden on a real site.
Dr. Nellie Deutsch
2011-09-02T14:07:45

I love the idea, Leslie. I've been heading a self-funded organization that provides free online professional development workshops for educators around the world since 2006. We need badges for our graduates. Let me know how I can be of help.

Firefox Sync server on Google App Engine

2010-07-05T23:00:56+00:00

TL;DR: I built an implementation of the Firefox Sync server API for Google App Engine.

To celebrate Independence Day, I figured I might take a shot at liberating Firefox Sync from the tyranny of Mozilla's servers.

Thus, over the past few days, I've built a sync server using the 1.0 Sync API, hosted on Google App Engine.

I lied about the tyranny thing, though—I just wanted to say something clever about the holiday. In reality, with respect to Firefox Sync, Mozilla has done all of the following:

Published the Sync API spec;
Released the source code for the server used in-house;
Explicitly included the option to use a custom server when setting up sync in the browser.

This means that, although Mozilla offers servers to go along with Firefox Sync, you're totally free to take your data elsewhere. Since your sync data is encrypted and practically opaque to the server, there's no direct profit for Mozilla in offering free sync hosting—not even through any clandestine data mining for devious purposes. It's just that sync makes Firefox a better browser, and somebody has to run some servers to make it work.

So, there's every incentive to make it easy for you to switch sync providers and stop freeloading on Mozilla's servers. Building a server on Google App Engine means I can freeload on Google's servers!

I kid, of course. No one's really complaining about freeloaders, and App Engine has quotas in place to head off any serious mooching—which is why I'm not telling you where to find my sync server deployed on Google App Engine, by the way.

No, I did this because:

Firefox Sync and Google App Engine are interesting and important technologies;
I've already done a bit of work on the PHP-based Firefox Sync server at Mozilla;
I really wanted to take a break from PHP and spend some time with my old friend Python.

There are, of course, a number of bugs in this server. But, it seems to be working between a number of machines and browser profiles I have at home. Things are really in need of optimization, it suffers from my inexperience with App Engine, and I keep running into those aforementioned App Engine resource limits—especially when updating or deleting large numbers of items (ie. 1000's to 10000's of items).

Pull requests and issue reports on GitHub are welcome!

A next step I'd like to take with this thing is to revisit another old friend, the desktop web app server. (Also known as the desktop website.) It seems to me that it would be interesting to scale this server down to a household appliance—say, just for use by my wife and I.

I'd be especially happy if the work I'm doing for a Google-hosted app could be self-hosted at home. Seeing as the development environment for App Engine runs on my laptop, I'm willing to bet I can hack the whole shebang into a simple, special-purpose app to download and double-click on a home desktop PC for use as your sync hub.

Anyway, check it out and let me know what you think.

Archived Comments

James Socol
2010-07-06T01:28:02

Assuming you already had a network and a Mac Mini or something at home, it would be pretty interesting to run your own small Sync server just to keep your home computers together. Maybe even use localtunnel for when you're away.
Mike
2010-07-06T01:36:37

You are a real-life hero!
l.m.orchard
2010-07-06T01:52:00

@James: Yup, exactly that. Maybe even roll in some SSL and UPnP port forwarding for easier setup behind a home router. And, if there were a Windows version, it could run on that cruddy desktop back in the spare room.

@Mike: Naw... this guy is the real hero, even John Lilly agrees!
Crash
2010-07-06T09:20:54

Is there already a App Id for this tool?
l.m.orchard
2010-07-06T10:18:57

@Crash: Sure, there's an app ID for my instance of this sync server. But, as I said in the blog post, I'm not sharing it. At least, not until or unless I get the quota usage down to a point that I wouldn't exhaust the free hosting limits.

It's pretty easy to deploy your own sync server on AppEngine with the source, though.
Peter Petrov
2010-07-06T13:18:11

@Leslie: The app ID of your instance is visible in app.yaml, so you've shared it anyway :)
l.m.orchard
2010-07-06T17:18:47

@Peter hah! Right you are! But, at least you had to look at the source of the app first to figure that out :) That is mostly the point of this blog entry after all. Anyone who does that and then uses my installation anyway will probably be sad when I regularly blow away data and eventually make it invite only
Tobias
2010-07-07T00:24:41

Neat! TyphoonAE http://typhoonae.googlecode.com might help you to build your household appliance.
Kumar McMillan
2010-07-07T20:59:25

Hey Les, this is super cool! Google App Engine's Datastore API is still very shaky though. In fact, it's been so bad lately that I've been considering porting one of my heavily used apps over to something else. Thankfully, this post suggests that fixing the Datastore is their top priority: http://googleappengine.blogspot.com/2010/06/datastore-performance-growing-pains.html
Robert Kaiser
2010-07-07T21:21:51

Well, I actually feel more comfortable with my data being on Mozilla servers than on Google servers - even with the decreased trust I have in the Mozilla organization, I'd trust it more than Google any day! ;-)
Chris Arndt
2010-09-28T11:06:55

Instead of porting GAE to your desktop, why don't you just port your app to plain Django? Django should run off your desktop with no problems.

A Curmudgeon Playing with the Zend Framework

2007-07-09T20:21:41+00:00

Because it's been awhile since I last surveyed the landscape, I played around a bit with web frameworks this weekend. Mainly, I tried getting something simple working with the Zend Framework. After intermittent play and much studying of documentation between Saturday and Sunday, I got to "hello world."

Unlike other web frameworks of late, Zend Framework doesn't come with any magical CLI tools to build out directory trees and / or generate code. Being somewhat suspicious and curmudgeonly about that sort of magic to begin with, this actually struck me as an attractive thing about ZF. In fact, what I really liked about ZF at first glance is that it's less of a monolithic system and more of a bucket of parts from a single source that fit together decently.

And, all my favorite parts are there: Among other things there's config, logging, a front controller with URL routing, template-system agnostic view support, and somewhat non-magical DB / model support.

These were all things I'd grabbed from PEAR for my more recent PHP mini-framework play, but I figured there should be some benefit from finding them in a unified collection like the Zend Framework.

The experience was somewhat similar to rolling my own, though. I stumbled through some docs and eventually found some blog entries and a tutorial. It's all very loosely-coupled and somewhat DIY, which is actually what I wanted. The experience has left me wondering if I really should have wanted it, but I'm guessing it'll go easier from here on out.

In the meantime, though, I feel like I have a decent grasp on what all those loosely-coupled parts are doing. I've skimmed through code, traced wires, broken things and watched the errors.

See, my problem with a lot of modern magic-imbued web frameworks is that I have a deep distrust of what they're doing. They come with a lot of moving parts doing things that I don't see - and when I do finally see them, I tend to run into decisions with which I disagree. I see abstraction where I think the gritty concrete needs to be left bare. I often see lots of astronaut acrobatics that just have to be burning tons of cycles. I see further acrobatics done to compensate, unrolling loops and pre-parsing config files - which requires further layers of magical tools to generate code and caches.

Too many moving parts wherein assumptions can hide, later to bite me when we're hundreds of miles into the post-launch desert and trying to fight scaling fires or develop a new feature with some weird angles to it.

For example, consider Symfony for PHP. Their configuration system runs PHP code that results in YAML. Parsing that YAML, along with cascading layers of other generated YAML, then drives a PHP code generation stage, composing all the static code necessary to bootstrap the framework with each request. There's an amazing amount of flexibility exposed by that system, and it's quite clever - but my curmudgeonly side catches whiffs of insanity.

Maybe that's really the issue - I've been doing this stuff long enough to have written my own one of everything, so I've got my own opinions and I like to have a grasp on everything. I've written front controllers, templating systems, and model abstraction layers. I've worked in a half-dozen or so languages doing all the above. I don't yet accept that web app development has passed the da Vinci point - a single developer should be able to grasp the structure of a web framework in mind without getting lost in too many fractal branchings.

Anyway, I'm rambling like an old grouch, but I think I've reached my point. Hopefully more actual experimentation soon.

Smarty is on my evil list

2006-01-08T01:33:47+00:00

Although I have my grousing about PHP in general, I have to say that it makes much better template language as-is before pulling something like Smarty into the mix.

At this point, I consider Smarty to be evil—though I'm willing to admit that I haven't grokked it fully yet. And this is coming from a templating addict who has been at times in love with Template Toolkit in Perl, Velocity in Java, and Cheetah and Woven and ZPT/TAL in Python.

WordPress seems to have put native-PHP-as-templating to good use, so I've already got a favorite for comparison. But the use of Smarty in reBlog rubs me the wrong way. Of course, reBlog has a lot of highly-abstracted MVC architecture going on. And this, itself, is something I haven't much in my PHP explorations thus far. (In fact, I almost hate to see PHP chopped up in this J2EE-esque fashion, but that's another story. Oh, and reBlog rocks, by the way.)

But either way, Smarty makes this right-handed hacker feel like he's writing with his left. It seems to have a lot of weird conventions and terminology for which neither knowledge of PHP nor knowledge of other templating systems in general seem much help. It could just be a learning curve, but to me it seems like an odd sideways curve at the moment.

In fact, you know what? Smarty reminds me a lot of DTML in Zope. I hate DTML—it, also, is most firmly on my evil list.

Yuck. Again, it could be just that I haven't known Smarty long enough to like it... but yuck.

Archived Comments

Luka Kladaric
2006-01-08T11:55:40

I agree with you... I used Smarty in one project and am now using it in another work-in-progress... and I have to say, the decision to use it with the second was a bad one... Smarty is pretty cool for simple templating, but when you need raw power, it leaves you shipwrecked... I had to turn to some really black magic to get things done... Anyway, a rewrite of the first project is in negotiations and I'm 100% sure I will NOT be using Smarty there, either.
kellan
2006-01-08T14:34:57

Agreed. Using a templating languages with PHP makes no sense. (though I'm not sure the fact that WP does or doesn't do something makes it good design)
l.m.orchard
2006-01-08T15:03:33

Kellan: Oh, no, WordPress isn't an automatic gold standard for me... There's plenty that could be better about WP. (Not having checked out 2.0 yet) But, I just had a much better time setting up the templates for this blog than I have had in hacking around with templates implemented in Smarty.
Bob Aman
2006-01-08T17:41:17

Me, I just wish there was a templating system for Ruby for me to fall in love with. With all due respect to the Rails people, ERb just doesn't really do it for me.
Aristotle Pagaltzis
2006-01-08T21:04:14

Bob: port Kid to Ruby. :-)
Keith Gaughan
2006-01-08T21:28:05

What's moronic is that people go to all this trouble building templating systems when all you really need is extract(). Sad, really.
Bryce
2006-01-12T13:33:10

PHP as a template engine is more evil than Smarty. The main issue is that template authors may not be trustworthy. Once you include or eval() a PHP template the app is no longer in control and I don't see any way of structuring a PHP app such that it cannot be subverted from within.

The only defense seems to be using the tokenizer functions to build a PHP parser that validates templates against a "whitelist" of allowed operations. Savant's Restricted Compiler is about the only implementation out there, but it's not quite complete and seems to be more of a proof-of-concept than battle-tested code.

Whatever Smarty's faults, it solves the untrusted templates problem nicely.
l.m.orchard
2006-01-12T14:27:42

Bryce: Oh, I totally understand the implications of offering too much capability to template authors. That's just how much I dislike Smarty—I'm willing to risk it, rather than put up with that particular implementation. On the other hand, in my situation, I'm on the same team with the template developers, so we're all on the same team to sink or swim. I know it's not that simple, in principle, but it covers the 80% of the 80/20 equation.
Dougal Campbell
2006-01-12T15:25:25

Yeah, I was never able to get into Smarty either, and I've never been able to put my finger on why. Philosophically, I like the idea of using template engines, particularly if you're collaborating with a non-programming designer. I even wrote one (albeit a very simple one) myself, once. But I've never used any of the PHP template packages that I've looked at.
Bob Aman
2006-01-13T01:10:16

Aristotle: When TurboGears first came out, I almost did, believe it or not. But then I remembered that FeedTools eats all of the time I have available for open-source projects, and I figured someone else would probably do it for me anyways...
Aristotle Pagaltzis
2006-01-13T09:02:38

Hehe. Well, I plan to port it to Perl, in any case. :-)
Brian Grayless
2006-01-19T16:00:24

Smarty is really a waste of time. Some food for thought....

Smarty for Dummies?

Just write better code!!!
Tali Banana
2006-02-01T07:33:41
Well, quite opinionated I see.

Smarty solves some fantastic problems and the cacheing system is beyond fabulous at this level.

I've used Smarty on several projects and have been using it now exclusively for 4 years. Of course, I used others but it was getting sick coming to a new phase in a project and finding out the template system had become RIP....abandoned before their time.

Smarty has done two great things for me, and I'll speak about them until the day that PHP is abolished:
1. Smarty has reduced my code to a reasonably sized, scan-able, masterpiece. Having the logic and presentation seperate is so nice. I recently rewrote one of my old projects using smarty and all of a sudden it made sense! No more php in the html and no more html in the php. And that's something you can't solve without templates for you juniors out there.
2. Smarty's caching has turned monster db queries into non issues. Consider this. A top 10 list from all-time sales. I'm talking about searching a db with sales spanning 6 years for all items purchased and creating a top sales page. Easy! Smarty does it once and caches the results until I decide they need to be refreshed. Took a 4 second execution time down to less than 0.4 seconds. And all without having to implement a CMS or otherwise. Even if you have limitless resources to buy new hardware, caching like this still makes sense. And the intermediate php/html cache is brilliant. Everything is so much faster with smarty.
I can't deny, I love Smarty.
l.m.orchard
2006-02-01T11:43:36

Tali: I'm usually opinionated, but I try to listen & learn :)

The benefits you describe with Smarty would seem to come with any decent web development system. Separating logic and presentation are the general idea behind templating, and some sort of caching is a must no matter what you're using as a presentation-side tool. My beef is that, out of all the templating systems I've used, Smarty is one of the ugliest.
Tali Banana
2006-02-01T17:30:03

Smarty does need some standards redesign on the interface, as does php itself which has been well documented but slow going.

Coming from the PERL background and previous php template systems I've been very happy with Smarty in general, even if there are a few things that could be done to speed up the learning curve.

May be worthwhile jumping on the dev team and making some suggestions.
Aristotle Pagaltzis
2006-02-02T01:30:31

But why would you need a template system when you’re already working with PHP, which is itself a template system? Why do You don’t have to have different languages to be able to put the business logic in one file and the display logic in another.

Perl (not PERL), btw, has the Template Toolkit on the CPAN, which is the spiritual father of Smarty. (And it’s vaguely more defensible in Perl’s case becase Perl is not a template system; though constrated with Perl the deficiencies of TT’s mini language are just all the more painfully obvious.)

By all of which I mean to say that whether people use languages in stupid ways does not mean the language is the problem; not in either PHP’s or Perl’s case.

I also don’t see why you need a templating system to implement caching; in fact, having caching coupled with the template engine sounds like a bad distribution of responsibilities. (Indeed, Template Toolkit has no such facilities.) But there’s the Cache::Cache family of modules on the CPAN, which you can use to cache things completely independently of what is being cached – whether it be the output of a template, or the query results themselves, or anything else, which might not even have anything to do without any output you generate.
M.
2006-02-16T14:13:18

I've good experiences with the Expose PHP template engine (www.kwasd.nl/expose). It doesn't require you to learn a new language, because it uses plain PHP and still offers security, because it contains a tokenizer and, in fact, a small compiler. I used it for some projects and it really offered all I needed.

And for people who wonder why one would need a template engine: I work with designers and I don't want them to mingle application code. By setting up a shared folder which contains the templates, they can safely work on design and never need to touch my controller or module files. Especially since this template engine has a tokenizer/compiler, they can't call unsafe PHP functions (only those that you, the programmer, register).
Michal Migurski
2006-03-05T23:16:25

Smarty appealed to us because it offered a (relatively) clean way to separate the code from the HTML output. It's definitely got its problems. A large part of the choice was based on my perception of Smarty as having much thought & effort behind it, plus I like the idea of templates caching down to plain PHP. I've done a few PHP projects where I used PHP as the templating language, and frequently succumbed to the temptation to stash app logic into the templates where it didn't really belong. Smarty has a pretty extensive vocabulary, but has (so far) resisted the pull to incorporate more real-language features, like PHP did. It's worth remembering that PHP was originally supposed to be a template language for "real" code to be written in C. shudder

When we forked Reblog from Feed On Feeds, we were starting with raw-PHP display code mixed in with database calls, and it was a freaking disaster. One of my first design goals was to make it hacker-friendly, clarifying and documenting as many seams as possible so people could make localized changes more easily. I'd like to be using Python under the hood with Smarty on top, though there's a lot to be said for using commonly-known tools like Smarty in apps where you want to encourage modifications.

And Wordpress.... oh, Wordpress. My eyes roll back in my head when I'm forced to touch its convoluted innards. I pushed hard for a Reblog plug-in API specifically so I could cheerfully deprecate the old-style Wordpress plug-in. =)

Not-so-deep PHP thoughts

2005-12-19T03:50:25+00:00

You know, not that I've been at it for long, but I've had an long-running observation about PHP:

As a language and an API, PHP kinda stinks. It's got more crud than I left behind in Perl, but without all the meta-magic that made it worth it. And, I get a little nervous tic when I think about it in relation to Python and Ruby.

On the other hand, it's ubiquitous and cheap—and because it's in the hands of people with all levels of skill, most popular PHP apps have by necessity managed a high standard of easy install and simplicity. The languages with more meta-magic tend to attract eggheads (myself included) who can forget about keeping things simple.

And, did I mention PHP is ubiquitous?

Archived Comments

Bill Humphries
2005-12-19T06:38:33
1. PHP has some future-proofing in that almost everyone doing web development knows it or can pick it up. That reduces risks for IT adoption.
2. $5/month web hosts have it installed.
Tim Howland
2006-01-10T14:27:54

While I don't care for the crappy structure that the language seems to revel in (SQL in your HTML? WTF?), what I despise about PHP is how hard it is to build and deploy it with a ny extensions to the basic feature set.

If you need to do images, you compile in LibGD, if you want to render text, you get the freetype libraries, if you need something that isn't mysql, you compile that in- and then off you go trying to find the version of libJpeg from 1993, because the ancient version of libGD that's bundled with PHP doesn't work with anything more recent.

Plain-vanilla PHP is ubiquitous, but trying to roll your own on an arbitrary box somewhere and do something that doesn't come with the default PHP that ships with your distribution is a huge and unnecessary pain in the ass. Perl can be annoying, but at least it isn't a monolithic binary- the library system drastically reduces the cross-component dependencies that PHP suffers from.

PHP sucks, and it amazes me that it is so ubiquitous. On the otherhand, so is Microsoft Windows.

IPython as networked command shell on XBMC?

2005-10-25T16:28:01+00:00

Just in case it's already been done, or I don't get around do doing it, hear me oh LazyWeb:

I want to cobble together a version of IPython (or at least an InteractiveConsole) that runs on a network socket for access via telnet. I then intend to upload this to my XBMC-equipped Xbox and run it as a command shell there for access from my LAN. It seems like an idea that should work, since XBMC packs a Python install.

Archived Comments

mike
2005-10-26T02:59:28

I don't have such a toy to play with but I'm wondering why the question wasn't asked, and answered, with ssh+screen?
l.m.orchard
2005-10-26T03:36:57

@Mike: Because XBMC doesn't run ssh+screen, only a Python interpreter embedded in an entertainment center media dashboard. (ie. Not an OS like Linux)
kaolin fire
2005-10-26T06:37:55

when you say 'like telnet' you mean just opening up port 23 and listening on it and executing said instructions? I don't know Python (though coincidentally I had to debug some 2-3yo python today; that was interesting), but I presume it makes it not-so-hard to open server sockets and listener sockets, and then execute the dynamic code in a readloop... but then I don't know XBMC beyond the above link you posted, so this is probably just all optimistic hot air. ;)
l.m.orchard
2005-10-26T10:44:00

@kaolin: Oh, yeah, I think it would / will be about as easy as that. I've just been too busy or lazy to have done it yet, and hoped that someone somewhere had already done something whizbangy with a Python shell-on-a-socket. :)
Chris
2005-10-27T11:50:52

Perhaps PYRO (Python Remote Objects) is a choice, I might give it a try on weekend (on my xbox of course). Though I cannot imagine any real use so far.
Itamar Shtull-Trauring
2005-10-27T18:02:14

twisted.conch lets you hook up a python interpreter to a telnet or ssh server (which are implemented in Python, of course).
l.m.orchard
2005-10-27T19:18:32

Hmm, it's been a year or so since last I tinkered with Twisted... It's tempting, since I remember all the nifty toys I played with in that environment-- and an Xbox with XBMC is just the place for it, since it doesn't have a shell or much OS infrastructure. Though, I was kinda looking for something that used the core Python modules so I didn't have to plunk down a lot of code on the HD.

On the other hand... Twisted could let me do a lot of things on that little box without installing Linux.
Andrew
2005-11-13T23:17:37

What about looking at Nokia's Python btconsole. I run the python script on my phone, and then open a screen to the /dev/tty.NokiaBTSocket and am presented with a console on the phone itself.

I imagine you could grab the btconsole.py or simplebtconsole.py and set it up on XBMC. Depends on what modules are available.

I'm running a REALLY old version of XBMP (predecessor/sibling to XMBC) and have happily been running Xebian and MythTV frontend. However, recently came across the MythTV front-end with XMBC as well as all these new features so will upgrading it myself. However, the fiance' doesn't much like me messing with her "Tivo".

Trac rocks like a really rocking thing

2005-09-30T16:11:09+00:00

Say hello to my new Trac installation!

For a couple of years now, I've been meaning to take a serious look at Trac. But, general level of busy, as well as random obstacles to getting it up and running kept me from getting very far. Such is so often the story with me.

Finally, though, I've gotten an instance running on our server here. All I can say is, "Wow!" This package does everything I ever wanted to cobble together using TWiki, MoinMoin, kwiki viewcvs, and any other pieces I ever looked at (ie. Bugzilla, Request Tracker, etc)

Trac has an amazingly dead-simple macro and plugin system which puts TWiki and possibly kwiki to shame. Just take a look at the MacroBazzar and this hacks collection. And, the system seems pretty cleanly architected—templates pages, simple extension points, SQLite database. I've already been able to easily wade in and make some tweaks of my own under the hood, as well as fix a few macro/plugins which were broken between v0.8 / v0.9 revisions.

Your mileage may vary, of course, but I think I've found my new favorite web app. It's now a replacement for the wiki and SVN repository browser I used to have running on 0xDECAFBAD v3.

Archived Comments

Simon Willison
2005-09-30T16:19:27

I couldn't agree more - Trac is an absolute gem. It's even written in Python :)
Martin Atkins
2005-09-30T18:54:02

My favourite thing is that you can use Wiki markup in the commit log. Sure, it doesn't work if you're not viewing it through Trac, but Wiki markup tends to be unobtrusive anyway.
Brandon Pierce
2005-10-02T19:31:27

My only complaint about Trac is how difficult it is to get installed and going. I consider myself a power user, though I'm not exactly a code whiz, and I still had a rough time getting it going on my OS X machine (and quite frankly never really did get it fully going). I suppose I'm spoiled by all of the great double-clickable installers that people make for these kinds of packages. :) If I knew more I'd figure out how to make one for Trac.
Simon Brunning
2005-10-04T13:47:23

Re Trac on OSX - anyone tried http://trac.darwinports.com/ ?
Richard Bogle
2005-10-09T23:06:50

I had the same grief with getting Trac running and then CVSDude offered it all nicely hosted for me. :) The price is right and the system works and it's exactly what I wanted.

Cross-breeding XSLT and ZPT

2004-12-03T01:15:52+00:00

I've recently been doing some side work involving Zope and, along with the rest of the suite of technologies it offers, I've been happy to be working with Zope Page Templates again. I dabbled with them a bit when they first came out, and a Zope-free implementation named SimpleTAL was one of the core components of the iteration of my news aggregator which came before FeedReactor.

Out of all the templating and content generation approaches I've used, Zope Page Templates are my favorite yet. Pretty expressive, yet unobtrusive; nicely powerful, yet not quite something with which you'd want to write an entire application (and that's a feature, not a bug).

I've yet to be in a work-a-day team that uses ZPT-- but I can see where a lot of production, delegation, and integration issues would have gone much smoother had I used ZPT instead of Template Toolkit for the web app framework I created at a previous company. (Though I do have to say TT2 is very nicely done!) And where I am now, I spend most of my days trying to pummel ASP 3.0 pages into some semblance of logic/presentation separation-- I would certainly dive at the chance to dump VBScript and <% cruft %> for a bit of Python and ZPT. (But, you know, it's a living.)

A close second favorite is XSLT. I've really been hot on it lately, having worked it into the core of FeedReactor in place of SimpleTAL. And in other hacks, I've really come to appreciate it's role as a filter segment in pipelines between REST web services and URL-as-command-line invocations.

Granted, both ZPT and XSLT very different technologies, but they are often used in similar contexts. More than once, I've wished that XSLT was as simple as ZPT (i.e. less verbose and intrusive, more document centered), and I've wished that ZPT had some of the features of XSLT (i.e. ability to be used as a transforming filter).

Reading Ryan Tomayko's description of Kid got me thinking, and googling. One thing I turned up from a mailing list archive asked about an “XSL implementation of TAL?” It struck me as a tad nutty at first, but then I started having inklings that just maybe it could be done. (Whether it should be done, well...) But the kernel of the idea grabbed me: Instead of using TALES path expressions to look up values in Pythonic space, why not use XPath expressions to look up values from a supplied XML document?

This strikes me as such an obvious idea that someone has to already have done it and possibly rejected it for good reason. On the other hand, maybe this is the sort of thing Ryan's thinking about-- I wonder how hard it would be to hack this into Kid? It would give only a subset of XSLT's capabilities in trade for simplicity, and would only offer the “pull” approach, but it would give XML-pipelining to a ZPT-ish technology.

I think this is something I want to look into a bit further at some point.

shortname=crossbreedingxsltzpt

Archived Comments

Ian Bicking
2004-12-04T17:16:09

I've been meaning to write about this on my blog, as I've done something similar. Instead of using XSLT, I've used ZPT for the transformations. It was an application where the user created a document in a WYSIWYG editor, then we wanted to pull information out of the content -- like a table of contents, or a title. To do this I parsed the content into a DOM, then put some objects in the ZPT namespace that manipulated it. For instance, the ToC object took a tag name (through getitem) and returned a list of the content and id of those tags (it created ids if necessary, modifying the content). Then you could easily create a ToC by looking through and creating anchor tags from, say, all the tags in the document. It should be easy to expand with other transformations (all coded in Python, of course). The actual code was only like 20 lines of Python, maybe less, and easy to understand from both sides (ZPT and Python).
Petri Savolainen
2004-12-13T05:21:41

See http://zope.org/Members/DaddyGravity/PT_XPath It would be great to have that in SimpleTAL, too.
Ian Bicking
2004-12-13T11:43:33

Another thing you might want to look at: in the last few days there's been discussion on the ZPT mailing list about an extension to stylesheets (TERSE) for ZPT that introduces transformations.
Bertrand Delacretaz
2004-12-13T15:15:55

FYI, people from the Apache Cocoon and BXE projects are working on similar stuff, in the opposite direction: we're taking TAL-like templates and converting them to XSLT, adding simple "match" templates for declarative rules. It's only prototypes and experiments for now, but the results look promising. More info at http://wiki.apache.org/cocoon/HtmlToXsltExperiments and http://blog.bitflux.ch/archive/further-improvements-on-xsl-tal.html
Manuzhai
2004-12-13T16:07:54

I like XSLT a lot, myself, and I have advocated it in the past on my weblog, but it just doesn't seem to be very wide-spread. Which is a pity, I think the world would be much easier if more people used this *standard* way of templating. One of the bigger problems with it seems to be the verbosity, so I've been thinking of a more compact syntax, kind of like RELAX NG has the .rnc compact stuff. I don't know if it's very feasible, but it seems like that would be not very hard; just have some compact syntax which maps onto a real XSLT-sheet (it could be "compiled" and cached, if need be).
Colin Stewart
2004-12-13T23:47:18

Integrating an XPATH implementation into SimpleTAL shouldn't be too hard as the TALES and TAL implementations share a fairly simple interface. There are 6 methods you'd have to provide to the simpleTAL module and that's about it. I don't know if XPATH is the right approach though - how would things like tal:define work if the paths were pure XPATH instead of TALES? Being able to mix XPATH and TALES would work better I think. A more promising approach would be to integrate ElementTree so that the 'find*' methods were usable from within TALES. Making it so that '/mydoc/root/find/.//searchElement' works would be fairly easy, but getting '/mydoc/root/find/.//searchElement/attrib/firstAtt' to work would require more co-operation between ElemenTree and SimpleTAL. I'll have a think about this though as it sounds like a promising approach.
Paul Everitt
2004-12-14T00:45:18

Chapter 11 of Jeni Tennison's "XSLT and XPath On The Edge" book has a section called "Using Page Templates". It includes an example of using substitution points. Like some of the others in this thread, playing with doing merges using DOM IDs and other patterns. This lets you write "themes" that are simpler than even ZPT, as they contain no non-XHTML namespace elements. Thanks for the article, which also seems to have brought out some interesting comments and URLs!
Bill Seitz
2004-12-14T20:23:08

Don't forget you can run Python within IIS/ASP! http://webseitz.fluxent.com/articles/PythonViaIis
Colin Stewart
2004-12-15T22:49:30

I've put together an experimental build of SimpleTAL that integrates ElementTree to provide some of the XPATH syntax. It's just an experiment, but see what you think. (More detail here: http://www.owlfish.com/weblog/2004/12/15122004.html#20:59:59)

dbagg3 code in CVS

2004-08-05T23:53:34+00:00

I've just dumped what code I have into my CVS repository. So, go ahead and poke fun at it:

http://www.decafbad.com/cvs/dbagg3/

Or, fetch it from CVS:

$ cvs -d:pserver:anoncvs@www.decafbad.com:/cvsroot login
(Logging in to anoncvs@www.decafbad.com)
CVS password: anoncvs
$ cvs -d:pserver:anoncvs@www.decafbad.com:/cvsroot co dbagg3

shortname=dbagg3cvs

Introducing dbagg3, an Atom-powered client/server aggregator

2004-08-05T13:04:56+00:00

Update: I've just dumped what code I have into my CVS repository. So, go ahead and poke fun at it:

http://www.decafbad.com/cvs/dbagg3/

Or, fetch it from CVS:

$ cvs -d:pserver:anoncvs@www.decafbad.com:/cvsroot login
(Logging in to anoncvs@www.decafbad.com)
CVS password: anoncvs
$ cvs -d:pserver:anoncvs@www.decafbad.com:/cvsroot co dbagg3

So, besides the funk, there's a little project in which I've gotten immersed. Here's a teaser diagram:

It's a feed aggregator, my third attempt at such. At present, things are roughly close to the diagram above. Everything is clunky and command-line driven at present--but I've got further plans, like a REST API for feed queries and manipulation of various things such as feed subscriptions and the read/unread state of items. Pair this with an XSLT-driven browser UI, and the possibility of other clients (not the least of include other Atom-consuming aggregators).

The goal is to make a Client/Server Aggregator. Somewhat serendipitously, I just caught Bob DuCharme's xml.com article on Amazon.com's web services, which I really like and have drawn inspiration from in thinking about this new aggregator. Eventually, I want to offer the same sort of XML+XSL style of service that they've put together, along with some futher inspiration from the Atom API.

Anyway, as another teaser, check out this early demo involving the XML this thing has been producing, coupled with some experimental XSL:

dbagg3 demo page

You'll notice that there are three URLs involved here:

The XSLT expects Atom (plus a few extensions of my own), so you can also do things like this:

These are the sorts of tricks I was looking forward to when I started thinking about things like a universal blog transfer format and rose-colored glasses. XSLT used like this could just as easily produce a blog or RSS 2.0 content.

Anyway, hoping to get some code into CVS by this weekend that's not entirely embarassing. So if you're interested in this stuff, stay tuned. I'm hoping that this thing can provide a base for others interested in feed aggregation--if you just want to play with UI, use the scanning and storage as-is and tinker with XSLT; if you want to play with storage and query, leave the scanning and UI alone; if you want to tinker with parsing... er, talk to Mark Pilgrim.

More soon!

(Oh yeah, and I will be working on coming up with a better name than dbagg3. Unfortunately, I probably won't be coming up with a more visually appealing design for the UI, since what you see is the best I can do. Heh, heh. Don't let your programmers do visual design...)

Archived Comments

Benjamin
2004-08-05T10:22:30

This is great. It seems that this could be the base, for what I've been dreaming/thinking about the last half year or so. In contrast to you, I haven't come up with any code yet (mostly due to the fact, that I don't python yet, and I found no other reasonably good feed parser, than the Universal Feed Parser), but with a name: Roosster Features I'd really like to have in such a ClientServerAggregator are: (a) search over entries the aggregator has seen to date (b) add and arbitrary number of notes to any entry (c) add multiple category-tags to entries (d) interlink entries (I know this could be solved by some 'Related Entries' stuff, but somtimes I don't trust these "automatic classifiers") Can't wait, until I can take a look at the code and see if I can extend it. This would finally be a real reason to learn python. By the way, I really like the interface. Might be, that's because I'm a programmer too?!
Christopher Drake
2004-08-05T18:30:41

Very nice. The demo has me salivating a little. Oddly enough, I like the UI. If you can slim it down, side-to-side-wise a bit, it would also be a great aggregator to rip whole chunks from and feed to a Palm Pilot or PocketPC, as well. They're not as big as they were in the past, but with decent strides every couple of weeks, they still have a lot of following. Your page-width is dynamic, and that means a lot to me, a Palm user. A feature I'd like to see is kind of like Bloglines, where you receive a small box of info from the RSS feed, or what-have-you to give you a taste of an article. If going 1 link deep from any link on the aggregator page linked to a small page with such a tidbit, it would make surfing the articles a lot faster. This would allow a PDA or cellphone to drag an aggregator page 1 link deep with specialised software and get all of the highlights from say, 20 blogs. That could be a major boon, especially for people that connect while at wireless access points on business trips, etc. Keep it up, this looks like it'll rock when you are done.
Christoph Görn
2004-08-08T12:41:08

Hi, and I will try to plug an jabber/XMPP bot into it, so it will deliver the aggragated content rigth to my beloved IM app...