startlingly bad moments in API design

Weka, the machine learning toolkit, has these nice filters that let you change what's in a data set, maybe the features on the instances, or the instances themselves. Pretty useful. One is called "Remove", and it removes features. Here's a case in Weka where order matters when you're setting up the parameters for an object.

Like so: this does not remove any features.

Remove remove = new Remove();
remove.setInputFormat(instances);
remove.setAttributeIndices("7,10,100");
remove.setInvertSelection(true); // delete the other ones.
Instances out = Filter.useFilter(instances, remove);

This works just fine, though:

Remove remove = new Remove();
remove.setAttributeIndices("7,10,100");
remove.setInvertSelection(true); // delete the other ones.
remove.setInputFormat(instances);
Instances out = Filter.useFilter(instances, remove);

How are you supposed to find that out?

goto: a utility for bash

Do you ever find yourself, while using bash, wanting to get to a particular file that you know is way down in a big tree of files -- maybe a big Java source tree, or some other nested file structure? You know you want Foo.java, but is it in src/com/foocorp/package/a/b/c, or maybe somewhere else?

Announcing goto, a bash utility which solves just that problem! Now you can just say:

you@computer:~/project$ goto Foo.java
you@computer:~/project/src/com/foocorp/whatever/path/to$ vim Foo.java

... instead of whatever business you were going to do with locate or find or your IDE, or just manually sifting for it.

Here's the README. Comments, complaints, and patches welcome! (and if you find this useful, I'd be really pleased if you'd let me know!)

Scala article on the Scala site!

I helped Lex Spoon and Toby Reyelts write an article about running Scala on App Engine.

Here's the article! As of right now, it's linked from the front page on the Scala site.

In the article, we mention my Scala/GWT sudoku solver, which is now available on Google Code. It's also live on App Engine right now!

Further steps: Scala/GWT/App Engine/Eclipse

I've been excited about Scala recently, for a number of reasons -- it's a modern language with lots of great features that you might expect if you've been looking at Haskell or ML. It's got pattern matching, a concise syntax, type inference, and first-order functions. The really killer thing about Scala, though, is how well it integrates with Java.

You can very easily have a mixed Scala and Java project, and make calls back and forth; Scala and Java packages sit in the same package hierarchy and compile to the same bytecode -- it all just works, pretty much seamlessly.

So, clearly, the right thing to do is hook Scala up to GWT RPC and run it on App Engine. Let's do that. This is as awesome as it's going to get, at least until the GWT compiler supports Scala and we can do our client side in Scala too.

I assume you've already installed both the Google Pluin for Eclipse and the Scala IDE for Eclipse. I'm using Eclipse 3.4.

Make a new "Web Application" project (with File > New, or just click the blue "g" icon). Click the boxes for both "Google Web Toolkit" and "App Engine" -- our client side will be in GWT, and the server on App Engine.

Right now, our new project has "Java" nature, but not the "Scala" nature. Add Scala by right-clicking the project and choosing Scala > Add Scala Nature. We're also going to make sure we have a copy of the Scala runtime library once we deploy to the server. Find out where scala-library.jar sits (expand out the "Scala Library" container in your project) and copy it into the project's war/WEB-INF/lib.

Now, down to business. There's a RemoteServiceServlet currently implemented in Java, and we're going to replace it with Scala code -- this is the thing that gets called on the server side when the client makes an RPC request. Find GreetingServiceImpl.java and delete it.

To replace it with a class implemented in Scala, right-click your "gwtscalademo.server" package and select New > Other > Scala Wizards > Scala Class. The class we deleted was a servlet and referenced by the web.xml, so give the Scala class the same name: GreetingServiceImpl.

Here's my version:
package gwtscalademo.server
import gwtscalademo.client.GreetingService
import com.google.gwt.user.server.rpc.RemoteServiceServlet

class GreetingServiceImpl
extends RemoteServiceServlet
with GreetingService {
def greetServer(input:String):String = {
return "Hello from Scala, " + input + "!"
}
}
Note that while in Java, we'd have said implements, in Scala we say with. As I understand it, this means that Java interfaces get mapped onto Scala traits. Hip!

Now you should be able to run the app locally -- right-click your project and select Run As > Web Application. Once you've learned Scala and made it do something interesting (you're on your own there), you can deploy it to App Engine!

Woo!

GWT 1.5 release!

Wooooo, new version of GWT! This is huge. We've been working really hard on this for quite a while.

http://code.google.com/webtoolkit

What's new? Check check check it!

reconsidering applets

Applets are a really tempting idea. The JVM is a serious piece of machinery; despite its startup time, once it gets rolling, Java can go fast indeed. And although it hasn't been very widely used, applets and javascript have had a means of calling back and forth for quite a while. It's called LiveConnect. (although LiveConnect may be replaced at some point...)

So, a beefy, fast VM that runs inside the browser, coupled with a nice web UI, and no manual installs for your users -- big win? Maybe! On considering this, one of my first thoughts was to put Jython into an applet and build something akin to JES -- a little Jython IDE in the browser! The kids would love it! So I set out to make this happen, or at least build a proof-of-concept.

I ran into a serious snag, though -- the Jython interpreter works by compiling to bytecode and bringing it up with the class loader, the latter of which applets aren't typically allowed to do. Special permission can be granted with a Java policy file, but that's a very high barrier to entry. It's certainly not the sort of thing a person might do on a public terminal. Signing the applets may also be an option -- but that's also a pain, and anecdotally, I think signed applets may not always get the same rights, cross-platform.

Once I understood the permissions issue, I did get a little Jython repl going -- it just hooks "eval" up to some javascript for output. Nothing fancy, but kind of satisfying. For now, though, it doesn't look like Jython IDEs are coming to the browser. Until we rewrite it to not use of the class loader. Or just use some more modern approach.

Some other people have had similar thoughts!

• Interactive Jython Console: this one, for the exact same reasons that mine does, requires a java policy file.
• the ruby-in-browser project. It does exactly pretty much what it sounds like it does, also by means of an applet and LiveConnect. Interestingly, JRuby doesn't seem to use the class loader, it just works! Check out the demo! [thanks for the heads-up, Lindsey Kuper!]

GWT 1.5RC1 -- we're pretty pumped.

Google Web Toolkit 1.5 RC1 is live!
http://googlewebtoolkit.blogspot.com/2008/05/google-web-toolkit-15-release-candidate.html

Get the bits while they're hot.
http://code.google.com/webtoolkit/download.html

GWT 1.5 has a whole lot of wild improvements, detailed here -- but importantly, compiler now supports Java 5 features and produces even tighter code than before, and the UI library has had some pretty serious reworking -- there's a new API for working directly with the DOM in a typesafe way, and GWT apps now come with nice styling by default. We've been working pretty hard.

Oh, and don't miss the new documentation browser :) Much better than browsing the GWT wiki docs by hand.

Share and enjoy!

Design Patterns Ahoy: Violator Pattern

So, at work, every so often somebody brings up the "Violator" Pattern, typically with a slight smirk. Up until recently, I wasn't at all well-versed in design patterns, but this seems like a field that one, as a professional programmer, should be at least familiar with. So I picked up the classic Gang of Four book and started reading through it.

And while, so far, it seems like just good, solid advice about software design -- and at the risk of sounding uncritical, is it really so bad to have common names for commonly-occurring software structures? -- ... no "Violator" pattern is to be found in Gang of Four. Or on the C2 design patterns wiki.

And I wanted to work out for myself what this mysterious Violator Pattern could be. And I figured it out today. In GWT, you can use the JavaScript Native Interface to make calls from code written in JavaScript into methods written in Java. And when you do this, the compiler totally ignores the access modifiers on the methods you're calling. That's the Violator pattern.

A paragon of OO design, verily :)

James Gosling, it turns out, thought harder about type inference than I did

So for a while, I'd been wondering about the type system in Java, thinking that perhaps I'd found a shortcoming -- if you have a more general type (say, "Thing"), and a more specific type (say, "Chair", which extends "Thing") -- then why isn't List<Chair> a subtype of List<Thing>, meaning that you could pass a List of Chairs into a method that takes a List of Things?

The mighty Toby R, in a conversation with me and Strick, shed some light on the situation and led me to understand why this is not the case. The particular use-case is: when you're in the method that takes List<Thing>, you can add Things to that list. And the Things you add could well be Basketballs. So when the method returns, the caller is still expecting to have a list of Chairs, not realizing that there are now Basketballs in the list... so Java cleverly disallows this case.

(also, somehow I missed an anonymous comment, probably from my mother, well-respected for her work on type theory, which explained that exact case)

Now in a purely functional language, where you don't go around getting references to objects and modifying them, I'd like to posit that this wouldn't be a problem -- but perhaps there are other problematic situations? ...

typing about typing about types

As we make the push to handle Java 1.5 features in GWT (it's about time!) I'm having flashbacks to undergraduate Compilers. Perhaps this is unsurprising, considering what I'm working on. Part of the difficulty in the problem is that whether you're talking about types in the code that constitutes the compiler itself, types in the code that the user is going to put in, or types in the output code... we use the same word. Oh my. (much worse, I suppose, is the code you get with the Appel book, where half of the words on the screen are helpfully "ty" or "typ". Jebus.)

But! All of this means that by the next GWT release, you'll likely have some code that I touched in your hands. Woo.

You get spoiled by languages with first-class functions

Earlier today, I needed to take a list of strings and grab all of them that end with ".html". Very simple task. My first thought is of course something like:

[fn for fn in fns if fn.endswith(".html")]

Or even:
filter( lambda x: x.endsWith(".html"), fns)

Or the equivalent Lisp. Y'know, with "loop" and "collect". (Common Lisp has, as they say, the Cadillac of loop syntax)

Or something to that extent. But then I remembered that I was writing in Java, and I ended up writing a for-loop. There's no clean, idiomatic way to say that in Java, is there? Do you have to build up the list procedurally? ...

"If I knew the answer ahead of time, I wouldn't be writing this program!"

On the plane back from San Francisco to Atlanta, I read Thomas and Hunt's Pragmatic Unit Testing. It's a very quick read, about 120 pages plus appendices. And while it's very light, it has what seems like a lot of helpful advice for good design and development practice. To be honest, I've never done much unit testing in the past, thinking "oh, that's what those corporate software engineer guys do -- pssh." But! It turns out that these days, I am a corporate software engineer, and anyway one should always be looking out for new ways to hack more effectively.

There's a bunch of important principles to take away from Pragmatic Unit Testing. The one that stuck with me the most, though, is that when designing and writing code, you've got to think: "how am I going to test this?". The significance of this is not just "oh man, I'm going to have to write a unit test for my method"; it gets back to that generally-understood but oft-ignored idea that every logical unit of your code really wants to be its own method, a modular thing that you can use separately -- because you're going to have to use that same calculation again somewhere else. And Don't Repeat Yourself.

For example: if you have a method that calculates how to do something and then does it (say with a call to another library), maybe you want to make that two methods: the calculation and then a call to that calculation coupled with the library call. This will be easier to test -- you're not really interested in testing a third-party library, just your own calculations -- and as a happy side-effect, your code is now cleaner and more reusable!

Similarly, separating out the backend code from the GUI (two of the things that get my hackles up the most in this life are terms "business logic" and "MVC") lets you properly test the Stuff That Does Stuff on its own. One example in the book hit particularly close to home -- a small GUI application where all the caculations and I/O happened mixed in with the Swing code.

That example, and in fact the whole book, brought on flashbacks to a project I'd worked on recently. One of my friends and I (and he's one of the sharpest guys I know) inherited a fairly involved program and ended up sinking months trying to fix it up. This system suffered from pretty much every pitfall in the book: random silently-caught exceptions, real work happening mixed in with the GUI code, needlessly long, opaque, nigh-untestable (let alone "tested") methods, repetition all over the place. Worse! It was built by a guy who'd supposedly specialized in software engineering -- and he did pretty much everything that Thomaas and Hunt warn against! Not a pleasant situation. I'm sure you can relate.

Of course, I knew at the time that this was atrocious code. But now maybe I'll be a bit more principled in my development, working with a lean towards easy testability. Unit testing will probably be a good discipline to get into.

On the other hand, I've been reading about (and writing a bit of) Haskell. All this murky business of setting up and tearing down state, "proving" to yourself that each function does what you think it does "for the boundary cases" -- it all relies on the idea that you're going to be able to predict where you're going to make the bugs (by heuristic, habits, and mnemonics). And if you're smart enough to predict where the bugs are going to pop up, it seems like there's something better you could do to keep them from being introduced at all. In ML for the Working Programmer, L.C. Paulson suggests that a mark of the professional in the future will be writing functionally (in ML). The modularity practices seem like the Right Thing, useful for the functional programmer as well as the OO, but if your code could be formally verified, how much more confident would you be that it was correct for the general case? Simple testing is nothing like a proper proof.

But who ever writes proof-carrying code?

(ping)

It's been a while since I've last posted: I should make an effort to have exciting technical things to say more often.

But! In the last few months, I've graduated, got a paper accepted to a conference, taught six weeks of summer camps to enthusiastic middle- and highschool kids, and made all the preparations to start up work with the Goog. My first day is Monday.

Topics that I've been looking into and will hopefully post about soon:
- Haskell -- maybe one day I'll have a more consistent opinion about how I feel about static vs. dynamic typing. Writing here on the issue will probably help sort it out. Or just building something big with Haskell.
- Statistics. I went out and bought All of Statistics and I've been thumbing through it a bit.
- CS Education and edutech. Good goodness, education. I've spent most of my summer with the childrens, trying out different ways of convincing them they want to know what I think they should know. It's been going well, for the most part.
- GWT and associated topics in Javascript and Ajax. I'm joining the GWT team in just a few days, so I'm working pretty hard on learning it!

more exciting edutech!

Scratch, from the Lifelong Kindergarten group at MIT, is this lovely environment for kids (or other novice programmers) where you can make funky animations and play sounds and do cool effects! The cool effects are a major selling point -- you can get funny animations right off the bat. Everybody loves a fisheye effect. And it has this lovely website, where kids can share and tag their projects! Web 2.0 ahoy!

It's really easy to figure out, especially if you've seen things like the LEGO mindstorms interface, or Alice -- commands snap together with a familiar building-blocks metaphor. This is to say that there's a fairly standard vocabulary for childrens programming environments, these days...

The Scratch intro "Facilitorial" video is here.

Also fairly interesting (and brand new on my radar as of today), is Greenfoot, which is another educational programming environment, perhaps for slightly older kids. It makes it easy to do simulations with different kinds of "actors" on these nice 2D worlds (they can be grid-worlds, but they don't have to be)... although you have to write some Java, it looks like, to build up your new behaviors. Maybe this is awesome too.

java 6: apparently even less of a loss than java 5!

Java 6. It's so hot right now. Java 6.

Well, I'm excited anyway. The new Scripting API provides a standard interface for embedding other languages in Java and making calls between the two. See if there's already a project to handle your favorite language here -- there probably is, unless you like Common Lisp. There's even a mechanism for manipulating namespaces in the embedded language, pretty snappy.

Apparently recent releases of Jython already have hooks to support the new API. Maybe the next version of JES should be rewritten with that in mind, say once Jython 2.2 is stable. And perhaps we'll see ABCL ported to the new standard...

Also in Java 6, the built-in support for splash screens is kinda cute. And they're saying that the whole shebang is faster and prettier. Good job, guys!

things that start with p

Programmable completion for bash (mentioned earlier) is still a pretty exciting idea. I came up against the first situation where I felt like it needed to be extended today, though.

I've taken to using jar to deal with .zip files, so I can use consistent tar syntax and don't have to remember how to use the zip options. But! The bash_completion file for Ubuntu doesn't include ".zip" as an extension that it looks for when tab-completing files for jar, oh noes!

Easy enough to fix, right? I pop open /etc/bash_completion and start searching for "jar". There's a section near the second occurrence that looks like:

         _filedir '?(e|j|w)ar'

After some fiddling, I change that one line to " _filedir '?(ear|jar|war|zip)' ". And it works! For reference, that the _filedir function looks like this. It's painfully obvious to everyone what this does, yes?

To be totally fair, there's some explanatory comments right above it... but it's obtuse things like this that make we want to switch to a shell with a more sensible scripting language. bash is often line noise. We claim that allowing users to modify their environments to fit their needs is one of the major benefits of Free Software, but are we doing enough to encourage that? How is your mom supposed to pick up bash script? It seems like scsh isn't meant for interactive use as your daily shell, but what if your everyday environment had a more modern language embedded in it? Are things like that already out there?

Also: speaking of Scheme embedded in things, JScheme is a dialect of Scheme with a very simple interface to Java, called the Javadot notation . It's by Peter Norvig and crew, fairly recently updated and feature-complete. Also on Peter's (fantastic) site, you can find his older "mercilessly small, easily modifiable version". I very badly to embed this in JES. Media Computation in Scheme ahoy.

Type-inference weirdness in Java 1.5 with Generics

Surely this has come up for somebody out there before, but I haven't yet found anything about it. It came up today. Essentially: in Java, if Chair is a sort of Thing, a Vector of Chairs is not recognized as a subtype of Vector of Things... this seems wrong.

What I wanted to do was have a method that takes a vector of a certain type (an interface particularly), then pass to it as an argument, a vector of a type that implemented that interface.

So for example:
- Make an interface "Thing" that requires its subclasses to provide "doSomething()"
- ... and a class Chair implements Thing.
- A variable Vector<Chair> chairs
- And a method doEachThing( Vector <Thing> things).
So you should be able to pass chairs as an argument to doEachThing, right?

I get the error:
ThingDoer.java:17: doEachThing(java.util.Vector<Thing>) in ThingDoer cannot be applied to (java.util.Vector<Chair>)

I don't understand this at all. Surely the type-checker should be able to do that inference, that every element in a list of Chairs is in fact a Thing? Am I totally missing the point, or is this a design wrongness about Java?

Anybody else run into something along these lines?