Thesis proposal!

Last week, I had my thesis proposal. I proposed, basically, that for doing machine translation into lower-resourced languages we're going to want better cross-lingual word sense disambiguation to help our MT systems make better word choices. And I outlined some methods that we might use to get that goal. I'm going to develop these approaches in the context of a few different kinds of MT systems, particularly focusing on translating from Spanish to Guarani. So I guess now all I have to do for the rest of the PhD is this project.


 

 If you're curious, I'm writing my dissertation in public, on github: http://github.com/alexrudnick/dissertation

Let's do this.

Computing Education and the ACM Paywall

Recently Mark Guzdial wrote a blog post in which he describes some of the particularities of research in computing education, and defends the continued paywalling of ACM articles in the Digital Library. Just to be clear, Mark is brilliant and friendly, and he does fantastic work. But I think he's mistaken on this particular issue.

Here is Mark's argument, to reduce it to bullet points:

• CS Ed research is typically not funded by public funding agencies, but done on researchers' own time, so the argument that it should belong to the public does not hold.
• Educators working in the developing world have different needs than those in the WEIRD world; we can't simply toss papers over the wall and let them figure it out.
• ... and anyway, the ACM is basically good people, and doing good work with the money it collects, especially for the education community.
• Ergo, the ACM should keep up its paywall.

Early in his post, Mark brings up the first sentence from the Tear Down This Paywall petition: "Computer science research is largely funded by the public, for the public good." He points out that lots of CS Ed research isn't supported by grants, and that people who are primarily educators do it on their own time, because it is important to them.

So firstly, Mark's own work is funded by the NSF (as he mentions), so the argument about funding would apply to his work, along with the bulk of CS research broadly. But even if we accept that the public can't demand access to the other CS Ed papers, we should consider: what's best for the careers and goals of the CS Ed researchers themselves?  What do they want?

Certainly CS Ed researchers trying to publicize their work -- people who care so much about it that they take it on as a labor of love -- would prefer to reach the broadest possible audience. They don't directly benefit from a paywall. They may like the ACM and want it to continue putting on events, but the paywall keeps them from readers.

But Mark takes a bizarre turn in framing the idea of dismantling the DL's paywall as forcing open access on unsuspecting researchers who didn't agree to it, "after the fact". OA wasn't part of the deal!  He says in the comments, "Certainly, volunteers can volunteer the fruits of their labors. They shouldn't be coerced.  It shouldn't be a requirement." It's hard to imagine a young researcher protesting a larger audience. People don't choose to publish with the ACM because of the paywall on the DL, but in spite of it. For many subfields, ACM conferences are simply where one must publish to be taken seriously, and dealing with the paywall is the cost of doing business.

As for the second point, about researchers and educators in the developing world -- while it is almost certainly not sufficient to release our papers if our goal is to help them develop their own curricula, it's verging on paternalistic to decide ahead of time what would and would not be helpful for them. Make the papers broadly available and let them decide what is relevant and useful. And by all means, we should develop other materials too, but this is a separate pursuit.

We find educators, working programmers, interested laypeople, and researchers from other disciplines in a similar boat -- they may not have the context to completely understand a paper intended for specialists, but they can still get something out of it. And to collaborate meaningfully with -- or join -- the specialist community, they're going to have to read lots of papers. We should reduce the barriers to entry for potentially-interested people, wherever they are. Working programmers and educators are empirically short on both time and ACM memberships.

So for most computing research, we are still seeing publicly funded work made harder to access than it should be. And for CS Ed research, we see work that researchers might want widely distributed made less available than it could and should be. Opening the DL would be an immense good for people around the world -- it's great that Mark and others put in the additional effort to make their personal papers available, but not everyone is so conscientious, or so web-savvy, or so still alive. And the current state of affairs still requires that people go hunt down each paper individually.

It would be silly to claim that the ACM doesn't need a revenue stream, and I think their continued existence is probably a good thing. But there are other funding models for scholarly societies. The current state of affairs is comfortable for Mark and other established researchers, but it could be much better for the up-and-coming looking for a broad audience, as well as for interested parties outside of well-funded academic institutions.

ACM's optional Open Access is effectively a NOOP

Not all academics have the great moral luck to be working in NLP, where almost everything we publish is going to be Open Access whether we care about OA or not -- barring some out-of-the-way venues who really need to get their acts together.

For example, Lindsey Kuper (both my favorite programming languages researcher and my wife) just put in a paper at the Functional High-Performance Computing workshop at ICFP. And roughly five minutes after she got the acceptance notification, she got the form to sign over publishing rights to the ACM.

Now the ACM has recently made open-access publishing available through their Digital Library -- for $1100 to $1700, depending on the circumstances. I’m not opposed to APCs (“article processing charges”) as such; this seems like a step in the right direction. But I’ll argue that this particular approach is effectively a no-op.

It was unclear to Lindsey’s advisor whether they could pay the Open Access fee out of their grant money -- and while he’s a great, upstanding guy, he’s also a young pre-tenure professor, so he didn’t have a lot of spare time to look into this. He’s trying to do some science, not get bogged down in policy details. They went with the “retain copyright, but the DL copy won’t be OA“ option. I imagine this scenario will be pretty typical.

So this new policy effectively won’t change anything for the ACM’s Digital Library: all old papers are still locked down, and for most of the new ones, the authors won’t fork over the money for the OA option.

It’s a giant missed opportunity; the Digital Library could be a phenomenally useful resource. But for people without ACM membership or institutional access -- e.g., almost every working programmer -- the situation is the same as before. If you accidentally click on a link to the DL, that’s just a momentary dead end. Hopefully you can find the paper somewhere else.

NAACL 2013 review

Just recently, I was in Atlanta for NAACL. So much fun! The hallway track is always the best -- I saw a bunch of friends from the NLP world, and especially a lot of Googlers, and met a bunch of new people! Also I managed to be present for Ray Mooney and David Forsyth and some other professors disagreeing animatedly about internal representations of meaning and to what extent you need to take the intentional stance with respect to other people.

Lots of really interesting papers this time around. There is of course Hal Daumé's expert opinion about the interesting papers at the main conference -- I saw a lot of those same talks, having mostly been hanging out at the machine translation and syntax/parsing tracks. On a personal note, it's exciting to see people I know and have worked with getting mentions on Hal's blog! (so, congratulations Greg Durrett and John DeNero and Juri Ganitkevitch!)

Additionally, here's what I thought was cool:

• Training Parsers on Incompatible Treebanks by Richard Johansson. You want to build a parser for your language. And you've got a treebank. No! You've got two treebanks. Even better, right? But what if those two treebanks use entirely different annotation schemes? ...
• In the invited talk on Wednesday, Kathy McKeown talked about, among other things, the idea that as NLP people we can provide evidence for or against ideas in comparative literature or literary theory, in collaboration with literature folks -- "well, the theory is that narrative works like this -- let's check!"
• At *Sem, but also in the main conference, people are talking about using richer, more structured semantic models in our applications again. The really major change in the field in the early 1990s was to not do this -- but now we've got bigger computers and more data, and as a community we know a lot more about stats! Kevin Knight and his group are launching their Abstract Meaning Representation project ("It's like a treebank, but for semantics.") -- maybe it'll work this time!
• Also at *Sem, Yoav Goldberg talked about the unreasonably enormous Syntactic Ngrams dataset -- it's basically chunks of parse trees from the English part of the Google Books corpus, indexed by time. That's going to be super useful.
• I popped in to some of the Computational Linguistics for Literature talks -- Mark Riedl's invited talk about programmatically generating stories for games (slides) was especially good!
• SemEval! There were fourteen different tasks -- lots of different aspects of understanding text! And people are using all these wildly different techniques to do it. An introductory talk about a task and then a single presentation about a system for performing that task is not always enough to really understand the problem, though...
• I think my presentation went pretty well! People I've been citing for a while were at my talk, and people seemed engaged and asked good questions! (slides, paper)

Alright! So now, full of encouragement and ideas -- back to work.

ACM Publications Board: "How can we minimally budge so you'll stop bothering us about open access?"

You may have seen the recent article from CACM, Positioning ACM for an Open Access Future. I found the article fairly upsetting. The first paragraph...

The age of open access is upon us. Increasingly, the consensus of authors of research articles and their funding institutions is that the fruits of taxpayer-supported research should be freely available to the public. This is a compelling argument and a noble goal.

However, we're not going to do that anytime soon! They then launch into a red-herring discussion of predatory OA publishers (which are a real thing! there do exist vanity presses that have sprung up to capitalize on the OA trend) -- but this ignores how such publishers come about. You don't accidentally become such a vanity press. Serious venues with good review boards won't have the problem of "a glut of third-rate publications that add noise rather than insight to the scientific enterprise".

Afterwards, they discuss four different approaches for the ACM to not go fully OA: the first is that, optionally, authors could pay an extra fee to have their articles available from the Digital Library. The other three are simply ways in which the paywall restrictions could be lifted under some circumstances.

None of these are acceptable. Not if we believe that "the fruits of taxpayer-supported research should be freely available to the public". Why is the current situation even sort of OK?

The ACM's resistance to OA so far, its claims that figuring out a way to do it is too hard, that it's too expensive or will lead to bad publications -- as far as I can tell, these mean at least one of two things:

• Perhaps the ACM is not as clever as USENIX, the Association for Computational Linguistics, NIPS and The Journal of Machine Learning Research (ie, most of the machine learning community), and all of those math folks...
• Or alternatively, people within the ACM, despite its apparent status as a non-profit, really like all that money; they like it a lot more than the purported mission to foster "the open interchange of information..." 

Which one of these is true? Both?

We need to get the ACM to stop thinking like a for-profit publisher and start thinking like their goal is to move the field forward and educate people. The ACM needs to drop both the paywall and its membership in the AAP.

Either that, or we as computing professionals need to drop the ACM.

this is something new and beautiful: Coursera and Udacity

Just last week, I finished the coursework for Coursera's machine learning class. It was great! I had a really good time with it, and I'm fairly proud of the accomplishment.

If you've been within earshot of me in the past few months, you probably know that I'm really excited about Coursera and Udacity and their ilk (including, but not limited to, edX, Khan Academy, and Duolingo). There are two experiences I'd like to contrast with taking a course on Coursera.

Some years ago, I was living in Atlanta and working a real job. And I went over to the Georgia Tech math department to see about taking some masters-level statistics classes, imagining that they would let me pay them lots of money in exchange for taking classes at the university where I had just graduated months prior. But it turned out that they wouldn't let me do this without being admitted for a full-time degree program.

Fewer years ago, I was starting my PhD at Indiana, and knowing exactly what I was there to learn, I picked out three classes: one on NLP, a computational linguistics class (from the Linguistics department), and one from Stats. I got a mild hassle from the department about my choices: these were all "fun" classes, and shouldn't I work on fulfilling my breadth requirements? I've since finished my IU coursework, and let me say: not all the classes I had to take as a result were very interesting, or even very well taught. Some were downright bad.

But now there are free online courses that are meant to be good, such that you take the ones you're interested in taking, as opposed to expensive in-person courses that may not be good, but you're obliged to take them anyway -- this is huge.

Whether or not you think that teaching in person is going to stay relevant, not everybody has access to good teachers in person. This remains true even for people at universities.

Moreover, online classes lower the barriers to entering or leaving a course to almost nothing. Want to sign up for a class just to try it out? Nothing could be easier! Don't enjoy it, or it's not what you thought it was, or find out you're busy with other stuff? Nothing lost, try a different one! But if you stick it out and put in the effort, then not only have you learned something, but also you get a certificate that says you finished! (maybe these could be OpenBadges sooner or later...)

There are going to be lots of bytes spilled about these things in the coming years, but just to make it clear: I'm jazzed about helping people who want to learn things get access to material about those things. And the World Music class is starting up soon, which my mother and I are going to take! Because why not?

take five minutes: support open access

tl;dr: Sign this petition to support open access for publicly-funded research!! http://wh.gov/6TH

Here's the situation: there's lots of scholarly work being done. And you, as a citizen of a country, are paying academics to do science (or whatever), write about it, and review the work of other scholars. The work that makes it through the reviewing process gets published, typically in a journal or at a conference.

Here's the problem: a lot of that scholarly work is then inaccessible to you. You have to pay to read it, and often you have to pay a lot. If you're at a well-funded academic institution, your university library has to pay a lot. It's a serious problem for universities as wealthy as Harvard. Where does this money go to? It doesn't go to the academics who wrote the papers, or those who reviewed them: it goes to publishing companies with absurd profit margins who have trouble pointing at what value they add to the process, aside happening to own prestigious journals.

Concretely, this is a problem for the independent researcher, for the small business developer-of-stuff who wants to get the latest developments, for the interested public who wants to read and learn and grow, for the precocious teenager. I've come to care kind of a lot about this issue: it's because I believe in science. I think it's pretty important: it should get out to as many people as possible, not just because the citizens paid for it in the first place, but also so we can make progress faster.

The National Institutes of Health have famously set up an Open Access mandate: all the research that they fund must be available to the public pretty soon after it's published. Many universities are doing the same thing. The Association for Computational Linguistics (who run the conferences and journals where I'm personally likely to publish), do a bang-up job of making all of their articles publicly available, and I'm really proud to be associated with them. But not every professional organization, and not every field's journal are like this. Most are not!

How can you help? Right now, there's a petition on the White House website where you can ask the administration to expand the NIH-style mandate to other funding agencies: I'd really appreciate if you'd take a minute to make an account and sign the petition. Click here: http://wh.gov/6TH

(hrm, I seem to have written about this back in 2007 too)

quals writeup: Tree Transducers, Machine Translation, and Cross-Language Divergences

I hope it's not too pretentious to put things I'm writing for my phd qualifiers on arXiv. I think arXiv is really exciting, by the way. Leak your preprints there! Also pretty exciting: tree transducers for machine translation.

Abstract:

Tree transducers are formal automata that transform trees into other trees. Many varieties of tree transducers have been explored in the automata theory literature, and more recently, in the machine translation literature. In this paper I review T and xT transducers, situate them among related formalisms, and show how they can be used to implement rules for machine translation systems that cover all of the cross-language structural divergences described in Bonnie Dorr's influential article on the topic. I also present an implementation of xT transduction, suitable and convenient for experimenting with translation rules.

Paper! http://arxiv.org/abs/1203.6136

Software! http://github.com/alexrudnick/kurt

CMU's Avenue Project

Before there was HLTDI, there was Carnegie Mellon's Avenue Project, which seems to have had basically the same goal as us -- produce good machine translation systems for under-resourced languages, especially those spoken by under-resourced indigenous people.

Avenue itself doesn't seem to have been under-resourced, though -- they sent people to South America (Chile, Peru, Bolivia...) to collect training data, and seemed to have a lot of contacts with local educators and language experts. They got quite a few papers out of this line of research, and apparently wrote a lot of good software. They had a much deeper pool of money (and arguably talent) than we do.

And now... the website is dormant, the PhD students involved seem to have graduated, the data and software are not publicly available, and the researchers seem to have moved on to other things. (one of the resulting doctors is the illimitable Kathrin Probst, who hipped me to Avenue when we were both at Google Atlanta, although I didn't really grasp how serious it was at the time -- darn her for being so humble!)

They were pretty gracious in giving us the Quechua data that they collected (and said we could redistribute it), and I've been reading a bunch of their papers, but I'm left some sadness about the whole enterprise -- they surely already went through a lot of the problems that HLTDI is going to have to address. Why can't we just check out and fork their code?

... maybe I should ask for their software too. Science is supposed to be easily replicable, isn't it?

empiricism, faith, computational linguistics

Mike sent me a fantastic piece by Ted Pederson, calling for NLP/CL researchers to care more about having reproducible results and maintainable software.

Empiricism Is Not a Matter of Faith.

It's sad that this is a problem; it should be easy to get other researchers' software up and running, reproduce the results reported in papers, and plug things into other things -- but I think we're moving in that direction. At least one CL conference, CICLING, explicitly calls for open software and reproducible results. Which is pretty cool.

workshop on FOSS machine translation!

Second International Workshop on Free/Open-Source Rule-Based Machine Translation. And it's in Barcelona! What could possibly be better?

I should probably go to this.

extended abstract for a workshop, MTMRL

On Wednesday, I submitted a mini-writeup of my project this semester to the Machine Translation and Morphologically-rich Languages workshop. My code definitely isn't in a useful state yet, but they say that works-in-progress are OK.

I'll let you know if my project gets accepted. The workshop is in Israel, which would be a really interesting place to visit!

Here's what I wrote:

Abstract

Here we describe a work-in-progress approach for learning valencies of verbs in a morphologically rich language using only a morphological analyzer and an unannotated corpus. We will compare the results from applying this approach to an unannotated Arabic corpus with those achieved by processing the same text in treebank form. The approach will then be applied to an unannotated corpus from Quechua, a morphologically rich but resource-scarce language.

See the rest here; it's short! (or as a pdf)

Just in case you're not familiar with the idea of valency for verbs: wikipedia!

drawing trees with LaTeX

So, whatever you think about producing documents with LaTeX (personally, I'm pretty ambivalent, for reasons that I may go into later), if you want to draw parse trees with it, there's a nice package to do that: qtree.

Installing packages from CTAN manually looks hard.

If you're on Ubuntu/Debian, though, you just need to install these two packages: texlive-humanities, texlive-pictures.

qtree itself is in texlive-humanities, but it depends on a package, pict2e, that's only in texlive-pictures, so you have to install them both or it won't work.

And then you can draw some trees just by specifying the bracketing of the phrases (see the qtree docs for exactly how).

explanatory power of working examples

The NLP algorithms I've been studying since I started back at school aren't particularly complex. But they're often described with really dense notation: maybe your field does this too! Here's a description, for example, of how to calculate an "outside probability" -- it's the (joint) probability that a particular nonterminal symbol covers a certain chunk of text, and the words outside the span of that nonterminal. This is from Fei Xia's lecture slides (and I think these are pretty good).



Maybe what I need is more practice picking apart dense notation, but in all honesty I have trouble keeping track of what the different letters mean. Maybe a nice dynamic programming implementation springs to mind for people smarter than me, but I have to stare at it (and the surrounding slides) for quite a while!

I think I'd be making a pretty good contribution to the world if I took the algorithms I'm learning and wrote down the most straightforward pseudocode and prose versions I can, with a running Python implementation and descriptive variable names. Surely many people out there would find code easier to digest!

Somebody's already done precisely this with the Viterbi Algorithm wikipedia page, and I'm very grateful to that somebody.

change of scenery and reviewing conference submissions

So you may not have heard yet, but last month, I left Google Atlanta, packed up my cats and my computers, and headed to Indiana University. Lindsey Kuper helped quite a bit, both in motivation to do this and in the actual moving process. So I'm a PhD student now, very exciting! I'll be working on natural language processing and machine learning -- something I'd wanted to focus on for quite a while.

But in my last two weeks at the Goog, I had this really interesting opportunity, presented by my colleague Katharina Probst. She's a reviewer for the Conference on Information and Knowledge Management (is this the same as being on the program committee?), and asked if I wanted to help. It seemed like good practice for my upcoming stint as an academic.

So we (I, with her guidance and sanity-checking) had this pile of eight papers to get through. Some were fairly mundane, like learning classifiers to determine if a message is a flame, or summarizing a group of documents; one was particularly targeted at finding references to rabbinic literature in other rabbinic literature (they don't use ACM-style citations, typically). And so on. A few were written very clearly and had well-motivated discussions on why the problem is interesting and important, and others... not so much. This is why we have peer review.

But it was an interesting experience, and I'm glad I took it. My tech lead, Miguel, graciously let me use my "20%" time to Advance Science and just review papers for a day. I wanted to do a good job of reviewing, so I read the papers really closely, took a lot of notes, and wrote a few paragraphs in response for each one. Katharina at least seemed to think that it was good feedback, so that was reassuring. (although I would have liked more feedback on my feedback).

Once the other reviewer's ratings came in, I was fairly pleased to see that my ratings weren't far off from the other reviews. If I missed some amazing gem of wisdom, then at least it was apparently hard to find -- the papers I liked the best were accepted. I was more concerned, honestly, that I would mistake some stale old idea as a clever new one. But again, that's why we have many eyes on these things.

Alright! So now I just have to produce some stuff for other people to review. To the lab!

this blog post: for you, $50.

I recently met a fellow who's working on a doctorate in literature, but his previous background is in Library Science^W. I wasn't sure what the interesting problems in library science might be, so I wandered over to the wikipedia article and started falling through the links.

A few links out, I ran into the Serials Crisis article. Apparently (and Wikipedia articles close to the "Library Science" one are never wrong), the costs of subscribing to scholarly journals keep on going up -- libraries only have so much money for subscriptions, but there are ever-more academics and subfields, thus more journals. And if a given library cancels its subscription from a particular journal, that publisher's fixed costs are still fixed, so prices increase for the remaining subscribers.

The traditional academic journal system had seemed pretty shaky, especially in light of the Web; upsetting publisher websites (Springer, ACM Portal, IEEE's site...) seem like their sole purpose is to keep the enterprising students from reading an article. In light of how most of the science behind the articles is publicly funded in the first place, the articles seem like they should be public as well.

I wouldn't mind seeing companies like Springer just going away; universities seem totally capable of hosting journals -- over the web especially! There may be some compelling reason for the current system, and I'll try to find it out... but for the short term, tools like Google Scholar could go a little further out of their way to help us find the full text of an article!

Also:
http://en.wikipedia.org/wiki/Open_access
http://en.wikipedia.org/wiki/Open_access_journal
http://en.wikipedia.org/wiki/Open_access_publishing
The Serials Crisis: A White Paper for the UNC-Chapel Hill Scholarly Communications Convocation
The Crisis in Scholarly Publishing