Saturday, April 22, 2006

Yegging Me On

I just can't stay away from Mr. Yegge's critique of Lisp. What is it about critiques of one's favorite language that makes one feel so righteously indignant? Perhaps it's some primordial fear that our favorite language will be taken away from us. Sort of like the basic fear by aged monolingual English speakers in Miami that as Spanish grows in visibility they will be unable to make themselves understood. As an aging almost-monolingual English speaker, I can easily imagine myself in their position and feel panic. The feeling that Lisp might vanish is very similar.

Anyway, putting these feelings aside, let me address one more little Yeggish point —

This is a problem. It's not a little teeny one, either. The Lisp communities (yeah, there are a bunch) are going to have to realize that if Lisp is ever going to be massively successful, it needs an overhaul. Or maybe a revolution. Contrary to what some might tell you, it doesn't need a committee, and it doesn't need a bunch of money. Linux proved exactly the opposite. Lisp needs a benevolent dictator. Lisp needs to ditch the name "Lisp", since it scares people. And Lisp needs to learn from the lessons of the 45 years of languages that have followed it.

At this point I begin to scratch my head. The point he started to try to argue was that Lisp is not an acceptable Lisp. But an acceptable Lisp would be an "overhaul" of the current language. As the paragraphs before the one quoted show, the overhauled Lisp would not have the "worthless spec" the current one has, it would not have CLOS, it would not have macros, and it would have a different type system. It would have a dictator (its own Guido van Rossum? or Larry Wall?). It would reflect 45 years of history somehow. And it wouldn't be called "Lisp."

To which I respond: Would we really want a Lisp that was an acceptable Lisp in this sense?

There is a lot I don't like about Common Lisp. None of it has much to do with Yegge's rather general observations. The only specific point he made that I agree with is that case-insensitivity is a big mistake. What were the Common Lisp people thinking? It might have been difficult to convert legacy code then, but it's not getting any easier, and sooner or later Lisp will change. Franz is already agitating for "modern" mode; I've adopted it, and I can't believe that it isn't spreading faster.

There are plenty of other flaws in CL. Brooks and Gabriel wrote a critique in 1984, when CL was proposed, that hit several nails on the head. I've posted a couple of essays (NIL Considered Harmful" and FORMAT Considered Ugly) on my website about flaws in CL, and I wrote the Nisp package because I think, contrary to the Lispish conventional wisdom, that static typing is a good idea.

But in spite of all that, Lisp embodies several good features that (a) I like, nay, passionately love; and (b) are so essential to the language that removing them would make it not Lisp any more:

  1. S-expressions
  2. Programs expressed as S-expressions
  3. Macros
  4. The read-eval-print loop
Let me elaborate.

S-expressions are symbols, strings, and numbers, plus lists of S-expressions and (why not?) arrays of S-expressions. That's their abstract syntax. Their concrete syntax is:

  • Symbols, strings, and numbers What you would expect.

  • Lists: (e1 ... eK)

  • Arrays: #(e1 ... eK) (Actually, this is just the 1-dim case, but let's not worry about that.)

The thing to notice is that the abstract syntax and the concrete syntax are essentially the same, so that S-expressions provide the same thing XML that provides: A notion of hierarchy that is representable and parseable before the hierarchy is classified as being an instance of this or that particular syntactic category, data structure, or whatever. (But S-expressions are more concise than XML, at the expense of being less general in some ways.)

It's important that programs are S-expressions, but not just Lisp programs. The JScheme system, by Ken Anderson, Tim Hickey, and Peter Norvig, is an implementation of Scheme in Java that allows java to be called from Scheme, using what it inventors call Javadot notation. The expression object.field becomes (.field$ class). The expression "new class(a b)" becomes (class. a b). This notation was developed prior to the addition of generic classes and methods to Java (version 1.5). It is now possible in Java to write, for instance, "(new List<String>())" to make an empty list of strings. But it's clear that augmenting the notation in an elegant way is not going to be difficult, because the S-expression notation allows us to capture the hierarchical structure of the Java expression without constraining how we do it. We might write (List< String)), or ((<> List. String)). I leave it to the JScheme people to arrive at an extended notation that actually fits their original conception.

Another example is the Common SQL notation developed by Lispworks. When sending an expression to the database, the expression might look like (select [Researcher] :from [SampleAreas]). (This notation depends on hacking the syntax of the character '[' as well as using S-expression notation.) In other languages you have to send SQL expressions as (brace yourself) strings. To construct what is obviously a hierarchical expression involving data objects, one must discard the hierarchy and the objects, convert both to strings, and send the result to the database, which will then invert the process, recovering the hierarchical structure. S-expressions are an obviously better idea.

Now for macros. Let me first observe that macros are not the only useful consequence of the fact that Lisp programs are S-expressions. Another is quote. It's easy to write Lisp as a package or class in Java, until you come to a point when you need to translate, say, (setq x 'foo). You suddenly have to allocate a new static variable Q_foo somewhere and bind it at static-variable initialization time to the symbol foo. This is a real buzz-kill.

But the existence of macros is the really good reason why programs should be S-expressions. The debate about whether this power is a good thing is not important. We are not debating here. We are explaining what Lisp is. Macros are a non-negotiable item. For one thing, if they were repealed, anyone could reintroduce them. (All you need to do is introduce a preprocessor before the compiler; introduce your own version of 'defun' that runs the preprocessor.)

Even Scheme has macros, although they are of the "hygienic" variety. I don't think I've ever written a hygienic macro; by chance, all the interesting ones seem to be un-hygienic. Nonetheless, I still have all my fingers and toes. The desire to restrict the syntactic transformations a macro can perform is in keeping with the general atmosphere around Scheme, that there is one right way to do any given thing, and until the Central Committee announces what it is, we should avoid doing that thing. Yegge recommends that the Lisp community elect (or suffer the ascension of?) a dictator, but anyone can decide to live under the benevolent rule of the Revisedn Report on Scheme. I gather this particular dictator is not appealing to him, and I wonder what the odds are that the next one will be either.

Finally, the read-eval-print loop. Some languages besides Lisp, or their IDEs, approximate it, but for various reasons they can't quite get to the right idea: a playground where lots of pieces of code are lying around, and you can work on whichever one catches your fancy next. It's like the shell in Unix, except the shell is more of an oil refinery than a playground. But once you're in Lisp, there's no reason to go back to the shell. It's like Emacs; correction: Emacs is like it; correction: Emacs is it, because Emacs is basically a special-purpose Lisp.

Obviously, the Lisp idea is not for everyone. But one must acknowledge that there is such a thing as the Lisp idea. John McCarthy once said that Lisp seemed to be a local maximum in the space of programming languages. Most successful languages are. It's a characteristic of local maxima in high-dimensional spaces that small steps in any direction can plunge you into the depths. Mr. Yegge's casual suggestions about Lisp reforms are actually more likely to be high dives into the lower rings of hell.


EntropyFails said...

Sometimes I think that Yegge says things just to get people riled up. This can be a good thing when intended to constructively point out flaws and impediments of a particular system. This goes doubly for programming languages which tend to suffer from true believer syndrome. That particular facet always amazes me as every complete programming language can mimic every other language with the proper encoding. (Though sometimes it involves “write out string to file and compile it” types of transformations! *laugh*) Since his article got you to make this blog, perhaps the strategy has some merit. *grin*

As for your love of Lisp, I definitely feel you there. I’ve checked over Nisp but I haven’t had a chance to really program in it. I hope to correct that at some point in the future. In the meantime, have you had a chance to check out Qi from lambda associates? It is a wrapper on top of Lisp that provides pattern matching, currying, guards, and optional strong typing. Given that you enjoy strong types, you may find some interest in Qi’s type system which is First Order Logic complete. Since that will forever be the maximum of what you can accomplish with types, it seems like a nice fit for someone who loves both Lisp and types.

You can find out more about Qi at
Or if you want a concrete example of the power of Qi types, check out my blog entry on Prime Types at

Please feel free to leave some feedback as to what you think. Given your stance on types, I feel very interested in your thoughts on the Qi type system.

EntropyFails said...

They really should make the links automatic....

Qi Homepage

My unusual prime type.

airfoyle said...

I have downloaded Qi, and looked at the documentation, but I've resisted actually playing with it. I guess the syntax just isn't Lisp-y enough! But that's a lousy reason not to play with it.

Anonymous said...

Several people in the Lisp camp are very good at verbal/written defense. Defense is necessary, but offense is what blows people's doors off. If you want to get these guys to shut up, you have to sink a 3-pointer in their face. The few guys that made the Lisp movies understand this. It would be cool for the aces that get it to create 3-pointer movies that SHOW people the power of Lisp, or do things like "Try Ruby", or Ruby on Rails.

I just went away from this post and watched one of the movies. It's incredible how powerful the movies can be. Lisp is so different that people starting out in it are so easily lost, confused and frustrated--I think the oldtimers forget this sometimes. I remember watching this movie for the first time a half a year ago (I've started learning Scheme since then). At that time, most of it went completely over my head. But watching it now, I just picked up a couple of things and realized a few more. More of this!

Mike said...

You know I think that everything that Yegge said can also be said about Unix:

1: Which Unix? Ok, that's obvious.

2: Worthless Spec: POSIX. Maybe not quite AS obvious, but every Unix I've ever worked with has both POSIX system calls and native ones. Seems that most people use the native (and often more efficient) ones.

3: CLOS: Well, this one doesn't carry over perfectly, but there are similarities from bolting on a object system to bolting on a new filesystems, binary compatibility models as so forth. Yes, it makes it bigger. So? Adding new posts to a blog makes it bigger too...

4: Macros: Yes, I hate complex bash scripts (Unix macros for sure) as well. "They're notoriously hard to debug, and honestly it needn't be thay way." Maybe he should get to work on a better debugger.

4: Type system: Yes, you can make sure that you don't put 2 files of the same name into the same directory, but you can't be sure that a file with a .txt doesn't really contain binary data that will mess up your terminal. It should be more extensible and skinnable. I'm arguing that it should be bigger, regardless about what I said about bigger equating to bad above.

There is no acceptable Unix.

We shouldn't use Unix. For all it's stability, security and robustness, it has some problems -- especially in scenarios for which it wasn't designed. File and print sharing suck. We should never use this tool again.

And the community may someday shrink too. If everyone else has problems that this OS doesn't fit, nether should you.

In fact, while you're at it, don't use any screwdrives either. Because hey, I have trouble using them to push in nails.

airfoyle said...

For that matter, all those comments about not knowing which Lisp to use carry over to most "serious" programming languages. Ever investigated which ML to use?

It's obvious which Unix to use? I assume you're alluding to Linux, but the question "Which Linux?" has many equally wonderful answers.

Mike said...

Not just Linux though there are a huge number of distros there. Unix can mean Solaris, Tru64 (DECOS, OSF/1, whatever they're calling it these days), Ultrix, Unicos, AIX, BSD (Net? Open? Free? BSDi?).

I've always believed in applying the right tools for the job at hand. I don't use LISP personally, but most of the applications I'm involved with focus on data transmission, not machine learning. That doesn't make the language useless.

His list was bordering on silly. I could apply the same logic he did to anything...toasters, cars, polymers, numbers, elements...the list goes on.

Michael H. said...

It's late at night, and I probably shouldn't be posting this, but ...

I must confess, I have occasionally thought that Lisp needs a benevolent dictator. This is partly because there are existing aspects that can be improved (you mention several, like format), and other aspects which I'd like to see a standard for (streams, threading, and graphics).

Ruby, Python, and even Perl seem to have grown quickly and well under their kindly rulers. Lisp is in a different boat. The community is doing a great job of self-policiing -- I'm thinking of language lawyers like Paul Dietz who run an enormous battery of tests against all the lisps they can find, trying to suss out what conforms to the CL standard and what doesn't. And there is a vast amount of code which will work under Allegro, Lispworks, SBCL, CMUCL, and OpenMCL, because the parties involved work to make it so.

But. But, it's hard for the informal committee to direct future growth. I've helped write a utility package for handling threads in different lisps in a consistent way -- it was a fair amount of work, and everyone else who wants to use threads in cross-platform, cross-vendor lisp has to do the same thing. Cross-platform graphics are another hole. While everyone will pretty much agree that Java's first attempt at desktop graphics was fairly terrible, at least it was there, and it held the promise of getting better in the future. There are many different possible standards and packages for lisp users, and I've never seen a clear way to choose between them. This the problem with a democracy.

From what I've read of the ANSI standardization effort, a common refrain was, "if we'd known it'd be this much work, we wouldn't have started in the first place". So what opportunities does the lisp standard have for growth?

airfoyle said...

> So what opportunities does the lisp
standard have for growth?

It's a good question, although there's a
difference between the standard
and the language itself. As you point
out, the community has lost its taste
for the work involved in
standardization, presumably because the
gain didn't seem worth it. However,
appointing a dictator is not an
alternative, because the language is so

I am not worried about Lisp's future
prospects. The language hit bottom
around 15 years ago when the combination
of AI Winter and the sneers of C
programmers about efficiency made it
look like a dead end. It was kept alive
by its beauty. Someone once said that
if we find life on other worlds, and
they play board games, then one of those
games will be Go. The rules are so
simple they are bound to be discovered
by any intelligent life form. The same
thing can be said about Lisp (well,
Scheme anyway).

Lisp is looking healthier because
performance is not the 600-pound gorilla
it once was, and because no one couples
it with AI so tightly any more. The
question is how it's going to
standardize on libraries and GUIs. I
know it's taking a while, but I think we
are converging on some de-facto
standards in these areas. What we need
is a really good book, something like
the LaTeX Companion.