a life of coding

Sunday, November 19, 2006

How Python is Killing Lisp

These are tumultuous times for lisp. Once the bastion of language academics and hackers and shunned by those of industry, it's received much lip service from successful dot commers, most notably Paul Graham. Lisp has been around a long, long time (its the second oldest language still in use, behind fortran), and its gotten very mature in that time. Industry languages continue to steal features from lisp, and some modern languages like python and ruby are more lisp than C. Still, lisp does not have the popularity one might expect. What is it that's holding it back while python prospers?

Lets start with what it gets right. It is said that lisp is very expressive, allowing extremely complicated ideas to be expressed succinctly (and hopefully quickly) by those fluent in the language. That's a rather heady idea, especially considering it is being applied to a practical industry. What this means in practice is that a great programmer is less limited by lisp than C++ or Java. This is really important because there are no natural limits on the productivity of a programmer. Unlike artisans and construction workers, there are no physical processes that limit how fast he can accomplish a task. The best person of another trade might be hundreds of times better than the novice, but they are limited by nature. Take away that limitation, and a handful of the best people could accomplish amazing things.

What makes lisp expressive aren't complex features like Object Orientation, but powerful simple ones like closures. The simplicity of these features let you build new complex ones as times change. Aspect Oriented Programming was part of the original Object Oriented lisp system based on closures. Continuations are a great way to deal with the non-linear flow of web applications.

Many of these powerful features have been adopted by new languages: Python can generate closures and has list comprehensions. Still, they don't contain all of lisp's power, and yet a search for open-source lisp applications on sourceforge returns 345 results to python's 6,571. Where are the hordes of elite programmers looking to change the world? Its easier to write things in lisp, right?

What lisp is getting wrong, opening the door for other lisp-like languages, is the new user experience. This means documentation, discoverability, sample code, libraries, and books. Its forums with solutions and smiling people. Unfortunately the lisp community is confident that their horse will win in the end, and they're just not willing to put in the effort. Other languages whose success is not so certain are more motivated to please. Python is one guy's foolish attempt to change the world, and you know that he spent a lot of time working to that end.

What lisp needs is is dedicated people behind it, laying the foundation of what people want. Specifically, people who are not currently lisp users. Given a python executable, some syntax, and knowledge of "import", "dir", and "help", anyone can learn python. Lisp should also be easy for newbies; there is even less syntax! SLIME is a great package, but please, stop hurting the cause... people who don't use Emacs are terrified by it.

Even the first step of a newbie has too many decisions: picking a lisp interpreter. There are simply too many of them, and their features are often inconsistent and incomplete. Perl, python, and ruby all managed to get one interpreter running on many platforms fairly uniformly. I don't worry about the networking or threading model inconsistencies between python on my mac and python on my linux server.

They also all have package managers that just work. This is a critical feature that just needs to work. CPAN really set the bar for package mangers in general. Asdf-install is okay, but it can be inconsistent across runtimes and packages don't always install correctly.

There are simple things as well. I am constantly irritated by the lack of prompt features in lisp interpreters. I'm lucky if they even recognize control characters... command history is a blessing. I haven't seen one that has any manner of auto-complete (something based on list-all-packages and do-symbols would be fine, thank you).

Its the unfortunate nature of lispers to work on the really long term things which are totally useless to their contemporaries. While I really appreciate that Paul Graham has decided to work on his own version of the language, I suspect that his lisp oriented incubator company would benefit more from a simple, usable IDE for sbcl and openmcl.

Ultimately, I am hopeful. Lisp is something that I believe in, and I'll keep trying to do more with it. If any language is ready to transcend old habits, this one is. Most things have already been tried in it once (distributed computing, hardware implementation, graphical editing), the next big thing is probably already in there somewhere waiting to be harnessed.


  • Vinny! This post is insightful and a half.

    I'm trying to remember a case where I successfully managed to use asdf-install -- a good package management system is totally vital. That, and a good way to distribute your packages once you've made them. More files should maybe start with "#!/usr/bin/sbcl --load" ...

    What if there was something like Slime for Eclipse?

    It does seem criminal that sbcl doesn't use readline by default. It's so good, otherwise! There's always rlwrap, of course...

    By Anonymous Anonymous, At 11/26/06 10:33 PM  

  • By Blogger asdf6661, At 11/30/06 4:08 AM  

  • Good point well made. I decided to learn Lisp a few weeks ago (having read a load of Paul Graham's essays, I realised that he just wouldn't shut up about it ;-). However, I've not put much effort into it and I still haven't become productive in the language. In the meantime, I've been learning Python and I've found the learning curve to be far smoother and Python is still pretty powerful and high level compared to what I'm used to (Java)... so I'm afraid that Lisp has probably lost me to Python unless I can find a way of getting in to Lisp without having to work too hard at it.

    By Blogger Chris, At 11/30/06 4:43 AM  

  • I agree, LISP is very alien for somebody who comes from an imperative language background e.g. C or Java.

    Python with its multi-paradigm allows one to start C-like and then grow into the more functional aspects of the language.

    Afterwards, the value added by LISP is simply not sufficient to justify the cost of learning it (at least in my humble opinion).

    Your comment about SLIME is head-on: I am a vi(m) guy and Emacs is very alien to me.

    By Anonymous Muharem Hrnjadovic, At 11/30/06 4:47 AM  

  • Actually, there is Slime for eclipse. It is called CUSP, and you can check it out here.

    By Anonymous Anonymous, At 11/30/06 9:31 AM  

  • Hi,

    For what it's worth, you may find the NewLisp project interesting (at http://www.newlisp.org). Although it is apparently not highly esteemed by the "Lisperati", as a beginner, I have found it quite congenial. It has very nice documentation with worked out examples and they seem to subscribe to the Python "batteries-included" library model. (If I hear one more Lispeon say "Oh you can do scripting with Lisp, just install 'lispx and 53 million supporting packages and you're good to go!", I may kill them.)

    It's a single executable, you can "compile" applications conveniently, and it runs (well) on many different platforms. Not to mention the documentation (or did I mention the documentation?). You have no idea how wonderful the documentation seemed after trying to hack my way through the Common Lisp HyperSpec with gun and camera when trying out CLISP.

    I have so far completed several middling sized projects on it and am looking forward to doing more. It seems to give me a lot of the power of Lisp, without the pain of the "blessed" versions. I use Python professionally (in fact, I introduced it to my workplace), but all my personal development has been in NewLisp for the last little while.

    Thanks for listening,


    By Anonymous Anonymous, At 11/30/06 11:52 AM  

  • A great post that brought two great LISP resources to my attention.

    I struggled with SLIME a couple weeks ago and gave up on it. I cannot stand Emacs, and trying to set it all up on Win32 was a horrible waste of time! Cusp is a great alternative!

    But I am especially impressed with newLISP. I really enjoyed their CodePatterns page. Its refreshing to see how to actually do stuff with LISP in a LISPy way.

    By Anonymous casey, At 11/30/06 1:13 PM  

  • Hello,

    Just one thing I want to know, what's the purpose of this post? It doesn't make non-Lispers want to program in Lisp, and it will hardly persuade any Lisper to start improving the "new-user experience". What is written here that's not been said before?

    Another day in blogosphere, another meaningless rant.


    By Blogger Chaitanya Gupta, At 11/30/06 2:36 PM  

  • Dear Sirs,

    Thanks anonymous for the pointer to newlisp. I may give it a try.

    After reading some of PG's essays, I did give Common Lisp the old college try, but without much success. I got the Touretzky book and did all the exercises, but didn't feel like I was learning anything useful.

    The main problem for Lisp 's adoption I think is Common Lisp. You have to learn, I don't know, 4 different types of assignment, 6 different equality functions, 20 different loop constructs that were maybe revolutionary in 1965 mainframes but today look like, well, like they were revolutionary in 1965 mainframes.

    Oh, and CADADR, CADADDADDR and CADADADADADDR -- is that some kind of joke? The dot... Symbol attributes (I think they were called, those useless bits of info that you "get" from a symbol). The FORMAT windings for printing a newline... The associative lists... a O(n) hashmap? again, not very productivity-enhancing in my opinion.

    And don't get me started with the "don't worry about the parens, we will fix it with indentation" nonsense. No you don't fix it with indentation. I am not getting any productivity boost if I have to poke around with a ruler and square to see where the conditional ends and the else clause begins. Hint: a paren-matching editor is of no use when you are reading a printout of a program.

    (They tell me Franz has a n if* macro that has "then" and "else" to make it more readable. Of course, the hardcore Lispers despise it.)

    I think the best idea if you want to change the world is: rescue the useful parts from under all that pile of revolutionary 1971 technology, and I assume there must be, take the good bits (macros, continuations, clothures, whatever) and build them into a language that can be described it in a K&R-sized book and that humans can learn without having to undergo 7 years of self-Pavlovization-cum-Stockholm Syndrome. As things stand today, I'm sure macros and closures are wonderful, but there is just too much useless junk you have to memorize before being able to use them.

    This is what I think Java got right: take the main object-oriented concepts from Smalltalk and leave off the 1975 conceptual breakthroughs like "the while loop is an object too", that in the end of the day are of little practical use.

    Still waiting for that language that has closures and macros and that I can learn without having to take a year of unpaid leave,

    Curly Brace

    By Anonymous Anonymous, At 12/1/06 2:59 AM  

  • I agree with you.
    Doing mainly .NET programming, I just started trying to learn LISP.

    First I had some problems to find and choose a development environment under Windows. There are several alternative but it's relativel hard to find them all and you don't know which one is the best/most used.

    I chose allegro CL ( http://www.franz.com/products/allegrocl/ ) becaused it seemed like one of the best GUI IDE on Windows.

    Coming from the full fledged Visual Studio it feels like you make a 10 years jump back.
    It doesn't look like a windows application. It also doesn't feel like a windows application (it doesn't use MDI but has a set floating windows ala Mac OS).
    Compare that with F# (an ML derivate running under .NET) where you simply have to add an add-in to visual studio and you're set.

    The documentation is also hard to navigate. I searched for a command history in the command interpreter and I couldn't find it. I searched how to auto-indent the code (because it's hard to know what's the right way to indent LISP code at first) and I couldn't find it either...

    The documentation ( http://www.franz.com/newtolisp/ ) states that there is a "tutorial included in Allegro CL to get a feel for the dynamic nature of Lisp" but yet again I didn't find it.

    And let me be clear. Allegro seem like a good, powerful and capable system. But when you start using it you can't help but feeling a bit lost.

    After that I went to find a good tutorial.book about LISP. Fortunately there are lot of quality ressources available online. Quite a few books with a free version available online, that's really nice!
    (Is it me or is there a rule that states that all websites containing LISP tutorials must look like they were last updated 10 years ago and end with .edu?).

    I started reading one of the LISP books. It looks well done but it uses emacs (SLIME?) as it's basis. And let me tell you that it's not fun to find the equivalent of that ctrl-x-y-z command.

    If you think that it has nothing to do with LISP and that's only superficial issues, You'll be completely right!
    I never had any major issues blocking me. Nevertheless I couldn't help but feeling that if you're not working at a university, growing a beard behind a unix machine you're not having the real deal.

    By Anonymous Monkeyget, At 12/1/06 3:40 AM  

  • But I am especially impressed with newLISP. I really enjoyed their CodePatterns page. Its refreshing to see how to actually do stuff with LISP in a LISPy way.

    For those people who like recipes without giving much thought:


    If you think some parts are missing, well, as you see, it is SourceForge, go and edit it to add what's missing ! ;-)

    (Hmm, why not make a Wiki-style book out of it?)

    By Anonymous Emre Sevinc, At 12/2/06 6:19 PM  

  • Thanks for all the comments - I hope they also help others who visit this post. This thread has received about four months worth of visitors in the last week, due to a link on reddit.com. Unfortunately, the comments on reddit have been much less useful than those posted here (imagine 100 copies of Chaitanya Gupta's comment).

    I had seen newLISP before, but I'm not too excited about it (it seems to have the speed of python but not all the power of lisp). It may serve as good learning platform, but I haven't tried it as such. It does sound extremely convenient, but in the name of performance and expressive power, I'm committing myself to using "traditional" systems like SBCL or OpenMCL.

    Having worked with Java in eclipse, cusp sounds exactly like something that I wanted to use. I eagerly jumped to their download page, only to discover that Mac/PPC is not supported. I suspect this has to do with thread support in SBCL/Mac-PPC. If anyone has experience with cusp, I would appreciate a review.

    After reading comments here and digging through the reddit thread, I have a number of responses brewing. I think they would be best formatted as another blog entry, so if you are interested, keep an eye on the RSS.

    Thanks again

    By Blogger ynniv, At 12/3/06 10:07 PM  

  • As Cusp's creator, I can assure you that the only reason ppc is not supported is that I don't have one. Theoretically, replacing Cusp's SBCL with your own ppc build of SBCL should be all you have to do. Somebody should try that out. And then zip it up and send me a copy.

    By Anonymous Tim Jasko, At 12/5/06 10:12 AM  

  • I took a couple of AI programming classes some time ago, they were taught in Lisp. They were most enjoyable classes, and Lisp had a lot to do with that. A couple of years later I tried to take a followup class. This used to be taught in Lisp also. The instructor had changed, the GTA was gone, and the language had been replaced with Java. It was ugly, every bit of it. I dropped the class. Another year, another institution, again tried to take a followup class. This time the instructor had switched to TCL. Much much worse. Dropped the class again.

    Instructors all over the place are switching to all kinds of "alternatives" to Lisp. Its definitely a trend and it is sad. Quality of AI classes is going down. Classic texts and software archives are no longer directly usable. Some 40+ years of accumulated pedagogic wisdom and culture is rapidly bleeding out of AI education. Lisp and AI community needs to get its act together, or the chapter on Lisp will be closing soon.

    By Anonymous Anonymous, At 1/18/07 1:33 AM  

  • This article and some of the comments are dead on. I bought a Common Lisp book that was published in this decade and gave up on it because of slime. Finally I found newlisp and liked it. I read a lot of negativity from hard-core lispers about newlisp but new lisp does right what these articles and comments say lisp does wrong. I can install it on Windows or Unix and follow tutorials that are relevant to today not 20 years ago.

    By Anonymous Anonymous, At 1/3/08 12:37 AM  

  • This is a great post! But title should be "How LISPers are killing Lisp". It has nothing to do with Python, Java, C#, and F# etc. It has everything to do with people/companies that are involved with Lisp. If we can't even make the tools easy and maybe even fun to use how can we convince people to learn lisp. Isn't it supposed to be easy to write great programs with lisp? Last time I checked programming tools are programs too.

    By Anonymous Anonymous, At 1/31/08 5:21 PM  

  • Clojure addresses many of these points.

    By Blogger k, At 12/12/09 9:58 PM  

  • +1 for newLISP, started using it a week ago and it's a joy so far.

    By Blogger blackdog, At 12/12/09 10:10 PM  

  • Don't think python will be becoming too popular behind the Great Firewall of China: www.python.org/download and subpages have been blocked for the past few months.

    By Anonymous Anonymous, At 12/13/09 1:34 AM  

  • IMHO Lisp is not for beginners.

    ESR - http://catb.org/~esr/faqs/hacker-howto.html#skills1
    If you don't know any computer languages, I recommend starting with Python. It is cleanly designed, well documented, and relatively kind to beginners. Despite being a good first language, it is not just a toy; it is very powerful and flexible and well suited for large projects

    LISP is worth learning for a different reason — the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot.

    So suppose you are experienced Python programmer and want to learn lisp:

    Go to freenode #lisp and get to know specbot and minion (bots) they provide documentation and general links aid which will help you.

    use PCL: http://www.gigamonkeys.com/book/
    If you have something to discuss and it is not stupid community on #lisp (including pcl author) are quite helpfull.

    You can do Lisp development both in Vim and Eclipse if you don't like Emacs. But If you don't like Emacs.. just get to know it better! For example type Alt+X in emacs and run erc, the emacs IRC client along SLIME. SLIME is not the only reason why so many people use Emacs.

    This is all about Common Lisp. Now, Scheme is a Lisp too! there is #scheme there is sicp there is htdp there (if you don't know what those are, do a lucky search) and there is PLT Scheme! There is Clojure also.

    In the end I want to remind you: Lisp is different, so is the community which by the way is more than 50 years old already.. maybe it is not the biggest community out there, but imho it is one of the best. If the fools who don't even know how to use Google properly are not part of it the better it is for it.vie

    By Blogger David Chkhikvadze, At 12/13/09 7:05 AM  

  • Lisp is back to the fore, albeit with a new name - Clojure.

    By Blogger Shantanu Kumar, At 12/13/09 12:19 PM  

Post a Comment


Create a Link

<< Home