Surveying Emacs Lisp

Published March 31st, 2009

Emacs is well known for being highly configurable. In large part, this is due to its close tie to some form of Lisp over the majority of its long and varied history. Within all extant implementations of the editor, the Lisp is Emacs Lisp, a language which is hard to love but easy to like.

1. Overview

Emacs Lisp (inconsistently abbreviated to Elisp) is descended from MacLisp and looks a lot like Common Lisp.

(defun fibonacci (n)
  (if (< n 2)
      1
    (+ (fibonacci (- n 1))
       (fibonacci (- n 2)))))

Being a Lisp, it is inherently extensible, and so is well-suited as an embedded language. Actually, a case can be made that Emacs Lisp is more powerful than many general-purpose programming languages. Some of its notable features:

  • lambdas and first-class functions
  • macros
  • byte-code compilation
  • an interactive shell/REPL (through the *scratch* buffer)
  • an integrated debugger and profiler

Considering its complete isolation to a sub-platform, Emacs Lisp is very successful. Tools that have been written in it include several IRC clients, a popular mail/news reader, two different terminal emulators, a web browser, and even a video editor. Here are several excellent programming resources I’ve found helpful:

2. Portability considerations

Emacs users tend to customize their editor to a great extent, so upgrading to a new version can be a dog, and therefore extension writers are quite conservative about portability.

And alas, the situation is doubly complicated. Developers must consider not only compatibility between versions (e.g. 21 to 22), but also compatibility between variants, as GNU Emacs and XEmacs are both popular.[1] My informal survey of ITA Software, an Emacs-heavy development house, seems to show that engineers stick with what they know; those who picked up Emacs during the periods GNU Emacs lagged behind XEmacs continue to use XEmacs, while most of the others use GNU Emacs. It’s split pretty evenly.

It’d be useful to identify the actual usage breakdown among the different versions of the editor, but these numbers don’t exist; Debian Popularity Contest can at least provide an estimate:

Tracked usage of Emacs packages in Debian
Package Installs Notes
emacsen-common 15890 (required package for all Emacs installs)
emacs21 7522 GNU
emacs22 4738 GNU
xemacs21 2211
emacs-snapshot 427 GNU (at the time of writing, v23.0)

Likewise, Ubuntu Popularity Contest:

Tracked usage of Emacs packages in Ubuntu
Package Installs Notes
emacsen-common 86805 (required package for all Emacs installs)
emacs21 32358 GNU
emacs22 16640 GNU
xemacs21 7394
emacs-snapshot 4764 GNU (at the time of writing, v23.0)

Again, at best these tables should be considered datapoints in a larger census, but they do seem to show GNU Emacs as significantly more popular than XEmacs, and 21 as the widest-used GNU Emacs version. (Can anyone provide additional stats?)

[1] Other variants such as Aquamacs may deserve a mention here as well — I have no information about their install bases.

3. A short case study

Two years ago I wrote a filesystem explorer / buffer switcher plugin for Vim and then one year ago brought it to Emacs. It had been written in Ruby, which has rough language parity with Emacs Lisp. (No macros vs. no closures.)

i. Initial development

Since Emacs Lisp is not really object-oriented, I dropped the original formal inheritance model. Also, I removed several classes outright, as existing functionality in Emacs — stuff which didn’t exist in Vim — made them redundant.

A minimal port was finished in a few hours and 200 lines-of-code. The current version sits at 494 lines. For comparison, the Vim plugin (on which development continues in parallel) is 1466 lines.

From the Vim plugin, here is an inefficient Ruby function to convert an array of strings into column_count columns (reading downward, i.e. the way ls works):

def columnize(strings, column_count)
  rows = (strings.length / Float(column_count)).ceil
 
  # Break the array into sub arrays representing columns
  cols = strings.inject([[]]) { |array, e|
           if array.last.size < rows
             array.last << e
           else
             array << [e]
           end
           array
         }
 
  return cols
end

And below, the close equivalent in Emacs Lisp. It uses push+nreverse instead of append (<<) above, which obfuscates the algorithm a little:

(defun columnize (strings column-count)
  "Break the list STRINGS into sublists representing columns."
  (let ((nrows (ceiling (/ (length strings)
                           (float column-count)))))
    (nreverse
     (mapcar 'nreverse
             (reduce (lambda (lst e)
                       (if (< (length (car lst))
                              nrows)
                           (push e (car lst))
                         (push (list e) lst))
                       lst)
                     strings
                     :initial-value (list (list)))))))

There is room for improvement in both cases — 10 pride points to the person who codes up the best rewrite of either function. 🙂

So far I’ve had to invest much less development time on the Emacs extension, though in fairness it has benefited from having an original to crib from. Also, it’s currently less featureful than the Vim plugin.

ii. Backporting

I had targeted GNU Emacs 23, since that’s the version I use. The port to GNU Emacs 22 was trivial. A port to GNU Emacs 21 is difficult for the lack of several convenience functions, so it’s on the back burner.

I spent about an hour on a tricky port to XEmacs before deciding that for two reasons, I couldn’t justify the effort:

  1. Ugly portability wrappers would complicate the otherwise compact code. A concise assert becomes an awkward block:[2] 
    ;; Before
(assert (minibufferp))
 
;; After
(assert (if (boundp 'xemacsp)
            (eq (window-buffer (minibuffer-window))
                (current-buffer))
          (minibufferp)))
  2. This package was written for my use, and I haven’t yet had reason to use XEmacs.

Perhaps if I had targeted GNU Emacs 21 as my baseline, portability wouldn’t have been as much of an issue; the library couldn’t have been as easy to write, however, and I believe side-projects especially should optimize for development time.

[2] A colleague — and also a more capable Emacs user — suggests to isolate portability wrappers as a partial solution. This is good practice in general. So, in perhaps a separate file:

(when (not (boundp 'minibufferp))
  (defun minibufferp ()
    (eq (window-buffer (minibuffer-window))
        (current-buffer))))
(provide 'lusty-xemacs)

Then, within the main code:

(when (boundp 'xemacsp)
  (require 'lusty-xemacs))
 
;; ...
 
;; Clean again!
(assert (minibufferp))

This is likely what I’ll do when I re-examine the XEmacs port.

4. The Common Lisp compatibility package

The comprehensive cl package provides many great extensions to Emacs Lisp which make the language more approachable, especially to Common Lisp programmers. Some of its nifty additions:

  • defun*, defmacro* (adding implicit blocks and keyword arguments)
  • flet, labels, macrolet
  • loop, do
  • lexical-let
  • multiple-value-bind, destructuring-bind
  • case, ecase, typecase
  • setf, incf/decf, define-modify-macro
  • push, pushnew, pop

Pragmatically, XEmacs loads this package by default. GNU Emacs, however, lightly discourages its use. From the reference manual:

… we have a policy that packages installed in Emacs must not load cl at run time. … If you are writing packages that you plan to distribute and invite widespread use for, you might want to observe the same rule.

And also:

Please don’t require the cl package of Common Lisp extensions at runtime. Use of this package is optional, and is not part of the standard Emacs namespace.

An imperfect workaround using eval-when-compile is advocated instead. I suggest to disregard the undertone of this note; Emacs Lisp benefits considerably from the inclusion of cl, and it’s time to move forward. In his article about Ejacs, Emacs expert Steve Yegge likewise recommends unrestrained loading of cl.

5. Conclusion

Under the all-important light of getting things done, Emacs Lisp is not a bad language. Emacs users are fortunate, and they should be thankful — users of other editors are not so lucky.

Thanks to Ron Gut and Chris Burke for their comments and suggestions.

Further discussion on the programming reddit

Published March 31st, 2009.


24 Responses to “Surveying Emacs Lisp”

  1. Volkan YAZICI Says:
    March 31st, 2009 at 11:09

    Here’s my try for columnize:

    (defun columnize (strings count)
  "Split the `strings' list into `count' sublists."
  (let ((l (ceiling (/ (length strings) (float count))))
        (sublists))
    (while strings
      (push (subseq strings 0 l) sublists)
      (setq strings (nthcdr l strings)))
    (nreverse sublists)))
  2. Stephen Bach Says:
    March 31st, 2009 at 11:41

    Volkan, that is much better! I’m almost embarrassed to leave the original one up, now. 🙂
  3. AndrewO Says:
    March 31st, 2009 at 11:48

    This is how I solved the your columnize problem in a previous project:


    class Array
    # Splits an Array into an Array of n sub-arrays
    def split_apart(num_ways)
    sub_length = (self.length.to_f / num_ways.to_f).ceil
    (0...num_ways).inject([]) do |arr, i|
    arr << self[i * sub_length..((i+1)*sub_length)-1]
    end.reject {|a| a.empty?}
    end
    end

    The only major difference is that I don’t iterate over every member of the array (instead looping over the number of columns). Not sure if there’s any real efficiency gain (and I’m not sure if the reject is necessary), but there it is. Do I get 10 points? 🙂
  4. AndrewO Says:
    March 31st, 2009 at 11:50

    Let’s try syntax highlighting this time:

    class Array
  # Splits an Array into an Array of n sub-arrays
  def split_apart(num_ways)
    sub_length = (self.length.to_f / num_ways.to_f).ceil
    (0...num_ways).inject([]) do |arr, i|
      arr << self[i * sub_length..((i+1)*sub_length)-1]
    end.reject {|a| a.empty?}
  end
end
  5. Anonymous Says:
    March 31st, 2009 at 12:30 
    (defun columnize (strings column-count)
  "Break the list STRINGS into sublists representing columns."
  (let ((nrows (ceiling (length strings) column-count)))
    (loop for i on strings by (lambda (x) (nthcdr nrows x))
          collect (subseq i 0 (min nrows (length i))))))
  6. Sami Samhuri Says:
    March 31st, 2009 at 12:39

    This is fairly idiomatic Ruby and is an order or two of magnitude faster by some quick informal tests.

    def clump(xs, n_cols)
  n_rows = (xs.length.to_f / n_cols).ceil
 
  # Break the array into sub arrays representing columns
  cols = []
  0.step(xs.size-1, n_rows) do |i|
    cols << xs[i..(i + n_rows - 1)]
  end
  return cols
end
  7. Stephen Bach Says:
    March 31st, 2009 at 15:02

    AndrewO, I like that you’ve extended Array. Makes sense.

    Anonymous, looks good, very succinct. Also, the min check
    in subseq is key.

    Sami, faster and shorter.
  8. Brian Oblivion Says:
    March 31st, 2009 at 15:29 
    (defun columnize (list n)
  (when list
    (cons (subseq list 0 (min n (length list)))
          (columnize (nthcdr n list) n))))
  9. Anselm Says:
    March 31st, 2009 at 16:25

    You said that you can use the scratch buffer as a REPL replacement in emacs. I’d like to add that you can get a “real” REPL with M-x ielm. But I find evaluating my elisp code from the scratch or lisp buffers much more convenient.
  10. Luke Gorrie Says:
    March 31st, 2009 at 17:40

    Good article!

    Beware: The cl package’s macros make backtraces in the debugger pretty miserable!

    Consider the elisp’ey:

    (mapcar (lambda (n) (/ 5 n)) '(1 2 0))
Debugger entered--Lisp error: (arith-error)
  /(5 0)
  (lambda (n) (/ 5 n))(0)
  mapcar((lambda (n) (/ 5 n)) (1 2 0))
  eval((mapcar (lambda (n) (/ 5 n)) (quote (1 2 0))))

    to the cl-package macro expanded:

    (dolist (n '(1 2 0)) (/ 5 n))
Debugger entered--Lisp error: (arith-error)
  /(5 0)
  (while --cl-dolist-temp-- (setq n (car --cl-dolist-temp--)) (/ 5 n) (setq --cl-dolist-temp-- (cdr --cl-dolist-temp--)))
  (let ((--cl-dolist-temp-- ...) n) (while --cl-dolist-temp-- (setq n ...) (/ 5 n) (setq --cl-dolist-temp-- ...)) nil)
  (catch (quote --cl-block-nil--) (let (... n) (while --cl-dolist-temp-- ... ... ...) nil))
  (cl-block-wrapper (catch (quote --cl-block-nil--) (let ... ... nil)))
  (block nil (let (... n) (while --cl-dolist-temp-- ... ... ...) nil))
  (dolist (n (quote ...)) (/ 5 n))
  eval((dolist (n (quote ...)) (/ 5 n)))

    Just another factor to trade-off each time you’re tempted to use a cl macro.
  11. Stephen Bach Says:
    March 31st, 2009 at 20:29

    Brian, I was waiting for a recursive approach.

    Anselm, thanks. I didn’t know about ielm.

    Luke, good point. There’s always a catch.
  12. Smug Vector Weenie Says:
    March 31st, 2009 at 23:54

    in q,:

    columnize:{flip r cut x,(r-(count x)mod r:(count x)div y)#enlist""}
  13. links for 2009-03-31 « My Weblog Says:
    April 1st, 2009 at 00:03

    […] Surveying Emacs Lisp – items.sjbach.com (tags: emacs) […]
  14. Fester Says:
    April 1st, 2009 at 06:53

    Maybe I am missing something from your version of columnize, but maybe you should consider this:

    def columnize(strings, columns)
  cols = []
  strings.each_slice(columns){|x| cols << x}
  return cols
end
  15. Stephen Bach Says:
    April 1st, 2009 at 13:41

    A friend points out that because of Emacs Lisp’s dynamic scope, the lambda in my columnize function is of uncertain correctness; if the implementation of reduce happens to bind a variable named nrows, the sky will fall.

    None of the proposed rewrites show that mistake, so good for you folks. 🙂
  16. bitti Says:
    April 2nd, 2009 at 14:54

    For columnize use each_slice:

    def columnize(strings, column_count)
    cols = []
    strings.each_slice(column_count) { |row| cols << row }
    cols
    end
  17. bitti Says:
    April 2nd, 2009 at 14:57

    Sorry, forgot the pre tag:

    def columnize(strings, column_count)
  cols = []
  strings.each_slice(column_count) { |row| cols << row }
  cols
end
  18. Stephen Bach Says:
    April 2nd, 2009 at 23:09

    Fester, bitti, your versions of columnize actually ensure there will be column_count rows, not columns. Slightly different.
  19. Ryan Davis Says:
    April 11th, 2009 at 05:23

    I’d actually come up with this before reading the numerous comments. The only thing bitti needed was to calculate the number of rows and use that for each_slice.

    require 'enumerator'
 
def new_columnize(strings, column_count)
  r = []
  rows = (strings.size / column_count.to_f).round
  strings.each_slice(rows) { |a| r << a }
  r
end
  20. Stephen Bach Says:
    April 11th, 2009 at 13:13

    Ryan, it’s a little counterintuitive, but even with your change the function doesn’t work exactly right. Try this input to see:

    strings = "a b c d e f g".split
column_count = 3
  21. linkfeedr » Blog Archive » lisp - RSS Indexer (beta) Says:
    April 14th, 2009 at 15:57

    […] This article was found on emacs life. Click here to visit the full article on the original website. Surveying Emacs Lisp – A good article that looks at Emacs Lisp and a little Ruby. … more on the original website […]
  22. Charmain Panama Says:
    February 4th, 2010 at 05:45

    There is evidently a lot for me to study outside of my books. Thanks for the important read,
  23. items.sjbach.com » Blog Archive » Extensibility in Vim and Emacs Says:
    April 7th, 2012 at 16:02

    […] I’ve written previously on writing Vim plugins and using Emacs Lisp. […]
  24. Robbie Morrison Says:
    August 31st, 2012 at 03:59

    Regarding the elisp columnize function, the EmacsWiki site has
    several useful links:

    http://www.emacswiki.org/emacs/ColumnizeWords

    http://www.emacswiki.org/emacs/CategoryAlignment

    http://www.emacswiki.org/emacs/columnize.el