1. What has changed

In a sentence, fewer backslashes. The notation is now more in line with that of scripting languages, where regular expressions are first-class literals, than that of general-purpose languages like C++ or Java, where regexes are just specialized strings.

Say we are given a stream including this text:

...
<img  src="images/11/apple1.gif"/>
<img   src="images/2/bulb2.jpeg"/>
<img src="images/354/citrus32_a.png"/>
...

We want to select IMG tags and capture the basename (without extension) of each source file. This can be done in many ways; here’s a blueprint which is just barely good enough:

<img [whitespace]+
     src=" [word-char]+ / [digit]+ / ([word-char]+) ...

Converting this to Clojure’s old syntax gives us a somewhat unwieldly #"<img\\s+src=\"\\w+/\\d+/(\\w+)". A quick test:

(let [lines "...
             <img  src=\"images/11/apple1.gif\"/>
             <img   src=\"images/2/bulb2.jpeg\"/>
             <img src=\"images/354/citrus32_a.png\"/>
             ..."]
  ;; Return only the captures, not the full matches.
  (map second
       (re-seq #"<img\\s+src=\"\\w+/\\d+/(\\w+)" lines)))
 
=> ("apple1" "bulb2" "citrus32_a")

The new update to the reader allows us to remove the double escaping of the regex specials in the literal:

(map second
     (re-seq #"<img\s+src=\"\w+/\d+/(\w+)" lines)))

2. Clojure vs foo

Since we’re on the topic, here’s how Clojure’s syntax compares to popular languages.

Ruby and Perl 5

# Regular usage
/<img\s+src="\w+\/\d+\/(\w+)/
 
# Choosing a different delimiter:
 m|<img\s+src="\w+/\d+/(\w+)|     # Perl
%r|<img\s+src="\w+/\d+/(\w+)|     # Ruby

The clearest of all extant languages (at least in this regard), Ruby and Perl can avoid some extra escaping by changing the delimiter character from / to |.

Emacs Lisp

"<img\\s-+src=\"\\w+/[[:digit:]]+/\\(\\w+\\)"

Well, the expression is long and ugly. An upside is that because of the quote delimiters, forward-slashes need not be escaped.

Java

"<img\\s+src=\"\\w+/\\d+/(\\w+)"

This is the same as Clojure’s original syntax. For reference, Clojure and Java share a regex engine and are equivalent in power.

Common Lisp

Edi Weitz’s professional CL-PPCRE package is essentially the standard for dealing with regular expressions in CL.

"<img\\s+src=\"\\w+/\\d+/(\\w+)"

Also, Edi’s CL-INTERPOL provides a reader macro which simplifies regex literals to the level of Perl’s:

#?r|<img\s+src="\w+/\d+/(\w+)|

Finally, the reader macro mastery of Doug Hoyte’s Let Over Lambda gives a method of making clear, functional literals:

;; This is a callable lambda:
;; #~m|<img\s+src="\w+/\d+/(\w+)|
 
'#~m|<img\s+src="\w+/\d+/(\w+)|
 
=> (LAMBDA (#:STR236)
     (CL-PPCRE:SCAN "<img\\s+src=\"\\w+/\\d+/(\\w+)"
                    #:STR236))

3. The real reason this is neat

The modification was proposed by Chris Houser (with a simple patch) on October 7, politely debated until October 10, and committed to Clojure in r1070 on October 15.* This syntax was better, and the discussion skipped if it should be applied, directly to how.

Turnaround time for a breaking change: one week. You have to respect that velocity.

There is a feeling in the development community that Clojure has a good chance of becoming an important language. Now is the brief time when any interested programmer could contribute something significant, in an environment which recognizes intelligent contribution and lacks — for the moment — politics and tradition.

Want to help? The Clojure mailing list is high signal-to-noise, and joining it is a good way to get acclimated. Also, communicating realtime with Rich Hickey and other Clojure experts is no more difficult than joining an IRC channel: #clojure on freenode.

* There was also a similar discussion in March, but it did not include a patch.

Note: no plugins are covered here, just vanilla Vim.

Essential .vimrc configuration items

For whatever reason, the following options are not set by default, but they should be.

1. Turn on hidden

   Don’t worry about the name. What this does is allow Vim to manage multiple buffers effectively.

   • The current buffer can be put to the background without writing to disk;
   • When a background buffer becomes current again, marks and undo-history are remembered.

   Turn this on.

   set hidden

2. Remap ` to '

   These are very similar keys. Typing 'a will jump to the line in the current file marked with ma. However, `a will jump to the line and column marked with ma.

   It is more useful in any case I can imagine, but it is located way off in the corner of the keyboard. The best way to handle this is just to swap them:

   nnoremap ' `
   nnoremap ` '

3. Map leader to ,

   The leader character is your own personal modifier key, as g is Vim’s modifier key (when compared to vi). The default leader is \, but this is not located standardly on all keyboards and requires a pinky stretch in any case.

   let mapleader = ","

   <SPACE> is also a good choice. Note: you can of course have several “personal modifier keys” simply by mapping a sequence, but the leader key is handled more formally.

4. Keep a longer history

   By default, Vim only remembers the last 20 commands and search patterns entered. It’s nice to boost this up:

   set history=1000

5. Enable extended % matching

   The % key will switch between opening and closing brackets. By sourcing matchit.vim, it can also switch among e.g. if/elsif/else/end, between opening and closing XML tags, and more.

   runtime macros/matchit.vim

   Note: runtime is the same as source except that the path is relative to the Vim installation directory.

6. Make file/command completion useful

   By default, pressing <TAB> in command mode will choose the first possible completion with no indication of how many others there might be. The following configuration lets you see what your other options are:

   set wildmenu

   To have the completion behave similarly to a shell, i.e. complete only up to the point of ambiguity (while still showing you what your options are), also add the following:

   set wildmode=list:longest

Recommended .vimrc configuration items

Most people like these.

1. Use case-smart searching

   These two options, when set together, will make /-style searches case-sensitive only if there is a capital letter in the search expression. *-style searches continue to be consistently case-sensitive.

   set ignorecase 
   set smartcase

   This is usually the most useful combination.

2. Set the terminal title

   A running gvim will always have a window title, but when vim is run within an xterm, by default it inherits the terminal’s current title.

   set title

   This gives e.g. | page.html (~) - VIM |.

3. Maintain more context around the cursor

   When the cursor is moved outside the viewport of the current window, the buffer is scrolled by a single line. Setting the option below will start the scrolling three lines before the border, keeping more context around where you’re working.

   set scrolloff=3

   Typing zz is also handy; it centers the window on the cursor without moving the cursor. (But watch out for ZZ!)

4. Store temporary files in a central spot

   Swap files and backups are annoying but can save you a lot of trouble. Rather than spreading them all around your filesystem, isolate them to a single directory:

   set backupdir=~/.vim-tmp,~/.tmp,~/tmp,/var/tmp,/tmp
   set directory=~/.vim-tmp,~/.tmp,~/tmp,/var/tmp,/tmp

   This is especially valuable after an unexpected reboot — you don’t have to track down all the leftover temp files. However: if you are editing files on a shared file system, it will be easier to clobber concurrent modifications, as other users’ Vim processes processes will not see your swaps.

5. Scroll the viewport faster

   <C-e> and <C-y> scroll the viewport a single line. I like to speed this up:

   nnoremap <C-e> 3<C-e>
   nnoremap <C-y> 3<C-y>

6. Enable limited line numbering

   It’s often useful to know where you are in a buffer, but full line numbering is distracting. Setting the option below is a good compromise:

   set ruler

   Now in the bottom right corner of the status line there will be something like: 529, 35 68%, representing line 529, column 35, about 68% of the way to the end.

7. A bunch of stuff your OS should already do

   If you are running Windows or OS X or a sloppy Linux distribution, you may not be using these:

   " Intuitive backspacing in insert mode
   set backspace=indent,eol,start
    
   " File-type highlighting and configuration.
   " Run :filetype (without args) to see what you may have
   " to turn on yourself, or just set them all to be sure.
   syntax on
   filetype on
   filetype plugin on
   filetype indent on
    
   " Highlight search terms...
   set hlsearch
   set incsearch " ...dynamically as they are typed.

   The filetype lines enable type-specific configuration, such as knowledge of syntax and indentation. E.g. foo.c will be opened with Vim’s pre-configured C settings, and bar.py will be opened with Python settings.

   If the search term highlighting gets annoying, set a key to switch it off temporarily:

   nmap <silent> <leader>n :silent :nohlsearch<CR>

8. Catch trailing whitespace

   The following will make tabs and trailing spaces visible when requested:

   set listchars=tab:>-,trail:·,eol:$
   nmap <silent> <leader>s :set nolist!<CR>

   By default whitespace will be hidden, but now it can be toggled with ,s.

9. Stifle many interruptive prompts

   The “Press ENTER or type command to continue” prompt is jarring and usually unnecessary. You can shorten command-line text and other info tokens with, e.g.:

   set shortmess=atI

   See :help shortmess for the breakdown of what this changes. You can also pare things down further if you like.

10. Stop distracting your co-workers

    Vim is a little surly, beeping at you at every chance. You can either find a way to turn off the bell completely, or more usefully, make the bell visual:

    set visualbell

    Instead of emitting an obnoxious noise, the window will flash very briefly. This is similar to screen’s interpretation of the bell in its default configuration.

Here is my own .vimrc, which includes all these settings (and some more which are less generally useful). A fairly good source for other configuration tips is the Vim Tips Wiki.

Thanks to Adam Katz and Chris Gaal for their input on a draft of this article.

1. Gatherers, like collecting or summing, for accumulating or reducing an expression;
2. Drivers, introduced with for or generate, for changing a value in a pattern or looping over a data structure.

The generate driver type did not appear in the article comparing LOOP and ITERATE (it is wholly a feature of ITERATE), so here first is a short introduction.

1. Generators

A generate clause is like a lazy for; its named variable will only update its value when told with NEXT. An example:

;; First, the regular `for`:
(iter (for el in '(a b c))
      (for i from 10)
      (when (primep i)
        (collect (list i el) into primes))
      (collect el into letters)
      (finally (return (values letters
                               primes))))
=> (A B C)
   ((11 B))
 
;; Now, using `generate` for el instead:
(iter (generate el in '(a b c))
      (for i from 10)
      (when (primep i)
        (collect (list i (next el)) into primes))
      (collect el into letters)
      (finally (return (values letters
                               primes))))
=> (NIL A A B B B B C C)
   ((11 A) (13 B) (17 C))

With these semantics, generate can appear anywhere for can appear:

(generate i upfrom 0)
(generate c in-string "black")
(generate form in-file "sexp.lisp")
(generate sym in-package :ITERATE)

2. Writing a new gathering clause

In the previous article I wrote a quick dividing clause:

(defmacro dividing (num &keys (initial-value 0))
  `(reducing ,num by #'/ initial-value ,initial-value))
 
(iter (for i in '(10 5 2))
      (dividing i :initial-value 100))
=> 1

This was easy, but it is not fully idiomatic; :initial-value must be specified as a keyword, and I can’t collect the value into a variable without modifying the macro. Updating the macro with this functionality makes it noticeably more complex:

(defmacro dividing (num &key (initial-value 0) into)
  `(reducing ,num by #'/
             initial-value ,initial-value
             ,@(when into
                 `(into ,into))))

We can solve both problems cleanly by using instead ITERATE’s DEFMACRO-CLAUSE:

(defmacro-clause (DIVIDING expr
                  &optional
                  INTO var
                  FROM start)
  "Divide into a variable"
  `(reducing ,expr by #'/ into ,var initial-value ,start))
 
;; Now this works:
(iter (for i in '(10 5 2))
      (dividing i from 100 into quotient)
      (multiplying i into product)
      (finally (return (values product quotient))))
=> 100
   1

DEFMACRO-CLAUSE takes care of setting up support for both keyword and regular symbol syntax, provides more informative error messages when the clause is misused, and adds the clause to DISPLAY-ITERATE-CLAUSES:

(display-iterate-clauses 'dividing)
DIVIDING &OPTIONAL INTO FROM  Divide into a variable

Lastly, to continue the LOOP idiom of naming clause actions in both their infinitival and present-participle form, we can use DEFSYNONYM:

(defsynonym divide dividing)

3. Writing a new driver

The easiest way to write new for clauses is with DEFMACRO-DRIVER. ITERATE offers in, on, in-vector, in-string, the more general in-sequence, in-hashtable, and more. Say we want to write a driver to iterate over the leaves of a tree. This is a little tricky since tree traversal is usually written recursively. To start us off, here is a function which will return all leaves:

(defun collect-leaves (tree)
  "Return all leaf nodes in depth-first order."
  (iter (with stack = (list tree))
        (while stack)
        (for node = (pop stack))
        (if (consp node)
            (destructuring-bind (l . r) node
              (unless (endp r)
                (push r stack))
              (push l stack))
            (collect node))))
 
(collect-leaves '(((a b) (c) d) e (f (g (h)) i) j))
=> (A B C D E F G H I J)

We need only splice this into a macro, with some modifications:

(defmacro-driver (FOR leaf IN-TREE tree)
  "Iterate over the leaves in a tree"
  (let ((gtree (gensym))
        (stack (gensym))
        (kwd (if generate 'generate 'for)))
    `(progn
       (with ,gtree = ,tree)
       (with ,stack = (list ,gtree))
       (,kwd ,leaf next
             (let ((next-leaf
                    (iter (while ,stack)
                          (for node = (pop ,stack))
                          (if (consp node)
                              (destructuring-bind (l . r)
                                  node
                                (unless (endp r)
                                  (push r ,stack))
                                (push l ,stack))
                              (return node)))))
               (or next-leaf (terminate)))))))

• The (if generate 'generate 'for) bit makes in-tree compatible with both generate and for.
• The form introduced by the symbol next is the code that will run on each iteration of for or on occurrence of the NEXT operator with generate.
• Here we’ve slipped an ITERATE call inside the driver, effectively nesting ITERATE loops. This wasn’t necessary; DO or LOOP could also have been used.
• TERMINATE is a special form in an ITERATE driver to indicate completion.

Finally, bringing it all together:

(iter (for leaf in-tree '(((2 3) (5) 1) 8 (4 (1 (2)) 2) 3))
      (collecting leaf into leaves)
      (multiplying leaf into product)
      (dividing leaf into quotient initial-value 2000)
      (finally (return (values leaves
                               product
                               quotient))))
=> (2 3 5 1 8 4 1 2 2 3)
   11520
   25/144

Appendix: Extending the LOOP macro

SBCL developer Richard Kreuter showed me an implementation-specific way to incorporate new syntax into LOOP. The linked code adds a for clause for binding multiple value returns (functionality that exists in ITERATE, but not the standard LOOP macro). This is meant less as a point of comparison than as proof that LOOP can indeed be extended.

loop-values-path.lisp

Usage:

(loop for dividend in '(1 2 3 4 5)
      for (quotient remainder) being the values of
          (floor dividend 2)
      do (format t "~D ~D~%" quotient remainder))
0 1
1 0
1 1
2 0
2 1

See also this file from CLSQL which provides LOOP extensions for iterating over records and supports seven Common Lisp compilers:

loop-extension.lisp

Thanks to Sam Freilich for his input on a draft of this article.

1. LOOP is a standard macro with an expressive syntax and built-in support for several iterative patterns. It is lauded and criticized in equal portion.
2. ITERATE is a second looping macro created outside of the standard as an answer to the problems and limitations of LOOP.

Superficially, the differences between them are few:

(loop for i from 0
      for el in '(a b c d e f)
      collect (cons i el))
=> ((0 . A) (1 . B) (2 . C) (3 . D) (4 . E) (5 . F)) 
 
(iter (for i from 0)
      (for el in '(a b c d e f))
      (collect (cons i el)))
=> ((0 . A) (1 . B) (2 . C) (3 . D) (4 . E) (5 . F))

The Common Lisp standard also includes the looping macros DO and DO*. They are widely used but can be terse and obscure, and they are not the subject of this article.

1. Views on LOOP

The following are the three most frequent criticisms of LOOP:

1. It does not look like Lisp. The degenerate case is hash table iteration. I need to look this up every time:

   (loop for key being the hash-keys in *some-table*
             using (hash-value val)
         collect (list key val))

   As a general compromise, some developers use keyword syntax for LOOP forms, though it is not a definite improvement:

   (loop :for counter :downfrom 20 :downto 0 :by 2
         :when (zerop (mod counter 3))
         :collect counter)
   => (18 12 6 0)

2. Editors do not auto-indent LOOP well. I work for one of the preeminent employers of Lisp developers, yet no one I know has an Emacs configuration which indents all corners of LOOP correctly. (That is, for my definition of correct; there is no standard.)
3. LOOP can behave unpredictably when for clauses interact in a certain way. However, in my experience this behaviour only presents itself in examples contrived to exhibit it.

For these reasons and others, Paul Graham does not recommend LOOP, and Peter Seibel remains neutral about it. But it does have its place.

See also these articles:

• Dan Weinreb discusses the tradeoffs in accepting LOOP into Common Lisp
• Joe Marshall’s “Why I am a knee-jerk anti-LOOPist”
• Geoff Wozniak’s “In defense of LOOP”.

2. The purpose of ITERATE

The ITERATE website makes two main claims:

1. It is extensible
2. It helps editors auto-indent by having a more Lisp-like syntax

Both are true. ITERATE is as much a mini-language as LOOP, but its clauses look like, and frequently are, regular Lisp forms.

;; A LOOP example
(loop for i upto 20
      if (oddp i)
        collect i into odds
      else
        collect i into evens
      finally (return (values evens odds))) 
=> (0 2 4 6 8 10 12 14 16 18 20)
   (1 3 5 7 9 11 13 15 17 19)
 
;; The equivalent in ITERATE
(iter (for i from 0 to 20)
      (if (oddp i)
          (collect i into odds)
          (collect i into evens))
      (finally (return (values evens odds))))
=> (0 2 4 6 8 10 12 14 16 18 20)
   (1 3 5 7 9 11 13 15 17 19)

The IF above is not an ITERATE construct — it is the normal Common Lisp operator. The LOOP if clause will be too, but only after a couple layers of macroexpansion.

Another example, harder to reproduce in LOOP:

(macrolet ((divisorp (n m)
             `(zerop (mod ,n ,m))))
  (iter (for i from 0 to 30)
        (cond ((divisorp i 2)
               (collect i into twos))
              ((divisorp i 3)
               (collect i into threes))
              ((divisorp i 5)
               (collect i into fives)))
        (finally (return (values twos threes fives)))))
=> (0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30)
   (3 9 15 21 27)
   (5 25)

It is true that distinguishing an ITERATE form from a regular form is less straightforward than it is with LOOP, but in practice the distinction is rarely important. A benefit of the intermixing is that ITERATE clauses such as collect may appear arbitrarily deep in the body, in contrast to LOOP where they must fall within the mini-language portion.

Also: for comparison with LOOP’s egregious hash table iteration syntax in (1), here is the cleaner equivalent in ITERATE:

(iter (for (key val) in-hashtable *some-table*)
      (collecting (list key val)))

ITERATE’s extensibility is discussed in-depth in a second article. Through some means, LOOP can apparently be extended, but it is not done often or easily. SBCL, at least, provides LOOP extension hooks, but using them ties you to that compiler.

3. Comparing looping clauses

In most cases, ITERATE is a superset of functionality.

*Accumulation*

LOOP offers collecting, nconcing, and appending. ITERATE has these and also adjoining, unioning, nunioning, and accumulating.

(iter (for el in '(a b c a d b))
      (adjoining el))
=> (A B C D)
 
(iter (for lst in '((a b c) (d b a) (g d h)))
      (unioning lst))
=> (A B C D G H)

accumulating is an accumulator builder. Here’s how to implement unioning:

(iter (for lst in '((a b c) (d b a) (g d h)))
      (accumulating lst by #'union))
=> (A B C G D H)

*Reduction*

LOOP has summing, counting, maximizing, and minimizing. ITERATE also includes multiplying and reducing. reducing is the reduction builder:

(iter (with dividend = 100)
      (for divisor in '(10 5 2))
      (reducing divisor by #'/ initial-value dividend))
=> 1

A simple macro to lessen code noise:

(defmacro dividing (num &keys (initial-value 0))
  `(reducing ,num by #'/ initial-value ,initial-value))
 
(iter (for i in '(10 5 2))
      (dividing i :initial-value 100))
=> 1

(Obviously the above is better stated (/ 100 10 5 2), but imagine leveraging this clause in a more complicated loop.)

The ITERATE package provides a macro, DEFMACRO-CLAUSE, to create new clauses more idiomatically; read more about it in the follow up article.

For comparison, this article describes one way of writing new reduction constructs for LOOP.

*Boolean aggregation*

These are the same in both: always, never, thereis, corresponding to the functions EVERY, NOTANY, and SOME. ITERATE can specify both always and never in the same loop, but not to much purpose.

*Finding*

There is no analogue for finding in LOOP. The ITERATE website provides a good use case:

(iter (for lst in '((a) (b c d) (e f)))
      (finding lst maximizing (length lst)))
=> (B C D) 
 
;; The rough equivalent in LOOP:
(loop with max-lst = nil
      with max-key = 0
      for lst in '((a) (b c d) (e f))
      for key = (length lst)
      do
      (when (> key max-key)
        (setf max-lst lst
              max-key key))
      finally (return max-lst))
=> (B C D)

finding is a pattern for using the result of one expression based on the result of another.

*Control flow*

ITERATE has next-iteration, which is like continue in C or next in Perl. It is a major inconvenience of LOOP that it does not have this construct. ITERATE also offers the (if-first-time then else) form and the first-iteration-p var for conditioning on the initial iteration in cases which aren’t covered by patterns.

*Destructuring*

In a for clause, ITERATE and LOOP have the same syntax and same destructuring capabilities. However, ITERATE can also “destructure” multiple-value returns:

(for (values (a . b) c d) = (three-valued-function ...))

The consing equivalent in LOOP:

for ((a . b) c d) = (multiple-value-list
                     (three-valued-function ...))

LOOP can do destructuring in a with clause, but ITERATE cannot. The manual cites implementation difficulty.

*Parallel binding*

This is what DO does and DO* does not, and it is strictly unsupported in ITERATE. LOOP includes this optionally with an and clause:

(loop for el in '(a b c d e)
      and prev-el = nil then el
      collect (list el prev-el))
=> ((A NIL) (B A) (C B) (D C) (E D))

The ITERATE documentation states:

“My view is that if you are depending on the serial/parallel distinction, you are doing something obscure”

I have to agree. Also, the useful case of parallel binding in the LOOP example above can be accomplished with another ITERATE concept, variable backtracking:

(iter (for el in '(a b c d e))
      (for prev-el previous el)
      (collect (list el prev-el)))
=> ((A NIL) (B A) (C B) (D C) (E D))

4. Documentation

There are many resources out there for learning to use LOOP. The LOOP for Black Belts chapter of Practical Common Lisp is my favourite. The Common Lisp Quick Reference is also excellent in its treatment of LOOP.

For ITERATE, there really is only the manual, but it is thorough. It includes another comparison of LOOP and ITERATE. There is also a DISPLAY-ITERATE-CLAUSES function which comes with the package and can provide dynamic assistance:

(display-iterate-clauses)
INITIALLY    Lisp forms to execute before loop starts
AFTER-EACH   Lisp forms to execute after each iteration
ELSE         Lisp forms to execute if loop is not entered
FINALLY      Lisp forms to execute after loop ends
;; ...
 
(display-iterate-clauses 'multiply)
MULTIPLY &OPTIONAL INTO   Multiply into a variable

It would be great to see this integrated into SLIME.

5. Obtaining ITERATE

It’s easiest to do this with ASDF-Install. To get ITERATE working in a repl or scratch buffer:

;; (Example below in SBCL.)
CL-USER> (asdf:oos 'asdf:load-op :ASDF-INSTALL)
;; stuff
 
CL-USER> (asdf-install:install "ITERATE")
;; lots of stuff
;; (only needs to be done once.)
 
CL-USER> (defpackage "MY-PACKAGE" (:use "CL" "ITERATE"))
 
#<PACKAGE "MY-PACKAGE">
CL-USER> (in-package "MY-PACKAGE")
 
#<PACKAGE "MY-PACKAGE">
MY-PACKAGE> (iter ...)  ;; etc.

6. Conclusion

In almost every case, ITERATE is more convenient to use and more powerful than LOOP. If you are in control of your own project and are not religiously partial to DO (or functional programming), ITERATE it is worth a try.

Thanks to Richard Kreuter, Alec Berryman, and John O’Laughlin for their input on a draft of this article.

Vim Script

Here is a neat lineage: the Unix line editor ed evolved into the more advanced ex which was used as the basis for the command mode in vi and extended into the roughly Turing-complete mini-language of Vim. And development continues; in his latest release, Bram Moolenaar has added native support for floating point numbers.

Vim Script supports many regular programming concepts: loops, lists, dictionaries, exceptions, etc. But the language is odd. Here is some code showing the hoops one must jump through to map a bit of functionality to a key:

function! s:doSomething()
  " stuff
endfunction
 
command DoSomething :call <SID>doSomething()
nmap k :DoSomething

• A trailing ! on function enables redefinition.
• The s: in the function definition and the <SID> in the command declaration are a thin but credible form of namespace management. They will expand to a unique name at read time so that similarly named functions in other files are not clobbered.
• function could be replaced equivalently with fu, fun, func, etc. This follows for all Vim commands. As long as a token can uniquely complete into a keyword, it is valid.

As in many other languages, statements can be wrapped using a \ character. Unlike in those languages, in Vim Script it must appear at the beginning of the succeeding line:

if some_exceedingly_long_expression ||
   \ a_second_expression
  echo 'Success'
endif

Alternative plugin languages

It is not widely known that Vim has interfaces into several popular scripting languages: Python, Ruby, Perl, Scheme, and Tcl. These are more powerful than Vim Script but have certain drawbacks in use.

1. Debugging is difficult. Foreign code is interpreted by what is essentially one giant eval. If you misplace a close parenthesis or an end keyword, you will have to track it down yourself.
2. Integration with Vim is slight. The calling interface is in a table below. Most interaction with the editor is tunneled through Vim::evaluate or Vim::command (or equivalent) as a string argument.
3. Many Vim installations do not include external language support by default. It is an easy fix for a user running a deb or rpm-based Linux distribution, but will require a recompile on Windows, something a Windows user is not wont to do.

Point (3) is particularly unfortunate if you are making an extension you intend to distribute. I wrote perhaps the largest Ruby-based plugin for Vim available; the majority of people who contact me about it are only looking for installation help!

This is what it looks like to interact with Vim from Ruby:

# Setting options inside the editor is pretty
# straightforward.
VIM::set_option "noinsertmode"
VIM::set_option "hlsearch"
 
# ...unless you want to set an option local to a buffer.
# There is no API call for this, so we must go up one
# layer of abstraction:
VIM::command "setlocal nowrap"
VIM::command "setlocal spell"
VIM::command "setlocal foldcolumn=0"

If we’re going to do an odd call in multiple places, it makes sense to add some glue:

def VIM::has_syntax?
  # All return values from `evaluate` are strings, and
  # "0" evaluates to true in ruby.
  VIM::evaluate('has("syntax")') != "0"
end

Obviously there is room for improvement in the API. See below the partial interfaces for a few languages:

(Tcl and Perl not shown.)

For some strange reason, the Scheme interface also offers (beep), and the Tcl interface, ::vim::beep.

As an interesting sidenote, these extension languages have access to window handles within Vim, allowing deterministic window management. Vim Script doesn’t seem to support this, so using an alternative language may offer a superset of functionality.

Choosing a language

This can be summarized like so:

Vim Script

Pros:

• Great integrated :help system.
• Lots of other plugins you can crib from.

Cons:

• The language is awkward.

Other (Perl, Python, Ruby, Tcl, Scheme)

Pros:

• Strong languages.
• Experience carries over into other pursuits.

Cons:

• Debugging is hard.
• Interface to Vim is slight.
• May require the user to install language libraries.
• Syntax highlighting in a .vim file is easily confused.
• Neglected by both plugin writers and Vim developers.

 

Using Vim Script means giving up the good stuff: closures, object-orientation, higher-order functions, reflection, and metaprogramming. So despite the extra work, selecting an alternative language is recommended for non-trivial extensions. Perhaps support within Vim will improve as more plugin writers follow this route.

Thanks to Jesse Funaro and Chris Gaal for their input on a draft of this post.

I fooled around with Clojure a little and wanted to share some of the things which haven’t gotten much play so far.

1. Implicit destructuring in binding forms

See examples below:

(def flat "flat")
(def tree '(("one" ("two")) "three" ((("four")))))
 
;; Simple binding (like Common Lisp's LET*).
(let [var1 flat
      var2 tree]
  (list var1 var2))
 
-> ("flat" (("one" ("two")) "three" ((("four")))))
 
;; Full destructuring.
(let [var1 flat
      [[a [b]] c [[[d]]]] tree]
  (list var1 a b c d))
 
-> ("flat" "one" "two" "three" "four")
 
;; Partial destructuring.
(let [[[a [b]] & leftover :as all] tree]
  (list a b leftover all))
 
-> ("one" "two" ("three" ((("four"))))
    (("one" ("two")) "three" ((("four")))))
 
;; Works on strings, too.
(let [[a b c & leftover] "123go"]
  (list a b c leftover))
 
-> (\1 \2 \3 (\g \o))

2. Unnamed arguments for short lambdas

Pretty straightforward:

(map #(+ % 4) '(1 2 3)) 
 
-> (5 6 7)
 
;; Multiple arguments.
(map #(* %1 %2) '(1 2 3) '(4 5 6))
 
-> (4 10 18)

This is roughly equivalent to Arc’s [+ _ 4] form, though allows for more than one argument. The standard lambda form is also similar to Arc’s:

(map (fn [x] (+ x 4)) '(1 2 3)
 
-> (5 6 7)

3. Data structures as functions

When a map or vector is called as a function, its argument is used as a value lookup.

(let [myvec [100 200 300]
      mymap {:a 1 :b 2 :c 3}]
  (list (myvec 1)
        (mymap :c)))
 
-> (200 3)

For maps, symbols and keywords can also work in a functional context:

(:a {:a 1 :b 2 :c 3})
 
-> 1
 
('b {'a 1 'b 2 'c 3})
 
-> 2

4. Implicit gensyms

Perhaps a minor point, but handy for cleaner macros.

(defmacro inc-safe [var]
  `(let [var# ~var]
     (if (integer? var#)
       (inc var#)
       0)))

In a backquoted form, ~var means to unquote (evaluate) var (like ,var in other Lisps) and var# creates a gensym whose name persists within the form. Compare to the equivalent in Common Lisp:

(defmacro 1+-safe (var)
  (let ((gvar (gensym)))
    `(let ((,gvar ,var))
       (if (integerp ,gvar)
           (1+ ,gvar)
           0))))

5. “[...]” syntax when code is not in an executing context

Look at the argument lists in any of the excerpts above to see what I mean. I’m kind-of on the fence on this one — on one hand it is semantically clearer, on the other it overloads the vector syntax. Also, I think round parentheses are just nicer looking.

6. SLIME integration

SLIME is the Superior Lisp Interaction Mode for Emacs, and it’s something you can’t live without once you’ve tried it. Clone this git repository and follow the instructions in the README.

$ git clone git://github.com/jochu/swank-clojure.git

There are, however, a couple design choices which are less appealing, perhaps especially to Common Lisp programmers.

1. Only single-value function returns

I believe this was briefly explained as being limited by the rules of stack allocation in the JVM. The restriction can be mitigated somewhat by returning a list of values which are then destructured:

(defn foo []
  (list 'aaa 'bbb))
 
(let [[a b] (foo)]
  (list b a))
 
-> (bbb aaa)

Common Lisp emulation in Emacs Lisp follows this path. It is a leaky abstraction, though, because foo in the example will always return a list. Even if we only care about the first value, we must acknowledge in every call that it returns others. Compare with Common Lisp’s GETHASH: it has two return values, yet in most contexts it is treated as having only one.

2. Iteration construct that looks more like recursion

Clojure is a functional language with immutable types. One might expect it would then tend toward recursive programming. Unfortunately, there is no tail-call optimization in Clojure currently, as it is not supported by the JVM; since it is certainly a sought feature, likely this will change soon. In the meantime, where efficiency matters one can use loop ... recur, which looks a lot like intra-function recursion.

(defn loop-test []
  (loop [collection (list)
         i 0]
    (if (< i 5)
      (recur (conj collection i) (inc i))
      collection)))
 
-> (4 3 2 1 0)

For simple iterative patterns, there are also for and doseq.

All in all, Clojure is very interesting. I should note I’ve only mentioned superficial stuff; the larger part of the second half of the presentation was Rich expounding on the ease of concurrency and the value of software transactional memory in the language, but these are larger topics (and over my head).

Thanks to Alec Berryman and George Polak for their input on a draft of this article.