Esh, the fabulous, forgotten Scheme implementation

In 1995, I moved from Ireland to the San Francisco Bay Area, because I’d been offered a job that promised to combine Unix and Scheme hacking. The prospect tickled me pink.

At Sun Microsystems, the SPARC design team used a home-built tool called DReAM to manage their ranch of servers as they ran huge batches of EDA design synthesis, verification and simulation runs. I was to join the DReAM group to maintain and extend the software.

DReAM was written in a combination of C++ and Scheme. The RPC and system code was all C++, while the scheduler was written in Scheme. It was a lovely piece of software to work on, and it scaled well. A single uninteresting server could handle a cluster of 3,000 servers with 600 clients without a hitch, not even breaking a sweat in the storm of network traffic after a power failure.

One reason DReAM scaled well was the choice of Scheme implementation for the scheduler. It was called esh (for Embeddable Shell); it was developed internally at Sun; and it was written by John Rose.

Esh compiled Scheme code to native (via C, as I recall), but it also had the usual interpreter, and interpreted and compiled code could call each other transparently. This doesn’t sound like a big deal if you’re used to Common Lisp, but it’s still a rare feature in Scheme implementations, 15 years later. The native code was idiomatic and performed well, and the environment coped just fine with native functions being replaced with interpreted versions at runtime.

Esh also had a very nice bidirectional foreign function interface, so it was easy to call into C from Scheme with a minimum of boilerplate, and vice versa.

We were able to write a lot of the core DReAM scheduling code in Scheme that was then compiled to native code, leaving “policy” decisions to interpreted code. I could telnet to a REPL to tweak variables, or replace entire functions with instrumented versions to hunt down problems on the fly without affecting the running system.

Unfortunately, by the time I joined Sun, John wasn’t working on esh any more; since there was essentially nobody else using it, it had already slipped into an early obscurity.

Even though I haven’t touched esh in ten years, I still remember it as a fine piece of software. A blog entry hardly qualifies as rescuing it from the Internet memory hole, but maybe John will be tickled to know that someone remembers his baby.

Posted in software
4 comments on “Esh, the fabulous, forgotten Scheme implementation
  1. I second your praise for esh (and for John, my college pal & roomie). I used esh in TBAG ( ) around ’91 or ’92 while at Sun. Lots of fun!

  2. Sounds interesting – perhaps the source code could still be dug up and made available?

    BTW, looks like there’s a paper on esh available from ACM’s digital library: Integrating the Scheme and C languages.

  3. John Rose says:

    Hi Bryan, hi Conal!

    …and what a pretty baby it was. Thanks for the trip down memory lane. Sun did authorize open-sourcing it, but before we got around to sanitizing the sources (non-trivial), Java exploded on the scene. So I’ve been doing Java for a decade.

    The most powerful and peculiar part of esh, the native interface, is described in the L&FP paper (Rose and Muller). I’ve posted an HTML copy.

    In the course of some of conversations about closures in Java, I was thinking again about the (very simple) bytecode design for esh.

    Here is some information; enjoy:

  4. John Rose says:

    Since is gone, I’ve moved the info bits here, for the record:

Leave a Reply

Your email address will not be published. Required fields are marked *