Progress on GHC’s I/O manager

Over the past couple of weeks, I have been working with Johan Tibbell on an event library to use for replacing GHC’s existing I/O manager. The work has been progressing rather nicely: I now have both the epoll and kqueue back ends working, while Johan has been focusing on a fast priority queue data structure for managing timeouts.

We’ve been working from a pair of git repositories:

(Incidentally, I’m using the excellent hg-git plugin for Mercurial, which has allowed me to continue to avoid using git. It’s been working very well. Many thanks to Scott Chacon and Augie Fackler for their work on it!)

Of course, now that we have the basic plumbing working, I have a few numbers to report. These are all for sending and receiving 1,000,000 messages over Unix pipes.

  • On 64-bit Linux, I can easily create 100,000 pipes (i.e. 200,000 file descriptors), and pass all messages through them in 6.69 seconds.
  • Under 32-bit Snow Leopard, for some reason I can’t create more than 2,048 pipes. Passing all messages takes 7.41 seconds.
  • With the same 2,048 pipes under Linux, the time required is 4.15 seconds.
  • Under Linux, if I try to create a very large number of pipes, I get an error message “VFS: file-max limit 291248 reached”, and as I just discovered, the machine starts to misbehave. Good times!

These are pretty gratifying numbers to have. This has been a very fun project so far, both in the technical parts and the dealings with the other people involved. I’m looking forward to continuing to work on it with Johan and others!

Posted in Uncategorized
5 comments on “Progress on GHC’s I/O manager
  1. Sam Martin says:

    Hi Bryan. Do you know what the state of the windows end is?

  2. Solaris guy says:

    Also, what about solaris? I know it’s kind of conservative, but it would be nice to have more flexibility with haskell programs.

  3. Dave Leimbach says:

    Hi Bryan,

    Try upping your ulimit on Mac OS X. It has some pretty low defaults.

    Dave

  4. The file-max limit on Linux can be increased by echoing a number into /proc/sys/fs/file-max.

  5. rinaldo sequitos says:

    Also curious about what is planned for the Windows side of things!

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>