Progress on GHC’s I/O manager

Over the past couple of weeks, I have been working with Johan Tibbell on an event library to use for replacing GHC’s existing I/O manager. The work has been progressing rather nicely: I now have both the epoll and kqueue back ends working, while Johan has been focusing on a fast priority queue data structure for managing timeouts.

We’ve been working from a pair of git repositories:

(Incidentally, I’m using the excellent hg-git plugin for Mercurial, which has allowed me to continue to avoid using git. It’s been working very well. Many thanks to Scott Chacon and Augie Fackler for their work on it!)

Of course, now that we have the basic plumbing working, I have a few numbers to report. These are all for sending and receiving 1,000,000 messages over Unix pipes.

  • On 64-bit Linux, I can easily create 100,000 pipes (i.e. 200,000 file descriptors), and pass all messages through them in 6.69 seconds.
  • Under 32-bit Snow Leopard, for some reason I can’t create more than 2,048 pipes. Passing all messages takes 7.41 seconds.
  • With the same 2,048 pipes under Linux, the time required is 4.15 seconds.
  • Under Linux, if I try to create a very large number of pipes, I get an error message “VFS: file-max limit 291248 reached”, and as I just discovered, the machine starts to misbehave. Good times!

These are pretty gratifying numbers to have. This has been a very fun project so far, both in the technical parts and the dealings with the other people involved. I’m looking forward to continuing to work on it with Johan and others!

Posted in Uncategorized
13 comments on “Progress on GHC’s I/O manager
  1. Sam Martin says:

    Hi Bryan. Do you know what the state of the windows end is?

  2. Solaris guy says:

    Also, what about solaris? I know it’s kind of conservative, but it would be nice to have more flexibility with haskell programs.

  3. Dave Leimbach says:

    Hi Bryan,

    Try upping your ulimit on Mac OS X. It has some pretty low defaults.


  4. The file-max limit on Linux can be increased by echoing a number into /proc/sys/fs/file-max.

  5. rinaldo sequitos says:

    Also curious about what is planned for the Windows side of things!

  6. Ramadan says:

    언어 바뀌는 것은 시스템 프리퍼런스의 international탭에 있는 input menu의 옵션(괜히 복잡해 보입니다;)에 있는 ‘Try to match koyreabd with text’ 옵션을 끄면 해결됩니다.

  7. Your writing style really helps in getting your point across here. I would be interested to know how much work you did to come up with these wonderfully original ideas.

  8. Hi Ami,Dan&Kids,I ejoyed reading your stories! Lilic started to bloom last month and still blooming. I love flowers! I shall take pictures and send to u. I heard this Saturday evening the moon will be closer to earth and brighter. Watch for it.Patti *

  9. Une situation qui résume une fois de plus toute la mainmise qu’à google sur les webamsters. Un bruit, une rumeur, et c’est la panique générale. A quand une concurrence digne de ce nom pour les faire vaciller un peu de leur piedestal ?

  10. Oh, Aidan, I am familiar with both this struggle and this surrender … I often feel that I’m failing on every single dimension of my life, and wonder if maybe I’d do better if I simply had fewer ‘sides,’ as you say. But then I return again to my image of being a kaleidoscope, not a laser, and to my deep sense that I need all of this aspects of identity, all of these roles, for some reason. And I try to surrender. But it’s hard. So, so hard. xox

  11. Your dress is amazing, and I'm loving that swan. I have worn a frock every day this week but today let the side down in a grey skirt and breton stripes – it is back to winter here in Scotland!

  12. He was MAGIC.At the end of an earlier film, also a generally realist set-up, a detective’s amputated fingers grow back. I think Miike just seems to delight in the strange game, even when he’s playing most straight….

  13. you couldn’t wait until whites were a minority in this country. Well.. why don’t you start by moving out and taking any kids you might have with ya.

Leave a Reply

Your email address will not be published. Required fields are marked *