Making GHC’s I/O manager more scalable

Posted in Uncategorized
7 comments on “Making GHC’s I/O manager more scalable
  1. Andrey Popp says:

    Hi, first of all — thanks for your work on thit ticket. This is my the most awaited feature I am missing in GHC.

    Why do you consider using libev over raw calls to kqueue/epoll? As an another option, there is pure Haskell Event library by Johan Tibell.

  2. Johan Tibell says:

    If we had a common benchmark I could try to drop the event library in GHC base and see how it performs. It already supports select, epoll, and kqueue and has a very similar interface to the EventManager in your patch.

  3. Andrey Popp says:

    Johan, I’d like to see your event library in base. Maybe, we should move discussion to GHC trac?

  4. Johan, do you have a darcs or git repo for your event library handy?

    I’ll write a benchmark some time in the next few days. It’s not a very hard job: simply open some number of connections between a client and a server, and then send a byte over a random socket every so often.

    The tricky part is in accounting where the time spent goes. We might want to use the new event logging mechanism in 6.12 and ThreadScope for that.

  5. Kazu Yamamoto says:

    Hello,
    First of all, thank you for your work. I have been waiting for this kind work for a long time.

    I’m implementing a Web server
    with GHC. To get over the select() limitation, I wrote a C10K server library which use the *prefork* technique. That is, N processces are preforked, and a process limits acceptable connection to M. So, N * M connections can be handled.

    I would contribute the benchmark stuff since my team already
    know curl-loader and jMeter well. Please tell me what you want to measure…

    Keep in touch.

  6. Johan Tibell says:

    Bryan,

    It’s at http://github.com/tibbe/event

    We could use the libev benchmark for inspirations. It opens a number of pipes and sends a byte over each like you said.

  7. Brian Bloniarz says:

    Really interesting and promising stuff.

    FYI: I investigated libev for an application, had a pretty good impression overall — well thought out and functional. It had a major drawback for us, though: the main loop supports various timers, and to implement that it calls gettimeofday() twice for each call to poll(). This occurs regardless of whether you’ve registered any timers or not.

    We decided that was too much overhead for our app (lots of small UDP reads). It also didn’t help that libev uses a direct gettimeofday() syscall by default, instead of glibc’s faster vdso implementation (see EV_USE_CLOCK_SYSCALL).

4 Pings/Trackbacks for "Making GHC’s I/O manager more scalable"
  1. [...] couple of weeks, I have been working with Johan Tibbell on an event library to use for replacing GHC’s existing I/O manager. The work has been progressing rather nicely: I now have both the epoll and kqueue back ends [...]

  2. [...] Making GHC’s I/O manager more scalable – Dec 17, 2009 [...]

  3. [...] there is work to make the IO manager more scalable — now with a paper on the design :: [...]

  4. […] there is work to make the IO manager more scalable now with a paper on the design :: […]

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>