Ease of use

The new decode function complements the longstanding encode function, and makes the API simpler.

New examples make it easier to learn to use the package.

Generics support

aeson’s support for data-type generic programming makes it possible to use JSON encodings of most data types without writing any boilerplate instances.

Thanks to Bas Van Dijk, aeson now supports the two major schemes for doing datatype-generic programming:

• the modern mechanism, built into GHC itself

• the older mechanism, based on SYB (aka "scrap your boilerplate")

The modern GHC-based generic mechanism is fast and terse: in fact, its performance is generally comparable in performance to hand-written and TH-derived ToJSON and FromJSON instances. To see how to use GHC generics, refer to examples/Generic.hs.

The SYB-based generics support lives in Data.Aeson.Generic, and is provided mainly for users of GHC older than 7.2. SYB is far slower (by about 10x) than the more modern generic mechanism. To see how to use SYB generics, refer to examples/GenericSYB.hs.

Improved performance

• We switched the intermediate representation of JSON objects from Data.Map to Data.HashMap, which has improved type conversion performance.

• Instances of ToJSON and FromJSON for tuples are between 45% and 70% faster than in 0.3.

Evaluation control

This version of aeson makes explicit the decoupling between identifying an element of a JSON document and converting it to Haskell. See the Data.Aeson.Parser documentation for details.

The normal aeson decode function performs identification strictly, but defers conversion until needed. This can result in improved performance (e.g. if the results of some conversions are never needed), but at a cost in increased memory consumption.

The new decode' function performs identification and conversion immediately. This incurs an up-front cost in CPU cycles, but reduces reduce memory consumption.

You may have seen my exciting news about our upcoming move to Facebook.

It’s been a total blast working on our product, and of course as we did so we released a number of open source libraries and tools. It only added to our pleasure to see so much of that code used outside of our own domain. I will continue to develop and maintain the code that we have released.

Here is a quick rundown of the code we have released, roughly ordered by significance. Yep, we wrote all of these projects in Haskell, definitely a decision that in retrospect I’m very happy about.

• pronk (not yet actually released) is an application for load testing web servers. Think of it as similar to httperf or ab, only more modern, simpler to deal with, and with vastly better analytic and reporting capabilities.

• configurator is a library that allows fast, dynamic reconfiguration of a Haskell application or daemon.

• aeson is a JSON encoding and decoding library optimized for high performance and ease of use.

• text-format is a library for printf-like text formatting.

• mysql-simple is an easy-to-use client library for the MySQL database. It is several times faster than its competitors, and easier to use. It is built on top of the low-level mysql library.

• riak-haskell-client is a client for the Riak decentralized data store.

• blaze-textual is a library for efficiently rendering Haskell data as text.

• double-conversion is a very fast library for rendering double precision floating point numbers as text, based on the code from the V8 Javascript engine.

• resource-pool is a fast resource pooling library.

• snappy provides Haskell bindings to Google’s extremely fast snappy compression library.

• base16-bytestring provides fast handling of base16-encoded data.

• hdbc-mysql provides a MySQL transport for the HDBC database access library. (Yes, we recommend using mysql-simple instead!)

Thanks to all of you who have contributed patches and bug reports. It’s going to be an exciting future!

I'd particularly like to thank Alexey Khudyakov for his wonderful work on this release.

New features:

• Student-T, Fisher-Snedecor, F-distribution, and Cauchy-Lorentz distributions are added.

• Histogram computation is added, in Sample.Histogram.

• Forward and inverse discrete Fourier and cosine transforms are added, in Transform.

• Root finding is added, in Math.RootFinding.

Major changes:

• The Sample.KernelDensity module has been renamed, and completely rewritten to be much more robust. The older module oversmoothed multi-modal data. (The older module is still available under the name Sample.KernelDensity.Simple).

• The type classes Mean and Variance are split in two. This is required for distributions which do not have finite variance or mean.

Smaller changes:

• The complCumulative function is added to the Distribution class in order to accurately assess probalities P(X>x) which are used in one-tailed tests.

• A stdDev function is added to the Variance class for distributions.

• The constructor Distribution.normalDistr now takes standard deviation instead of variance as its parameter.

• A bug in Quantile.weightedAvg is fixed. It produced a wrong answer if a sample contained only one element.

• Bugs in quantile estimations for chi-square and gamma distribution are fixed.

• Integer overlow in mannWhitneyUCriticalValue is fixed. It produced incorrect critical values for moderately large samples. Something around 20 for 32-bit machines and 40 for 64-bit ones.

• A bug in mannWhitneyUSignificant is fixed. If either sample was larger than 20, it produced a completely incorrect answer.

• One- and two-tailed tests in Tests.NonParametric are selected with sum types instead of Bool.

• Test results returned as enumeration instead of Bool.

• Performance improvements for Mann-Whitney U and Wilcoxon tests.

• Module Tests.NonParamtric is split into Tests.MannWhitneyU and Tests.WilcoxonT

• sortBy is added to Function.

• Mean and variance for gamma distribution are fixed.

• Much faster cumulative probablity functions for Poisson and hypergeometric distributions.

• Better density functions for gamma and Poisson distributions.

• The function Function.create is removed. Use generateM from the vector package instead.

• A function to perform approximate comparion of doubles is added to Function.Comparison.

• Regularized incomplete beta function and its inverse are added to Function.

Strange Loop is a pretty good approximation to my ideal conference, covering a narrow family of topics I’m interested in, mainly leading-edge matters in programming languages and distributed systems. The focus is not at all academic, instead being on open source software that you can download and get to work with. I have to strongly commend the organizers for finding a great venue, excellent speakers, and running a fabulous show at an impressively low price.

I’ve published the source for the slides from both my workshop and talk in a github repository, but you might prefer to look at the slides directly:

• My Haskell workshop (141 slides, and we managed to cover the whole lot at a pleasant pace!)

• My talk on startups and Haskell (41 slides)

I buried a teaser slide 3/4 of the way through the startup talk, which turned out to be an excellent way to accidentally find out how many people on Twitter were reading through the slides (the answer surprised me: a lot!). The enigmatic bug in question was quite spectacular: a development version of GHC would delete a source file if it contained a type error. Ouch!

One of the people in the capacity crowd at the Haskell workshop was none other than Gerry Sussman, who was finally learning Haskell for the first time (doubtless after several decades of overexposure to parentheses). He pronounced the workshop "pretty good", which I believe is MIT-speak for "OMG teh awesome!"

At one point, Gerry had a pretty amusing epigram to offer. "Haskell is the best of the obsolete programming languages!" he pronounced, with a mischievous look. Now, I know when I’m being trolled, so I said nothing and waited a moment, whereupon he continued, "but don’t take it the wrong way – I think they’re all obsolete!"

For instance, in this case, I could immediately tell by inspection that something bad was afoot in the inner loop of the UTF-8 decoder. A decoder more or less has to read a byte of input at a time, as this heavily edited bit of Core illustrates:

let x2 :: Word8
    x2 = case readWord8OffAddr# {- ... -} of
           (# s, x #) -> W8# x
in {- ... -}

What's important in the snippet above is that the value x2 is boxed, i.e. packaged up with a W8# constructor so that it must be accessed via a pointer indirection. Since a decoder must read up to 4 bytes to emit a single Unicode code point, the loop was potentially boxing up 4 bytes, then immediately unboxing them:

case x2 of 
  W8# x2# ->
    case x3 of
      W8# x3# ->
        case x4 of
          W8# x4# -> {- ... -}

While both boxing and unboxing are cheap in Haskell, they're certainly not free, and we surely don't want to be doing either in the inner loop of an oft-used function.

We can see why this was happening at line 96 of the decodeUtf8With function. I'd hoped that the compiler would be smart enough to unbox the values x1 through x4, but it turned out not to be.

Fixing this excessive boxing and unboxing wasn't hard at all, but it made the code uglier. The rewritten code had identical performance on pure ASCII data, but was about 1.7 times faster on data that was partly or entirely non-ASCII. Nice! Right?

Not quite content with this improvement, I tried writing a decoder based on Björn Höhrmann's work. My initial attempt looked promising; it was up to 2.5 times faster than my first improved Haskell decoder, but it fell behind on decoding ASCII, due to the extra overhead of maintaining the DFA state.

In English-speaking countries, ASCII is still the king of encodings. Even in non-English-speaking countries that use UTF-8, a whole lot of text is at least partly ASCII in nature. For instance, other European languages contain frequent extents of 7-bit-clean text. Even in languages where almost all code points need two or more bytes to be represented in UTF-8, data such as XML and HTML still contains numerous extents of ASCII text.

What would happen if we were to special-case ASCII? If we read a 32-bit chunk of data, mask it against 0x80808080, and get zero, we know that all four bytes must be ASCII, so we can just write them straight out without going through the DFA (see lines 82 through 110).

As the numbers suggest, this makes a big difference to performance! Decoding pure ASCII becomes much faster, while both HTML and XML see respectable improvements. Of course, even this approach comes with a tradeoff: we lose a little performance when decoding entirely non-ASCII text.

Even in the slowest case, we can now decode upwards of 250MB of UTF-8 text per second, while for ASCII, we exceed 1.7GB per second!

These changes have made a big difference to decoding performance across the board: it is now always between 2 and 4 times faster than before.

As a final note, I haven't released the new code quite yet - so keep an eye out!

Prior to Steele and White’s "How to print floating-point numbers accurately", implementations of printf and similar rendering functions did their best to render floating point numbers, but there was wide variation in how well they behaved. A number such as 1.3 might be rendered as 1.29999999, for instance, or if a number was put through a feedback loop of being written out and its written representation read back, each successive result could drift further and further away from the original.

Steele and White effectively solved the problem with a clever algorithm named "Dragon4" (the fourth version of the "Dragon" algorithm, which acquired its name because the authors were inspired to obscure puns by Heighway's dragon curve).

The Dragon4 algorithm spread quickly across language runtimes, such that few programmers today understand that this was ever a problem, much less how hairy it was (and is). Indeed, prior to last year, there was almost no activity in this area: two papers proposed widely used refinements to Dragon4, and that was about it. (Alas, the problem was originally solved around a decade before Steele and White published their work, but nobody noticed. If you have a clever idea and sufficient chutzpah, try to enlist Guy Steele as a coauthor. Your work will be read.)

But how solved was the problem? Dragon4 and its derivatives are complicated and tricky, and they have a hefty performance cost, since they rely on arbitrary-precision integer arithmetic to compute their results. There might be a significant performance improvement to be gained if someone could figure out how to use native machine integers instead.

In 2010, Florian Loitsch published a wonderful paper in PLDI, "Printing floating-point numbers quickly and accurately with integers", which represents the biggest step in this field in 20 years: he mostly figured out how to use machine integers to perform accurate rendering! Why do I say "mostly"? Because although Loitsch's "Grisu3" algorithm is very fast, it gives up on about 0.5% of numbers, in which case you have to fall back to Dragon4 or a derivative.

If you're a language runtime author, the Grisu algorithms are a big deal: Grisu3 is about 5 times faster than the algorithm used by printf in GNU libc, for instance. A few language implementors have already taken note: Google hired Loitsch, and the Grisu family acts as the default rendering algorithms in both the V8 and Mozilla Javascript engines (replacing David Gay's 17-year-old dtoa code). Loitsch has kindly released implementations of his Grisu algorithms as a library named double-conversion.

And of course I can't talk about performance without mentioning Haskell somewhere :-) I've taken Loitsch's library and written a Haskell interface, which I've measured to be 30 times faster than the default renderer used in the Haskell runtime libraries. This has some nice knock-on effects: my aeson JSON library is now 10 times faster at rendering big arrays of floating point numbers, for instance. I accidentally noticed in the course of that work that my Haskell text Unicode library's UTF-8 encoder wasn't as fast as it could be, so I improved its performance by about 50% along the way. Hooray for faster code!

(By the way, the punnery in algorithm naming continues: the Grisu algorithms are named for Grisù, the little dragon.)

Original article, for posterity:

A couple of days ago, I released version 0.9 of my attoparsec parsing library. Although its visible API remains completely unchanged over 0.8, I made a large and important change to the semantics. So if you use attoparsec, read on before you bump your build dependencies!

Prior to 0.9, attoparsec backtracked too aggressively, and hence held onto too much memory.

Consider the following parser:

(char 'f' *> char 'o') <|> (char 'f' *> char 'i')

Under attoparsec 0.8 and earlier, the <|> (choice) operator saves the input state before executing the left branch. If the left branch fails, it restores the input state, then executes the right branch. So if we supply an input of "fo", the left branch will succeed, and so the whole parser succeeds. But if we supply an input of "fi", the left branch will fail. The <|> operator will restore the input to "fi", and then the right branch will succeed.

This behaviour is very appealing, because it makes it easy to write readable parsers. Unfortunately, it comes with a big downside: it makes it too easy to write inefficient parsers, since <|> needs to hold onto the entire input of its left branch in case it fails.

In attoparsec 0.9, I've dropped that backtracking-by-default behaviour. This will require changes to some parsers that use attoparsec; I'll describe what's involved below. (By the way, this change makes attoparsec's behaviour more compatible with the classic Parsec library.)

Given the parser above, here's how it will now behave when given an input of "fi". The left branch will succeed on its first match against 'f', so the input will become "i" after 'f' has been consumed. At this point, the left branch will fail because it cannot match 'o' against the input of "i". The <|> combinator will switch to executing the right branch, but it will not reset the input. At this point, the right branch will attempt to match 'f' against an input that is still "i", and will thus fail.

This change affects not just the <|> combinator, but also its synonyms mplus and mappend. It also affects the behaviour of higher-level combinators that are constructed using <|>, such as many.

If your parser is affected by this change, there are two good courses of action.

The easier path is to use the try combinator on the left hand side of a <|> branch. This combinator causes a parser to backtrack on failure: if a parser wrapped by try fails, it will reset the input before passing the failure along. What would this look like in practice?

try (char 'f' *> char 'o') <|> (char 'f' *> char 'i')

This is certainly easy to do, and it won't affect the performance of your code compared to attoparsec 0.8: you'll have exactly the same backtracking and memory usage behaviours as before.

There is an alternative that doesn't require backtracking, and which should make your parsers both faster and less memory-hungry: the classic technique of left factoring your grammar. For the toy parser above, this is easy to do:

char 'f' *> (char 'o' <|> char 'i')

It's clear that we'll see a small performance boost here because we only match against 'f' once in this parser, while our first version did some unnecessary extra work by matching 'f' on each side of the branch. Left factoring is not always this easy, and it can make parsers more difficult to read, so sometimes using try is the better course to pursue.

Given that this change in semantics introduces some awkwardness, you might reasonably wonder why I made it. By making the use of backtracking explicit, parsers that don't need backtracking will become faster "for free". A realistic instance of this is my aeson JSON library, where by using the no-backtracking version of attoparsec, I saw improvements of up to 10% in parsing performance, and slightly larger reductions in memory footprint.

In September, I’ll be teaching a Haskell workshop at the Strange Loop Conference in St Louis. Here’s the abstract:

Modern programming presents a daunting array of challenges: proliferating technologies, messy inputs, unreliable networks, huge volumes of data, how to verify that results are correct, making it all fast enough. The Haskell programming language is well suited to addressing this broad range of needs, as it uniquely combines conciseness, safety, and high performance.

In this fast-moving, interactive tutorial we will learn Haskell by developing a realistic analytic application: we’ll crawl a web site and rank its pages in order of authority. This combines some very modern concerns: network programming; handling dodgy HTML; big data; and number crunching. With our emphasis on getting real work done, we’ll show off some of Haskell’s compelling features and demonstrate how they help us to develop dependable, easy to understand code.

There’s a ton of other great speakers lined up for this year’s Strange Loop, so if you like learning from, and talking with, people who are excited about software, this will be a great conference to attend.

As if that weren’t enough, I’m excited to be working with David Mazières on a new class at Stanford University, "CS240H: Functional Systems in Haskell".

As far as I know, this will be the first academic course of its kind, where we focus on the use of functional programming techniques to build solid, fast, secure systems software. Intrinsic merits of the class aside, it’s going to be a lot of fun to teach this class this in the educational cradle of Silicon Valley, where systems software is the bread and butter of a lot of research and commercial innovation.

We’ll start off with some familiar topics: the basics of Haskell, laziness, monads, parsers, and all that. But from there, we’ll integrate testing, reliability, and debugging; performance tuning; interfacing to native code; concurrency and I/O paradigms; language extensions; meta-programming; and applications to the web and security.

We’re also looking at bringing in some prominent guest lecturers from the Haskell community to talk about topics of interest to them. I’ll name names when we have them confirmed.

David and I are both very much motivated by writing code that’s clean and fast, so as much as we’ll spend time talking about theory, research, and academics, this will be a very hands-on "let’s bang out some awesome software!" kind of affair.

Here’s a rough outline of what the course will involve:

• A couple of "Haskell basics" classes.

• Laziness: its applications and pitfalls.

• Monads and parsers; programming in continuation-passing style.

• Tricks for debugging and performance tuning.

• Reliability: design for robustness; property-based testing with QuickCheck; code coverage.

• The Iteratee/Enumerator paradigm for I/O.

• How to use the foreign function interface, build interfaces to native code, deal efficiently with data in various contexts, and generally think about memory. Possibly also coupled with discussion of the Haskell networking API.

• Concurrency and parallelism: event- vs thread-based concurrency, Software Transactional Memory, parallel Haskell, data parallel Haskell, repa.

• Library-level optimization: function specialization, dictionary passing, RULES pragmas, stream fusion.

• All the gnarly things you can do with functional dependencies and type families, as well as other relevant extensions such as GADTs.

• How Haskell is implemented, to give students some idea of what is actually going on.

• Generic programming: SYB and fun with Data.

• Web-related topics? formlets, plus maybe some cool Continuation type tricks. Haskell database interfaces and interesting things you can do with types, HList/HaskellDB.

• Language-level information flow control.

• Case studies of real systems in Haskell.

I know that some SF Bay Area locals have questions about the possibility of auditing the class, whether materials will be available online, and the like. I don’t have answers yet, but stay tuned.

And the performance news is good: on real-world data, I’ve improved encoding performance by about a factor of 4. Why don’t we let the graphs do the talking.

Enjoy!

About a week ago, I received a message from Finlay Thompson asking about my Haskell statistics package: he wanted to know how to generate pseudo-random variables using it. I redirected him from that to my mwc-random package, where my pseudo-random number generation code lives.

The mwc-random package currently provides generators for two widely used distributions: uniform and normal. When I was originally writing it, I paid particular attention to making it high quality, fast, and easy to use.

"High quality" sounds a little nebulous, but in the world of pseudo-random number generation, it's actually pretty well defined: a good PRNG should have a large period (the number of samples you need to pull out of it before it repeats itself, assuming a good seed), and the numbers it generates should withstand stringent tests of apparent independence (simply put, given one datum, you shouldn't be able to predict the next).

One algorithm that satisfies these criteria of quality is George Marsaglia's multiply-with-carry algorithm MWC256 (also known as MWC8222). It has a period of about 2⁸²²² (huge enough for all conceivable practical purposes), and stands up well to the "testu01", "diehard" and "big crush" statistical tests.

Due to its simplicity, MWC256 is also very fast, and under appropriate circumstances (e.g. on a 64-bit machine) it can be even faster than the well known Mersenne Twister algorithm (which also fails some statistical tests that MWC256 passes).

The Mersenne Twister is itself available for Haskellers to use, in the form of the mersenne-random package. This package is a wrapper around the Mersenne Twister library, and unfortunately it imposes on its users the underlying library's typically horrible constraints borne of too much Fortran programming: you can only have one PRNG per application, and it can only be used from a single thread! The mwc-random package is less restrictive: fire up as many PRNGs in different threads as you like, and they'll all operate independently. You can also use the PRNGs in either the ST or the IO monad, for further convenience.

When generating normally distributed random variables, the mwc-random package uses an algorithm known as the "modified ziggurat". One of the more popular algorithms for generating normally distributed variables is called the ziggurat, but its popularity belies an ill-understood quality problem: the numbers it generates aren't independent enough! It turns out that they are noticeably correlated. The modified ziggurat is almost as fast, and it sacrifices a little speed in the name of improved independence.

The base-level performance of the random number generators looks like this on my Mac using 32-bit GHC 6.12.3, where the time quoted is to generate a single double-precision floating point number:

• Uniform: 142.6 nanoseconds

• Normal: 15149 nanoseconds

Where does the question of being "done" or not come in? Well, while poking around tonight, I was a little surprised at the large difference in speed between the uniform and normal PRNGs, so I investigated. The ziggurat algorithm gets its name from the precomputed lookup tables it uses to gain its speed. It turns out that GHC's inliner was being too aggressive with the table-related code, causing the ziggurat tables to be regenerated over and over instead of precomputed just once. Ouch!

One small and very quick change, and the performance of the PRNG for normally distributed variables changed dramatically:

• Before: 15149 nanoseconds

• After: 246.8 nanoseconds

That's a little over 61 times faster. Not bad for a couple of lines of changes!

As a final note, now that GHC can build 64-bit programs on a Mac, you might wonder how it performs. Here's a comparison between 32-bit and 64-bit versions of GHC 7.0.2 (times in nanoseconds):

Those are some pretty nice performance improvements! Of course, not all applications come in the form of nice tight numeric kernels, so don't take it as given that you'll see improvements like this in your code.