Ease of use

The new decode function complements the longstanding encode function, and makes the API simpler.

New examples make it easier to learn to use the package.

Generics support

aeson’s support for data-type generic programming makes it possible to use JSON encodings of most data types without writing any boilerplate instances.

Thanks to Bas Van Dijk, aeson now supports the two major schemes for doing datatype-generic programming:

• the modern mechanism, built into GHC itself

• the older mechanism, based on SYB (aka "scrap your boilerplate")

The modern GHC-based generic mechanism is fast and terse: in fact, its performance is generally comparable in performance to hand-written and TH-derived ToJSON and FromJSON instances. To see how to use GHC generics, refer to examples/Generic.hs.

The SYB-based generics support lives in Data.Aeson.Generic, and is provided mainly for users of GHC older than 7.2. SYB is far slower (by about 10x) than the more modern generic mechanism. To see how to use SYB generics, refer to examples/GenericSYB.hs.

Improved performance

• We switched the intermediate representation of JSON objects from Data.Map to Data.HashMap, which has improved type conversion performance.

• Instances of ToJSON and FromJSON for tuples are between 45% and 70% faster than in 0.3.

Evaluation control

This version of aeson makes explicit the decoupling between identifying an element of a JSON document and converting it to Haskell. See the Data.Aeson.Parser documentation for details.

The normal aeson decode function performs identification strictly, but defers conversion until needed. This can result in improved performance (e.g. if the results of some conversions are never needed), but at a cost in increased memory consumption.

The new decode' function performs identification and conversion immediately. This incurs an up-front cost in CPU cycles, but reduces reduce memory consumption.

You may have seen my exciting news about our upcoming move to Facebook.

It’s been a total blast working on our product, and of course as we did so we released a number of open source libraries and tools. It only added to our pleasure to see so much of that code used outside of our own domain. I will continue to develop and maintain the code that we have released.

Here is a quick rundown of the code we have released, roughly ordered by significance. Yep, we wrote all of these projects in Haskell, definitely a decision that in retrospect I’m very happy about.

• pronk (not yet actually released) is an application for load testing web servers. Think of it as similar to httperf or ab, only more modern, simpler to deal with, and with vastly better analytic and reporting capabilities.

• configurator is a library that allows fast, dynamic reconfiguration of a Haskell application or daemon.

• aeson is a JSON encoding and decoding library optimized for high performance and ease of use.

• text-format is a library for printf-like text formatting.

• mysql-simple is an easy-to-use client library for the MySQL database. It is several times faster than its competitors, and easier to use. It is built on top of the low-level mysql library.

• riak-haskell-client is a client for the Riak decentralized data store.

• blaze-textual is a library for efficiently rendering Haskell data as text.

• double-conversion is a very fast library for rendering double precision floating point numbers as text, based on the code from the V8 Javascript engine.

• resource-pool is a fast resource pooling library.

• snappy provides Haskell bindings to Google’s extremely fast snappy compression library.

• base16-bytestring provides fast handling of base16-encoded data.

• hdbc-mysql provides a MySQL transport for the HDBC database access library. (Yes, we recommend using mysql-simple instead!)

Thanks to all of you who have contributed patches and bug reports. It’s going to be an exciting future!

I'd particularly like to thank Alexey Khudyakov for his wonderful work on this release.

New features:

• Student-T, Fisher-Snedecor, F-distribution, and Cauchy-Lorentz distributions are added.

• Histogram computation is added, in Sample.Histogram.

• Forward and inverse discrete Fourier and cosine transforms are added, in Transform.

• Root finding is added, in Math.RootFinding.

Major changes:

• The Sample.KernelDensity module has been renamed, and completely rewritten to be much more robust. The older module oversmoothed multi-modal data. (The older module is still available under the name Sample.KernelDensity.Simple).

• The type classes Mean and Variance are split in two. This is required for distributions which do not have finite variance or mean.

Smaller changes:

• The complCumulative function is added to the Distribution class in order to accurately assess probalities P(X>x) which are used in one-tailed tests.

• A stdDev function is added to the Variance class for distributions.

• The constructor Distribution.normalDistr now takes standard deviation instead of variance as its parameter.

• A bug in Quantile.weightedAvg is fixed. It produced a wrong answer if a sample contained only one element.

• Bugs in quantile estimations for chi-square and gamma distribution are fixed.

• Integer overlow in mannWhitneyUCriticalValue is fixed. It produced incorrect critical values for moderately large samples. Something around 20 for 32-bit machines and 40 for 64-bit ones.

• A bug in mannWhitneyUSignificant is fixed. If either sample was larger than 20, it produced a completely incorrect answer.

• One- and two-tailed tests in Tests.NonParametric are selected with sum types instead of Bool.

• Test results returned as enumeration instead of Bool.

• Performance improvements for Mann-Whitney U and Wilcoxon tests.

• Module Tests.NonParamtric is split into Tests.MannWhitneyU and Tests.WilcoxonT

• sortBy is added to Function.

• Mean and variance for gamma distribution are fixed.

• Much faster cumulative probablity functions for Poisson and hypergeometric distributions.

• Better density functions for gamma and Poisson distributions.

• The function Function.create is removed. Use generateM from the vector package instead.

• A function to perform approximate comparion of doubles is added to Function.Comparison.

• Regularized incomplete beta function and its inverse are added to Function.

Strange Loop is a pretty good approximation to my ideal conference, covering a narrow family of topics I’m interested in, mainly leading-edge matters in programming languages and distributed systems. The focus is not at all academic, instead being on open source software that you can download and get to work with. I have to strongly commend the organizers for finding a great venue, excellent speakers, and running a fabulous show at an impressively low price.

I’ve published the source for the slides from both my workshop and talk in a github repository, but you might prefer to look at the slides directly:

• My Haskell workshop (141 slides, and we managed to cover the whole lot at a pleasant pace!)

• My talk on startups and Haskell (41 slides)

I buried a teaser slide 3/4 of the way through the startup talk, which turned out to be an excellent way to accidentally find out how many people on Twitter were reading through the slides (the answer surprised me: a lot!). The enigmatic bug in question was quite spectacular: a development version of GHC would delete a source file if it contained a type error. Ouch!

One of the people in the capacity crowd at the Haskell workshop was none other than Gerry Sussman, who was finally learning Haskell for the first time (doubtless after several decades of overexposure to parentheses). He pronounced the workshop "pretty good", which I believe is MIT-speak for "OMG teh awesome!"

At one point, Gerry had a pretty amusing epigram to offer. "Haskell is the best of the obsolete programming languages!" he pronounced, with a mischievous look. Now, I know when I’m being trolled, so I said nothing and waited a moment, whereupon he continued, "but don’t take it the wrong way – I think they’re all obsolete!"

For instance, in this case, I could immediately tell by inspection that something bad was afoot in the inner loop of the UTF-8 decoder. A decoder more or less has to read a byte of input at a time, as this heavily edited bit of Core illustrates:

let x2 :: Word8
    x2 = case readWord8OffAddr# {- ... -} of
           (# s, x #) -> W8# x
in {- ... -}

What's important in the snippet above is that the value x2 is boxed, i.e. packaged up with a W8# constructor so that it must be accessed via a pointer indirection. Since a decoder must read up to 4 bytes to emit a single Unicode code point, the loop was potentially boxing up 4 bytes, then immediately unboxing them:

case x2 of 
  W8# x2# ->
    case x3 of
      W8# x3# ->
        case x4 of
          W8# x4# -> {- ... -}

While both boxing and unboxing are cheap in Haskell, they're certainly not free, and we surely don't want to be doing either in the inner loop of an oft-used function.

We can see why this was happening at line 96 of the decodeUtf8With function. I'd hoped that the compiler would be smart enough to unbox the values x1 through x4, but it turned out not to be.

Fixing this excessive boxing and unboxing wasn't hard at all, but it made the code uglier. The rewritten code had identical performance on pure ASCII data, but was about 1.7 times faster on data that was partly or entirely non-ASCII. Nice! Right?

Not quite content with this improvement, I tried writing a decoder based on Björn Höhrmann's work. My initial attempt looked promising; it was up to 2.5 times faster than my first improved Haskell decoder, but it fell behind on decoding ASCII, due to the extra overhead of maintaining the DFA state.

In English-speaking countries, ASCII is still the king of encodings. Even in non-English-speaking countries that use UTF-8, a whole lot of text is at least partly ASCII in nature. For instance, other European languages contain frequent extents of 7-bit-clean text. Even in languages where almost all code points need two or more bytes to be represented in UTF-8, data such as XML and HTML still contains numerous extents of ASCII text.

What would happen if we were to special-case ASCII? If we read a 32-bit chunk of data, mask it against 0x80808080, and get zero, we know that all four bytes must be ASCII, so we can just write them straight out without going through the DFA (see lines 82 through 110).

As the numbers suggest, this makes a big difference to performance! Decoding pure ASCII becomes much faster, while both HTML and XML see respectable improvements. Of course, even this approach comes with a tradeoff: we lose a little performance when decoding entirely non-ASCII text.

Even in the slowest case, we can now decode upwards of 250MB of UTF-8 text per second, while for ASCII, we exceed 1.7GB per second!

These changes have made a big difference to decoding performance across the board: it is now always between 2 and 4 times faster than before.

As a final note, I haven't released the new code quite yet - so keep an eye out!

Prior to Steele and White’s "How to print floating-point numbers accurately", implementations of printf and similar rendering functions did their best to render floating point numbers, but there was wide variation in how well they behaved. A number such as 1.3 might be rendered as 1.29999999, for instance, or if a number was put through a feedback loop of being written out and its written representation read back, each successive result could drift further and further away from the original.

Steele and White effectively solved the problem with a clever algorithm named "Dragon4" (the fourth version of the "Dragon" algorithm, which acquired its name because the authors were inspired to obscure puns by Heighway's dragon curve).

The Dragon4 algorithm spread quickly across language runtimes, such that few programmers today understand that this was ever a problem, much less how hairy it was (and is). Indeed, prior to last year, there was almost no activity in this area: two papers proposed widely used refinements to Dragon4, and that was about it. (Alas, the problem was originally solved around a decade before Steele and White published their work, but nobody noticed. If you have a clever idea and sufficient chutzpah, try to enlist Guy Steele as a coauthor. Your work will be read.)

But how solved was the problem? Dragon4 and its derivatives are complicated and tricky, and they have a hefty performance cost, since they rely on arbitrary-precision integer arithmetic to compute their results. There might be a significant performance improvement to be gained if someone could figure out how to use native machine integers instead.

In 2010, Florian Loitsch published a wonderful paper in PLDI, "Printing floating-point numbers quickly and accurately with integers", which represents the biggest step in this field in 20 years: he mostly figured out how to use machine integers to perform accurate rendering! Why do I say "mostly"? Because although Loitsch's "Grisu3" algorithm is very fast, it gives up on about 0.5% of numbers, in which case you have to fall back to Dragon4 or a derivative.

If you're a language runtime author, the Grisu algorithms are a big deal: Grisu3 is about 5 times faster than the algorithm used by printf in GNU libc, for instance. A few language implementors have already taken note: Google hired Loitsch, and the Grisu family acts as the default rendering algorithms in both the V8 and Mozilla Javascript engines (replacing David Gay's 17-year-old dtoa code). Loitsch has kindly released implementations of his Grisu algorithms as a library named double-conversion.

And of course I can't talk about performance without mentioning Haskell somewhere :-) I've taken Loitsch's library and written a Haskell interface, which I've measured to be 30 times faster than the default renderer used in the Haskell runtime libraries. This has some nice knock-on effects: my aeson JSON library is now 10 times faster at rendering big arrays of floating point numbers, for instance. I accidentally noticed in the course of that work that my Haskell text Unicode library's UTF-8 encoder wasn't as fast as it could be, so I improved its performance by about 50% along the way. Hooray for faster code!

(By the way, the punnery in algorithm naming continues: the Grisu algorithms are named for Grisù, the little dragon.)

And the performance news is good: on real-world data, I’ve improved encoding performance by about a factor of 4. Why don’t we let the graphs do the talking.

Enjoy!

About a week ago, I received a message from Finlay Thompson asking about my Haskell statistics package: he wanted to know how to generate pseudo-random variables using it. I redirected him from that to my mwc-random package, where my pseudo-random number generation code lives.

The mwc-random package currently provides generators for two widely used distributions: uniform and normal. When I was originally writing it, I paid particular attention to making it high quality, fast, and easy to use.

"High quality" sounds a little nebulous, but in the world of pseudo-random number generation, it's actually pretty well defined: a good PRNG should have a large period (the number of samples you need to pull out of it before it repeats itself, assuming a good seed), and the numbers it generates should withstand stringent tests of apparent independence (simply put, given one datum, you shouldn't be able to predict the next).

One algorithm that satisfies these criteria of quality is George Marsaglia's multiply-with-carry algorithm MWC256 (also known as MWC8222). It has a period of about 2⁸²²² (huge enough for all conceivable practical purposes), and stands up well to the "testu01", "diehard" and "big crush" statistical tests.

Due to its simplicity, MWC256 is also very fast, and under appropriate circumstances (e.g. on a 64-bit machine) it can be even faster than the well known Mersenne Twister algorithm (which also fails some statistical tests that MWC256 passes).

The Mersenne Twister is itself available for Haskellers to use, in the form of the mersenne-random package. This package is a wrapper around the Mersenne Twister library, and unfortunately it imposes on its users the underlying library's typically horrible constraints borne of too much Fortran programming: you can only have one PRNG per application, and it can only be used from a single thread! The mwc-random package is less restrictive: fire up as many PRNGs in different threads as you like, and they'll all operate independently. You can also use the PRNGs in either the ST or the IO monad, for further convenience.

When generating normally distributed random variables, the mwc-random package uses an algorithm known as the "modified ziggurat". One of the more popular algorithms for generating normally distributed variables is called the ziggurat, but its popularity belies an ill-understood quality problem: the numbers it generates aren't independent enough! It turns out that they are noticeably correlated. The modified ziggurat is almost as fast, and it sacrifices a little speed in the name of improved independence.

The base-level performance of the random number generators looks like this on my Mac using 32-bit GHC 6.12.3, where the time quoted is to generate a single double-precision floating point number:

• Uniform: 142.6 nanoseconds

• Normal: 15149 nanoseconds

Where does the question of being "done" or not come in? Well, while poking around tonight, I was a little surprised at the large difference in speed between the uniform and normal PRNGs, so I investigated. The ziggurat algorithm gets its name from the precomputed lookup tables it uses to gain its speed. It turns out that GHC's inliner was being too aggressive with the table-related code, causing the ziggurat tables to be regenerated over and over instead of precomputed just once. Ouch!

One small and very quick change, and the performance of the PRNG for normally distributed variables changed dramatically:

• Before: 15149 nanoseconds

• After: 246.8 nanoseconds

That's a little over 61 times faster. Not bad for a couple of lines of changes!

As a final note, now that GHC can build 64-bit programs on a Mac, you might wonder how it performs. Here's a comparison between 32-bit and 64-bit versions of GHC 7.0.2 (times in nanoseconds):

Those are some pretty nice performance improvements! Of course, not all applications come in the form of nice tight numeric kernels, so don't take it as given that you'll see improvements like this in your code.

The major advance in the new version of aeson is a considerable speed improvement.

The datasets I’m using are Twitter search results, from the Twitter JSON search API. For mostly-English results, 0.2.0.0 is up to 30% faster than before, while on Japanese data (which makes heavy use of Unicode escapes), I’ve bumped performance by more than 50%.

To see how well aeson performs compared to JSON parsers for other languages, I compared it against the json module in Python 2.7. That module’s JSON parser is written in C, so it’s very fast indeed, and the amount of actual Python being executed in my microbenchmark is tiny. How do we fare?

On mostly-English data, aeson is actually faster than Python’s native-code json parser. Nice! And on Japanese data, we’re a little slower, but still very competitive.

What if you’ve been using the Haskell json package, which was the first open source Haskell JSON parser to be published? Well, I do think that aeson is easier to use, but it’s also 3x faster than the json package:

The new version of aeson introduces some other useful improvements.

• There’s a new Generic module, which lets you convert almost any instance of the Data typeclass to and from JSON without writing boilerplate code. (Be warned: generics are slow. If performance is important to you, write that boilerplate!)
• We introduce a Number type that represents integers to full accuracy, and which handles floating point numbers efficiently.
• Instead of parsing via the Applicative typeclass, we now use a custom parsing monad, improving both ease of use and performance.

Since I’ve spent the past two decades in the world of open source, of course I have goodies to show off:

• A fast, powerful library for working with the Riak decentralized data store (mailrank/riak-haskell-client on github). I think it’s the best Riak client library available for any programming language
• An efficient, easy to use JSON library for Haskell (mailrank/aeson on github). Twice as fast as the competition.