Big data[1][2] is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage,[3] search, sharing, transfer, analysis,[4] and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to “spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.”

Now what if we get tired of the current hype cycle?

Big fucking deal[1][2] is a collection of deals so fucking large and complex that it becomes difficult to process using on-hand fuck giving tools or traditional shit giving techniques. The challenges include capture, curation, storage,[3] search, sharing, transfer, analysis,[4] and all kinds of who the fuck knows what else. The trend to larger fucking deals is due to the additional shit derivable from giving a fuck about a single large fucking pile of related shit, as compared to separate smaller piles with the same total amount of bullshit, allowing correlations to be found to “spot business shit, determine quality of whatever, prevent some nasty shit, link legal shit right the fuck together, combat fucking crime no I am not making this up it’s like fucking Batman, and determine real-time traffic shittiness.”

A couple of days ago, Alexey Khudyakov did a little digging into the accuracy of criterion’s measurements. I thought his results were interesting enough to be worth some deeper analysis.

First, let’s briefly discuss Alexey’s method and findings. He created 1,000 identical copies of a benchmark, and looked to see if the measurements changed over time. They did, slowly increasing in a linear fashion. (This is a phenomenon known to statisticians as serial correlation, where a measurement influences a future measurement.)

If every benchmark is the same, why do the measurements increase? Criterion does its book-keeping in memory. For every run, it saves a piece of data in memory. Not until all benchmarks have finished does it write out that data to a Javascript+HTML report or CSV file.

I thought that the slow increase in measurements was probably related to this book-keeping, but how to test this hypothesis?

I created 200 identical copies of the same benchmark (I’m not patient enough to wait for 1,000!) and dumped a heap profile while it ran, then I plotted the numbers measured by criterion against heap size.

For this particular benchmark, criterion spends about 4% of its time in the garbage collector. The size of the heap increases by 300% as the program runs. If we expect garbage collection overhead to affect measurements, then the time we measure should increase by 12% as we repeat the benchmark over and over, slowly accumulating data.

This prediction exactly matches what we actually see: we start off measuring exp at 25.5 nanoseconds, and by the end we see it taking 28.5 nanoseconds.

The obvious next question is: how realistic a concern is this? A normal criterion program consists of a handful of benchmarks, usually all very different in what they do. I have not seen any cases of more than a few dozen benchmarks in a single suite. If only a few benchmarks get run, then there is essentially no opportunity for this inflation effect to become noticeable.

Nevertheless, I could definitely make some improvements (or even better, someone could contribute patches and I could continue to do nothing).

• It would probably help to write data to a file after running each benchmark, and then to load that data back again before writing out the final report. [Edit: I wrote a patch that does just this; the increase in memory use vanishes, and along with it, the gradual inflation in measured times. Bingo!]

• There is no benefit to looking for serial correlation across different benchmark runs, because nobody (except Alexey!) makes identical duplicates of a benchmark.

• For the series of measurements collected for a single benchmark, it would probably be helpful to add an autocorrelation test, if only to have another opportunity to raise a red flag. Criterion is already careful to cry foul if its numbers look too messy, but first-order serial correlation would be likely to slip past the sophisticated tests it uses (like the bootstrap). I’ve long wanted to add a Durbin-Watson test, but I’ve been lazy for even longer.

If you were to run every benchmark in a large suite one after the other in a single pass, then your final numbers could indeed be inflated by a few percent [edit: at least until I release the patch]. However, there are many other ways to confound your measurements, most of which will be far larger than this book-keeping effect.

• If you simply change the order in which you run your benchmarks, this can dramatically affect the numbers you’ll see.

• The size of the heap that the GHC runtime uses makes a big difference, as do the threaded runtime, number of OS threads, and use of the parallel garbage collector. Any of these can change performance by a factor of two or more (!).

• You should close busy tabs in a web browser (or preferably quit it entirely), kill your mail client and antivirus software, and try to eliminate other sources of system noise. You’ll be surprised by how big a difference these can make; anywhere from a handful to a few hundred percent.

If you want high-quality numbers, it is best to run just one benchmark from a suite at a time; on the quietest system you can manage; to watch for criterion's warnings about outliers affecting results; and to always compare several runs to see if your measurements are stable.

[Edit: Here is a chart of the measurements with the bug fixed, complete with a linear fit to indicate that the numbers are basically flat. Hooray!] ]]> http://www.serpentine.com/blog/2013/04/13/whats-in-a-number-criterion-edition/feed/ 1 http://www.serpentine.com/blog/2013/03/20/whats-good-for-c-is-good-for-haskell/ http://www.serpentine.com/blog/2013/03/20/whats-good-for-c-is-good-for-haskell/#comments Wed, 20 Mar 2013 19:29:32 +0000 Bryan O'Sullivan http://www.serpentine.com/blog/?p=970 …

positive :: (Integral a) => a -> Builder
positive = go
  where
    go n | n < 10    = digit n
         | otherwise = go (n `quot` 10) <> digit (n `rem` 10)
    digit n = singleton . intTodigit . fromIntegral

countDigits :: (Integral a) => a -> Int
countDigits v0 = go 1 (fromIntegral v0 :: Word64)
  where go !k v
           | v < 10    = k
           | v < 100   = k + 1
           | v < 1000  = k + 2
           | v < 10000 = k + 3
           | otherwise = go (k+4) (v `quot` 10000)

positive :: (Integral a) => a -> Builder
positive i
    -- we win when we special-case single-digit numbers
    | i < 10    = writeN 1 $ \marr off ->
                  unsafeWrite marr off (i2w i)
    | otherwise = let !n = countDigits i
                  in writeN n $ \marr off ->
                     posDecimal marr off n i

posDecimal :: (Integral a) =>
              forall s. MArray s -> Int -> Int -> a -> ST s ()
posDecimal marr off0 ds v0 = go (off0 + ds - 1) v0
  where go off v
          | v < 10 = unsafeWrite marr off (i2w v)
          | otherwise = do
              unsafeWrite marr off (i2w (v `rem` 10))
              go (off-1) (v `div` 10)

i2w :: (Integral a) => a -> Word16
i2w v = 48 + fromIntegral v

countDigits :: (Integral a) => a -> Int
countDigits v0 = go 1 (fromIntegral v0 :: Word64)
  where
    go !k v
      | v < 10    = k
      | v < 100   = k + 1
      | v < 1000  = k + 2
      | v < 1000000000000 =
          k + if v < 100000000
              then if v < 1000000
                   then if v < 10000
                        then 3
                        else 4 + fin v 100000
                   else 6 + fin v 10000000
              else if v < 10000000000
                   then 8 + fin v 1000000000
                   else 10 + fin v 100000000000
      | otherwise = go (k + 12) (v `quot` 1000000000000)
   fin v n = if v >= n then 1 else 0

posDecimal :: (Integral a) =>
              forall s. MArray s -> Int -> Int -> a -> ST s ()
posDecimal marr off0 ds v0 = go (off0 + ds - 1) v0
  where go off v
           | v >= 100 = do
               let (q, r) = v `quotRem` 100
               write2 off r
               go (off - 2) q
           | v < 10    = unsafeWrite marr off (i2w v)
           | otherwise = write2 off v
        write2 off i0 = do
          let i = fromIntegral i0; j = i + i
          unsafeWrite marr off $ get (j + 1)
          unsafeWrite marr (off - 1) $ get j
        get = fromIntegral . B.unsafeIndex digits

The 1.2 release of hashable is not backwards compatible, and for several good reasons. Read on for the details.

Hash flooding

The threat landscape faced by applications that use hash-based data structures has shifted radically this year, with hash flooding emerging as an attack vector after a long period of obscurity (the earliest publication I know of on the subject dates back to 1998, but I doubt that it was new even then).

We know that both networked and local applications that use weak hash algorithms are vulnerable, with new attacks being published all the time.

Recently, even modern hash algorithms such as the MurmurHash family and CityHash have succumbed to surprising attacks via differential cryptanalysis, in which large numbers of collisions can be generated efficiently without knowledge of the salt.

SipHash

The SipHash algorithm offers a strong defence against differential cryptanalysis. SipHash is now the algorithm used for hashing all string-like types.

We automatically choose optimized implementations for 64-bit and 32-bit platforms, depending on their capabilities (e.g. availability of SSE2 on 32-bit systems).

For inputs of just a few bytes in length, SipHash is about half the speed of the previous string hash algorithm, FNV-1. Its performance breaks even with FNV-1 at about 50 bytes, and it improves to become more than twice as fast for large inputs.

(FNV-1 is also particularly easy to attack, so escaping from it was imperative, even if that cost a little performance.)

Random choice of salt

To provide an additional measure of security against hash flooding attacks, the standard hash function chooses a random salt at program startup time, using the system’s cryptographic pseudo-random number generator (/dev/urandom on Unix, CryptGenRandom on Windows).

Effortless, fast hashing

I have also demoted hash out of the Hashable typeclass, so the API now looks like this.

class Hashable a where
    hashWithSalt :: Int -- salt
                 -> a   -- value to hash
                 -> Int -- result

hash :: Hashable a => a -> Int
hash = hashWithSalt defaultSalt

In other words, if you are writing an instance of Hashable by hand, you had better pay attention to the salt, and if you use the friendly hash function, you get the randomly-chosen default salt every time.

Now that writing Hashable instances by hand is slightly more painful, I have also made it almost effortless to hash your custom datatypes.

Using GHC’s no-longer-so-new generics machinery, you can have an efficient Hashable instance generated “for free” with just a few lines of code. Good code is generated for both product and sum types.

{-# LANGUAGE DeriveGeneric #-}

import GHC.Generics

data MyType = MyStr String
            | MyInt Integer
    deriving (Generic)

instance Hashable MyType

If your types are polymorphic, no problem; they’re just as easy to generate hash functions for.

data Poly a = Poly a
    deriving (Generic)

instance Hashable a => Hashable (Poly a)

Improved avalanche for basic types

We do not currently use SipHash to hash basic types such as numbers, as strings are by a huge margin the most common vector for hash flooding attacks.

Nevertheless, we’ve made improvements to our hashing of basic types. Previous versions of hashable did nothing at all, meaning that the hash of a number was simply the identity function. While (obviously) fast and good for locality of reference in a few narrow cases, the identity function makes a terrible hash for many data structures.

For instance, if a hash table uses open addressing with linear probing, the identity hash can cause quadratic performance when inserting values into the table that are identical modulo the table size.

It’s generally considered desirable for a hash function to have strong “avalanche” properties, meaning that a single-bit change in an input should cause every bit of the output to have a 50% probability of changing.

For numberic types, we now use a couple of algorithms developed by Thomas Wang that are both fast and have good avalanche properties. These functions are more expensive than doing nothing, but quite a lot faster than SipHash. If it turns out that integers become a popular vector for hash flooding, we may well switch to SipHash for everything at some point.

In summary

This release of the hashable library is a fairly big deal. While the performance implications aren’t entirely happy, I’ve done my best to write the fastest possible code.

I hope you’ll agree that the improvements in ease of use, security, and applicability (thanks to no longer using identity hashes) are more than worth the cost. Happy hacking!

Once we get that submission accepted, we can fold Johan’s excellent hash-based data structures from his unordered-containers package into the standard containers package. For many applications, the hash array mapped tries in unordered-containers offer a decent performance advantage over the current standard of ordered search trees.

Prior to making the submission, I found myself wanting to spring clean the hashable package. Although it is simple and well put together, it has some robustness problems that are important in practice.

Chief among these weaknesses is that it uses the FNV-1 hash, which is well known to be vulnerable to collisions. Since it currently uses FNV-1 with a fixed key, Internet-facing web applications that use hash trees are at risk.

The current state of the art in reasonably fast, secure hash algorithms is Aumasson and Bernstein’s SipHash. There already existed a Haskell implementation in the form of the siphash package, but I thought it was worth implementing a version of my own.

My criteria for developing a new Haskell SipHash implementation were that it had to be fast and easily reusable. I have a version in progress right now, and so far it is turning out well.

The main features of this implementation (specifically when hashing ByteStrings) are as follows:

• All arithmetic is unboxed. I carefully inspected the generated Core at every step to ensure this would be the case. (It's usually easier to improve a code base that starts out with good performance properties than to retrofit good performance into existing code.)

• The main loop performs 8-byte reads when possible, and if the final block is less than 8 bytes, unrolls that last bytewise fill to be as efficient as possible.

• The common cases of SipHash's c and d parameters being fixed at 2 and 4 are unrolled.

• The implementation is written in continuation passing style. This had the happy consequence of making unrolling particularly easy (read the source to see).

It’s worth looking at what “unrolling” means in this context. The SipHash algorithm is organized around a loop that performs a “round” a small number times, where a round is a step of an ARX (add-rotate-xor) cipher.

sipRound :: (Sip -> r) -> Sip -> r

Each 8-byte block of input is put through two rounds. After processing is complete, the state is put through four rounds.

Using a CPS representation, we can easily special-case these values so that each can be executed with just a single conditional branch.

runRounds :: Int -> (Sip -> r) -> Sip -> r
runRounds 2 k = sipRound (sipRound k)
runRounds 4 k = sipRound (sipRound (sipRound (sipRound k)))

GHC 7.6 generates excellent Core for the above code, inlining sipRound and the continuation k everywhere, and avoiding both jumps and uses of boxed values.

I measured the performance of my in-progress SipHash implementation against the FNV-1 hash currently used by the hashable package, and against two versions of Vincent Hanquez’s siphash package. Different input sizes are across the x axis; the y axis represents speedup relative to FNV-1 (less than 1 is slower, greater is faster).

My SipHash implementation starts off at a little over half the speed of FNV-1 on 5-byte inputs; breaks even at around 45 bytes; and reaches twice the speed at 512 bytes. For comparison, I included a C implementation; my Haskell code is just 10% slower. I’m pretty pleased by these numbers.

(After I published an initial benchmark, Vincent Hanquez sped up his siphash package by a large amount in just a couple of hours. The dramatic effect is shown above in the huge jump in performance between versions 1.0.1 and 1.0.2 of his package. This demonstrates the value of even a brief focus on performance!)

Incidentally, SipHash is another case where compiling with -fllvm really helps; compared to GHC’s normal code generator, I saw speed jump by a very pleasing 35%. (Sure enough, the numbers I show above are with -fllvm.)

With some extra work and a little time to focus, this new code should be ready to incorporate into the hashable package, hopefully within a few days.

A few months ago, reports began to filter in of an unhappy problem with the Haskell text package: it was causing huge object files to be generated when a file contained lots of string literals.

I didn’t notice the initial report (it was posted to a busy mailing list that I don’t try to keep up with), but Michael Snoyman was kind enough to take that message and file a bug.

The culprit was this very simple function definition, which converts a string from the venerable Haskell String type to the more modern Text.

pack :: String -> Text
pack txt = unstream
             (Stream.map safe
           (Stream.streamList txt))

The definition of pack is too innocent to be at fault; the problem lies with the extra directives that text gives to the compiler.

{-# INLINE pack #-}
{-# INLINE Stream.unstream #-}
{-# INLINE Stream.map #-}
{-# INLINE Stream.streamList #-}

By asking the compiler to inline every function, we guaranteed that every string literal would result in a lot of code being generated. Worse, all of this space would be entirely redundant, consisting of repeated copies of exactly the same code.

(You might well wonder why we’d insist on inlining any of these functions, if the cost in space is so high. The answer is that inlining is key to why the text package achieves good performance. That deserves an article of its own, so I’ll return to the subject soon.)

Have you ever wondered how GHC represents string literals? Instead of somehow statically constructing a linked list of characters and emitting that into an object file, it’s smarter.

For strings of pure ASCII, GHC generates a packed zero-terminated byte sequence that looks like this.

.const
.align 3
.align 0
_co0_str:
        .byte   102    # f
        .byte   111    # o
        .byte   111    # o
        .byte   0

(For strings that contain Unicode or control characters, GHC still generates a packed sequence of bytes, but this time they’re specially encoded.)

In Haskell, these byte sequences have a type that is simply a fixed address, Addr#. During compilation, GHC takes a string literal and prefixes it with a function to convert from Addr# to String.

-- What we write:
foo :: String
foo = "foo"

-- What GHC generates:
foo :: String
foo = GHC.CString.unpackCString# co0_str
  where co0_str :: Addr#
        co0_str = "foo"

One of the lovelier features of GHC is that it exposes some of its internal machinery to authors. We’re paying a price for our aggressive use of its INLINE directive; is there another GHC feature we can use to save the day?

Enter the rewrite rule, a way of telling GHC how to perform source-to-source transformations.

Here is a naive attempt to specify a rewrite rule that might help us. First, we define a version of pack that we tell the compiler to never inline, then we supply a rewrite rule that tells the compiler to substitute the never-inlined version of pack for the normal version.

packNOINLINE :: String -> Text
packNOINLINE = unstream . Stream.map safe . Stream.streamList
{-# NOINLINE packNOINLINE #-}

{-# RULES "TEXT literal" forall a.
    pack s = packNOINLINE s
  #-}

Although this rule works and generates correct code, it swaps one problem for another: the object files we generate shrink dramatically, but we’ve defeated some of the compiler’s opportunities to improve the code it emits.

Oh, and during compilation, remember that after GHC has finished processing a string literal, we start out with an Addr#, then GHC converts to a String for us, and finally we convert to a Text. That intermediate step galls me, even though it really has no practical consequences.

Happily for us, GHC’s rewrite rules are applied cleverly: rather than being a simple one-shot affair, GHC keeps trying to apply rewrite rules as it optimises a program.

The critical addition to our rule is to recognise that when we write a string literal, it will be transformed into an application of GHC.String.unpackCString#, and target our rule to an expression containing this.

-- Introduce a new function ...
Text.unpackCString# :: Addr# -> Text
Text.unpackCString# addr# 
  = unstream (Stream.streamCString# addr#)
{-# NOINLINE unpackCString# #-}

-- ... and use it!
{-# RULES "TEXT literal" forall a.
    unstream (Stream.map safe
      (Stream.streamList
        (GHC.String.unpackCString# a)))
      = Text.unpackCString# a #-}

With this rewrite rule, GHC will transform code that we /never actually wrote/, using a type (Addr#) that we don’t use in our code. The conditions that trigger this rule will arise only when we define a literal Text value. This means that productive uses of stream fusion will not be affected. Even better, this rule eliminates that pesky intermediate String value, since the new unpackCString# performs a direct translation. Not a bad trick!

As I waited for the sun to go down so that I might venture out in the heat without being cooked by both air and glare, I decided to return my attention to the Haskell text library for the first time in a while.

This library is in a happy state of feature and bug stability, and it has grown in popularity to the point where it is now one of the ten most used libraries on Hackage (counting the number of other packages depending on it).

Very pleasing, right? But there are always improvements to be made, and I came across a couple of nice candidates after a little inspection.

I’m going to talk about one of these in some detail, because it’s worth digging into a worked example of how to further improve the performance of code that is already fast, and this is nice and brief.

The easiest of these candidates to explain is the function for decoding a bytestring containing UTF-8 text. Generally when we’re decoding UTF-8, the strings we use are relatively large, and so the small memory savings below will not usually be significant.

The optimisation I discuss below appears in other performance sensitive contexts within the library in which small strings are common, so this is definitely a meaningful optimisation. For instance, the code for converting between the internal Stream type and a Text admits the same optimisation.

Somewhat simplified, the beginning of our UTF-8 decoding function reads as follows:

decodeUtf8With :: OnDecodeError -> ByteString -> Text

decodeUtf8With onErr bs = textP ary 0 alen
 where
  (ary,alen) = A.run2 (A.new (length bs) >>= go)
  go dest = {- the actual decoding loop -}

The A.run2 function referred to above is important to understand. It runs an action that returns a mutable array, and it “freezes” that array into an immutable array.

run2 :: (forall s. ST s (MArray s, a)) -> (Array, a)

run2 k = runST (do
                 (marr,b) <- k
                 arr <- unsafeFreeze marr
                 return (arr,b))

Our go function is given an initial mutable array allocated by A.new, fills it with text, and returns the final mutable array (which may have been reallocated) and its size.

I was curious about the performance of this code for very short strings, based on visual inspection alone.

• The run2 function forces a tuple to be allocated by its callee, and both of the values in that tuple must be boxed.

• It then allocates another tuple, containing one new boxed value.

• That tuple is immediately deconstructed, and its components are passed to textP. This is a simple function that ensures that we always use the same value for zero-length strings, to save on allocation.

• Because the caller of decodeUtf8With may not demand a result from textP, it is possible that the result of this function could be an unevaluated thunk.

For very short inputs, the overhead of allocating these tuples and boxing their parameters (required because tuples are polymorphic, and polymorphic values must be boxed) worried me as potentially a significant fraction of the "real work". (For longer strings, the actual decoding will dominate, so this becomes less of an issue. But short strings are common, and should be fast.)

But first, can we be sure that those tuples really exist once the compiler has worked its magic? GHC has for a long time performed a clever optimisation called constructed product result analysis, or CPR. This tries to ensure that if a function returns multiple results (e.g. in a tuple or some other product type), it can (under some circumstances) use machine registers, avoiding boxing and memory allocation. Unfortunately, I checked the generated Core intermediate code, and CPR does not kick in for us here. It’s often fragile, and a case like this, where we carry a tuple from the ST monad into pure code, can easily defeat it. (It’s far from obvious when it will or will not work, so always best to check if it matters.)

Remaining optimistic, we can manually get a little bit of a CPR-like effect by judicious choice of product types. Recall the type of run2:

run2 :: (forall s. ST s (MArray s, a)) -> (Array, a)

There are two profitable observations we can make here.

To begin with, we know that the a parameter of the first tuple will be an Int in some particularly important cases. We use a specialised tuple to represent this idea.

-- Instead of (MArray s, a):
data Run s = Run (MArray s) {-# UNPACK #-} !Int

We have explicitly instructed GHC to not box the Int; instead it will be stored unboxed, directly in the Run structure. This eliminates one memory allocation and the performance-sapping pointer indirection it would induce. (You might wonder why we don’t direct GHC to unbox the MArray s parameter. This value is just a pointer to an array, and we certainly don’t want to copy arrays around.)

Our next observation is that any time we want to return a Run, we will always use the result to construct a Text. So why not construct the Text directly, and save our callers from doing it in many places?

runText :: (forall s. ST s (Run s)) -> Text

Here’s what the body of runText looks like. We are no longer allocating a second tuple, and the len variable will never be boxed and then immediately unboxed. This brings us from allocating two tuples and an Int down to allocating just one Run.

runText act = runST $ do
  Run marr len <- act
  arr <- A.unsafeFreeze marr
  return (textP arr 0 len)

Experienced eyes will spot one last catch: because we are not forcing the result of the textP expression to be evaluated before we return, we are unnecessarily allocating a thunk here.

Nevertheless, even with that small inefficiency, both time and memory usage did improve: we allocated a little less, so we did a little less work, and thus a microbenchmark that decoded millions of short words became a few percent faster.

As gratifying as this is, can we improve on it further? Apart from the silly oversight of not forcing textP, there’s still that bothersome allocation of a Run. Reading the Core code generated by GHC indicates that CPR is still not happening, and so a Run really is being allocated and immediately thrown away. (I think that the polymorphic first parameter to Run might be defeating CPR, but I am not sure of this.)

All along, we’ve been either allocating tuples or Runs and then immediately throwing them away. Allocation might be cheap, but ceteris paribus, not allocating anything will always be cheaper.

Instead of returning a Run value, what if we were to have our action call a continuation to construct the final Text value?

runText ::
  (forall s.
   (MArray s -> Int -> ST s Text)
   -> ST s Text)
  -> Text

The function that we call as our continuation captures no variables from its environment, so it has no hidden state for which we might need to allocate memory to save.

runText act = runST (act $
  \ !marr !len -> do
    arr <- A.unsafeFreeze marr
    return $! textP arr 0 len)

We’ve now avoided the allocation of a Run, and just as importantly, we remembered to have the new runText force the result of the textP expression, so it will not allocate a thunk.

The only downside to this approach is that GHC does very little inlining of continuation-based code, so our use of a continuation leaves a jump in the code path that we’d have preferred to see the inliner eliminate.

This change in tack causes a significant reduction in memory allocation: on my small-decode microbenchmark, we allocate 17% less memory, and run 10% faster. I see the same improvement with another microbenchmark that exercises the Stream to Text conversion code, where I made the same optimisation. Given that I changed just a few lines of code in each case, this result makes me happy. If you’re interested, it might help to take a look at the final continuation-using version of decodeUtf8With in context.

Although simple, I hope that working through this in some detail has been interesting. Please let me know if you’d like to see more hands-on posts like this.

Statistical profiling (also known as sampling profiling) is simple and sweet: the profiler periodically wakes up and samples the stack, then when all is done, it prints a simple report of which lines showed up most often in the profile.

Why would this matter, though? Python already has two built-in profilers: lsprof and the long-deprecated hotshot. The trouble with lsprof is that it only tracks function calls. If you have a few hot loops within a function, lsprof is nearly worthless for figuring out which ones are actually important.

A few days ago, I found myself in exactly the situation in which lsprof fails: it was telling me that I had a hot function, but the function was unfamiliar to me, and long enough that it wasn’t immediately obvious where the problem was.

After a bit of begging on Twitter and Google+, someone pointed me at statprof. But there was a problem: although it was doing statistical sampling (yay!), it was only tracking the first line of a function when sampling (wtf!?). So I fixed that, spiffed up the documentation, and now it’s both usable and not misleading. Here’s an example of its output, locating the offending line in that hot function more accurately:

  %   cumulative      self          
 time    seconds   seconds  name    
 68.75      0.14      0.14  scmutil.py:546:revrange
  6.25      0.01      0.01  cmdutil.py:1006:walkchangerevs
  6.25      0.01      0.01  revlog.py:241:__init__
  [...blah blah blah...]
  0.00      0.01      0.00  util.py:237:__get__
---
Sample count: 16
Total time: 0.200000 seconds

I have uploaded statprof to the Python package index, so it’s almost trivial to install: “easy_install statprof” and you’re up and running.

Since the code is up on github, please feel welcome to contribute bug reports and improvements. Enjoy!

I doubt that these difficulties are unique to me, or even related to the fact that I’ve got a new baby (so I have the cognitive sharpness of a cotton ball). So here’s what I’m seeing; I hope that these observations are helpful to the github folks in understanding how their service is used.

Firstly, a spot of cognitive organizing: I really like the newish “issues across all of my projects” dashboard, but when I’m thinking about “stuff that’s mine”, I tend to navigate to github.com/bos, and that dashboard isn’t there. Instead, I kick myself and navigate to plain old github.com. You could reasonably respond “okay, fine, just remember that, and you’re done”. And yet somehow this knowledge refuses to stick in my head.

What I find more confusing is the visual clutter at the top of a project page. There are now seven short-but-wide horizontal rows of stuff (both information and links) at the top of a project’s main page. Here’s an annotated screenshot that I hope illustrates what I’m talking about.

I frequently find myself looking for the commits page, which is in the middle of row number 6. At least for me, there seems to be no escaping the need to scan across every row in turn until I reach row 6, where I find the word “commits”. That is, I usually find it; I can easily miss it among all the similar entries if I’m not paying close attention. I find it difficult to visually distinguish the rows at a glance, so there’s no skipping past clusters of stuff that aren’t relevant.

These aren’t killer problems by any stretch, but I do all too often find myself staring at github web pages for 30 seconds at a time, wondering “am I looking at the right page? Did I miss the row of stuff I’m looking for?” I imagine there might be a way to organize these things better, though I’m no visual designer, and I’m afraid I don’t have any crisp suggestions for what might work.

Ease of use

The new decode function complements the longstanding encode function, and makes the API simpler.

New examples make it easier to learn to use the package.

Generics support

aeson’s support for data-type generic programming makes it possible to use JSON encodings of most data types without writing any boilerplate instances.

Thanks to Bas Van Dijk, aeson now supports the two major schemes for doing datatype-generic programming:

• the modern mechanism, built into GHC itself

• the older mechanism, based on SYB (aka "scrap your boilerplate")

The modern GHC-based generic mechanism is fast and terse: in fact, its performance is generally comparable in performance to hand-written and TH-derived ToJSON and FromJSON instances. To see how to use GHC generics, refer to examples/Generic.hs.

The SYB-based generics support lives in Data.Aeson.Generic, and is provided mainly for users of GHC older than 7.2. SYB is far slower (by about 10x) than the more modern generic mechanism. To see how to use SYB generics, refer to examples/GenericSYB.hs.

Improved performance

• We switched the intermediate representation of JSON objects from Data.Map to Data.HashMap, which has improved type conversion performance.

• Instances of ToJSON and FromJSON for tuples are between 45% and 70% faster than in 0.3.

Evaluation control

This version of aeson makes explicit the decoupling between identifying an element of a JSON document and converting it to Haskell. See the Data.Aeson.Parser documentation for details.

The normal aeson decode function performs identification strictly, but defers conversion until needed. This can result in improved performance (e.g. if the results of some conversions are never needed), but at a cost in increased memory consumption.

The new decode' function performs identification and conversion immediately. This incurs an up-front cost in CPU cycles, but reduces reduce memory consumption.