Sometimes, the old ways are the best

Over the past few months, the Sigma engineering team at Facebook has rolled out a major Haskell project: a rewrite of Sigma, an important weapon in our armory for fighting spam and malware.

Sigma has a mission-critical job, and it needs to scale: its growing workload currently sees it handling tens of millions of requests per minute.

The rewrite of Sigma in Haskell, using the Haxl library that Simon Marlow developed, has been a success. Throughput is higher than under its predecessor, and CPU usage is lower. Sweet!

Nevertheless, success brings with it surprises, and even though I haven’t worked on Sigma or Haxl, I’ve been implicated in one such surprise. To understand my accidental bit part in the show, let's begin by mentioning that Sigma uses JSON internally for various purposes. These days, the Haskell-powered Sigma uses aeson, the JSON library I wrote, to handle JSON data.

A few months ago, the Haxl rewrite of Sigma was going through an episode of crazytown, in which it would intermittently and unpredictably use huge amounts of CPU and memory. The culprit turned out to be JSON strings containing zillions of backslashes. (I have no idea why. If you’ve worked with large volumes of data for a long time, you won’t even bat an eyelash at the idea that a data store somewhere contains some really weird records.)

The team quickly mitigated the problem, and gave me a nudge that I might want to look into the problem. On Sunday evening, with a glass of red wine in hand, I finally dove in to see what was wrong.

Since the Sigma developers had figured out what was causing these time and space explosions, I immediately had a test case to work with, and the results were grim: decoding a mere megabyte of continuous backslashes took over a second, consumed over a gigabyte of memory, and killed concurrency by causing the runtime system to spend almost 90% of its time in the garbage collector. Yikes!

Whatever was going on? If you look at the old implementation of aeson’s unescape function, it seems quite efficient and innocuous. It’s reasonably tightly optimized low-level Haskell.

Trouble is, unescape uses an API (a bytestring builder) that is intended for streaming a result incrementally. Unfortunately the unescape function can’t hand any data back to its caller until it has processed an entire string.

The result is as you’d expect: we build a huge chain of thunks. In this case, the thunks will eventually write data efficiently into buffers. Alas, the thunks have nobody demanding the evaluation of their contents. This chain consumes a lot (a lot!) of memory and incurs a huge amount of GC overhead (long chains of thunks are expensive). Sadness ensues.

The “old ways” in the title refer to the fix: in place of a fancy streaming API, I simply allocate a single big buffer and blast the bytes straight into it.

For that pathological string with almost a megabyte of consecutive backslashes, the new implementation is 27x faster and uses 42x less memory, all for the cost of perhaps an hour of Sunday evening hacking (including a little enabling work that incidentally illustrates just how easy it is to work with monad transformers). Not bad!

Posted in haskell
88 comments on “Sometimes, the old ways are the best
  1. Marco says:

    Great article! It’s interesting to see how sometimes the “old ways” can still be effective in solving modern problems. The author’s approach of allocating a single buffer for handling JSON strings with backslashes resulted in significant improvements in performance and memory usage. It’s a good reminder that simpler solutions can often be the best ones. | Charlotte Drywall Contractors

  2. Angel17 says:

    I agree with this, Old ways are better than today’s ways in different ways.

  3. Malisa says:

    Download All books and Novels for free on just one click:
    https://novelabx.com/

  4. Malisa says:

    Download your favourite book on one click:
    https://novelabx.com/

  5. Malisa says:

    Free Novels in PDf and ePub Formate:
    https://novelabx.com/

  6. Malisa says:

    Download all books and novels free on website
    novelabx.com

  7. Malisa says:

    Download all the novels free
    https://novelabx.com/

  8. Malisa says:

    “Fantastic tips! I especially love using fresh herbs to add flavor to my dishes. Thanks for sharing! You can also visit this website
    to download free Novels, https://novelabx.com/

  9. Wilburngap says:

    Hello friends!
    Today luck has decided to knock on your door!
    We invite you to visit our online casino and get a welcome bonus for registration.
    It is we who have the most honest loyalty compared to others!
    We have:
    Sports betting
    The newest and rated slot machines
    Instant withdrawal of your deposit to the account
    And much more, it’s better to see for yourself
    Go to us and complete a quick registration
    enter your private promo code – 997660 And it is to you that fortune will smile today!
    Sincerely and see you soon!

  10. Wilburngap says:

    Hello friends!
    Today luck has decided to knock on your door!
    We invite you to visit our online casino and get a welcome bonus for registration.
    It is we who have the most honest loyalty compared to others!
    We have:
    Sports betting
    The newest and rated slot machines
    Instant withdrawal of your deposit to the account
    And much more, it’s better to see for yourself
    Go to us and complete a quick registration
    enter your private promo code – 77132 And it is to you that fortune will smile today!
    Sincerely and see you soon!

  11. Kelly says:

    If you’ve worked with large volumes of data for a long time, you won’t even bat an eyelash at who is the best drywall company and at the idea that a data store somewhere contains some really weird records.

  12. Malisa says:

    Nice Article. You can also visit our website to Download all the books and Novels for free https://novelabx.com/

  13. Malisa says:

    Nice Article. You can also visit our website to Download all the books and Novels for free https://novelabx.com/

  14. Maria says:

    Absolutely! ‘Sometimes the old ways are the best’ rings true in many aspects of life, including some timeless traditions and practices. There’s a certain charm and effectiveness in embracing tried-and-true methods.

    When it comes to maintaining the aesthetics and functionality of our homes or businesses, old-fashioned attention to detail can make all the difference. For instance, in the world of window cleaning, relying on a trusted and experienced service provider can truly stand the test of time. If you’re in need of top-notch window cleaning services, I’d recommend checking out https://www.windowcleanersburnaby.com/industrial-window-cleaning-company. Their commitment to quality and professionalism reminds us that classic values still have a place in our modern world!

  15. Kysha Isabel says:

    Thank you for sharing such valuable information. It’s definitely going to help me in my work. Pressure Washing Services Fayetteville

  16. Rovic Medrano says:

    Thanks for letting us know! https://www.sidingcontractorvictoriabc.com/siding.html

  17. Shanaia Jane says:

    I am glad that I found this article, as it was precisely what I was looking for. I had been having trouble finding information on this topic, but this article was extremely helpful. I am grateful to the author for taking the time to write such a comprehensive and informative article. SEO companies Tampa

  18. Scarlet Collins says:

    Thank you for letting us know this kind of information. I’m sure I’ll remember this in the future. Drain Cleaning Massachusetts

  19. Amber Brion says:

    The passage highlights the importance of efficient coding and the impact that seemingly small changes can have on the performance of critical systems like Sigma. It also demonstrates the power of Haskell and its libraries for solving real-world engineering challenges. http://www.nexgenproservices.com/

  20. Tradition holds the key to preserving cultural heritage and passing it on to future generations.

  21. Victoria Meterri says:

    Your content is a breath of fresh air. It’s thought-provoking and offers a new perspective. Wrought Iron Fence Panels Boston

  22. Miko says:

    When it comes to finding the most effective approach, we often tend to look for the latest and greatest methods. However, we shouldn’t underestimate the power of traditional techniques, for they can be just as reliable and sustainable as modern ones. Our drywall company still believes in it. After all, the wisdom of the past has been tested and honed through time and experience. So before dismissing the old ways, let’s remember that they have stood the test of time and can still be valuable in today’s world. Nonetheless, we must also embrace progress and innovation to keep advancing and improving our lives.

  23. Tempest says:

    That is very interesting to hear! I am curious to know more about the benefits of rewriting Sigma in Haskell.

  24. Emma Nagets says:

    Absolutely, ‘Sometimes, the old ways are the best.’ There’s a timeless wisdom in traditional approaches that has stood the test of time. It’s always valuable to learn from the past and apply those lessons to the present. A blend of the old and the new can often lead to innovative solutions and a deeper appreciation for the foundations of knowledge. This topic reminds us to cherish the wisdom of our ancestors and keep it alive for future generations, Junk Car Removal Lynn MA

  25. Diana says:

    It’s truly fascinating how sometimes, the old ways prove to be the most effective, as demonstrated in Bryan O’Sullivan’s insightful blog post. The Sigma engineering team at Facebook embarked on a significant Haskell project, rewriting Sigma, a vital tool in their fight against spam and malware. Despite the mission-critical nature of Sigma, it needed to scale to handle a growing workload of tens of millions of requests per minute. The team’s decision to rewrite Sigma in Haskell using the Haxl library paid off, resulting in higher throughput and lower CPU usage. However, success often comes with unexpected challenges, as Bryan recounts his accidental involvement in one such surprise related to Sigma’s use of JSON internally.

    Finding new customers is a cornerstone of business growth, and one of the most effective ways to achieve this is by utilizing the most accurate mailing lists available. In the ever-evolving landscape of marketing and sales, precision matters more than ever, and accurate mailing lists can be a game-changer. These lists are meticulously curated, ensuring that your outreach efforts are directed towards the most relevant audience, saving you both time and resources. Whether you’re launching a new product, expanding into new markets, or simply looking to boost your customer base, having access to up-to-date and precise mailing lists empowers your marketing campaigns to reach the right people at the right time, enhancing your chances of success in a competitive business environment.
    CHECK THIS SITE:https://www.digitaldirectmailservices.com/mailing-list.html

  26. The power of optimization and practical problem-solving shines through! 😄🚀

  27. Indeed! Some people focus on new approaches to achieve great results without realizing that the old ways are still the best.

    George from https://www.jacksonvillencdrywall

  28. Mark Elliot says:

    Meta,Inc. VP of Engineering, AI Software

  29. Kath says:

    The team quickly mitigated the problem and gave our drywall company a nudge that I might want to look into the problem.

  30. mavan24 says:

    This is a reminder that even in modern rewrites, unanticipated challenges can arise, emphasizing the importance of thorough testing and continuous optimization.

    Mavan, drywall contractor

  31. Amber Brion says:

    This accidental bit of performance optimization turned out to be highly effective, demonstrating the power of revisiting and simplifying certain implementations to achieve significant gains in efficiency and resource usage. gallery

  32. Brett says:

    That was cool. Amazing read for sure! Gravity Wall

  33. rivot41458 says:

    Kudos to the Sigma team for successfully navigating and optimizing their Haskell project! | https://www.drywallwaco.com

  34. Sio says:

    I admit this is the greatest read I had so far!
    Excavation Contractor

  35. You have a talent for simplifying complex topics! Learn about water heater replacement in Goodyear here.

  36. Kazuha Mishima says:

    I like your takes on the matter. Great writing on the article, too! Thanks for the content. Victoria Stairs

  37. Don says:

    Greetings from our drywall company! We agree that the old ways is still the best!

  38. Shanaia Jane says:

    It’s evident that you prioritize the reader’s understanding, taking the time to break down concepts and provide context. The use of real-world examples and practical applications further enhances the educational experience.
    Stone Valley Meadows

Leave a Reply

Your email address will not be published. Required fields are marked *

*