No F*cking Idea

Common answer to everything

Pong Tests Are Wrong

| Comments

After short talk with tomash and his gist https://gist.github.com/2871286 with performance of vibe.d i decided to write this post. In my opinion pong tests are wrong and do not show real “performance”. I said to him to check sinatra with thin handler not webrick and that showed ~1.6k req/sec which is not bad at all. While at my box it was ~900 req/sec so significantly less (Macbook Pro i5).

From this gist we can see that his vibe.d benchmark set pong test at 8425.85 req/sec (He used ab and i used httperf).

Warp

My first candidate in this competition of pong tests is haskell warp handler.

code:

pingpong.hs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{-# LANGUAGE OverloadedStrings #-}

import Network.Wai
import Network.Wai.Handler.Warp
import Network.HTTP.Types (status200)
import Blaze.ByteString.Builder (copyByteString)
import qualified Data.ByteString.UTF8 as BU
import Data.Monoid
import Data.Enumerator (run_, enumList, ($$))

main = do
    let port = 8000
    putStrLn $ "Listening on port " ++ show port
    run port app

app req = return $
    case pathInfo req of
        ["pong"] -> pong
        x -> index x

pong = ResponseBuilder status200 [ ("Content-Type", "text/plain") ] $ mconcat $ map copyByteString
    [ "pong" ]

index x = ResponseBuilder status200 [("Content-Type", "text/html")] $ mconcat $ map copyByteString
    [ "<p>Hello from ", BU.fromString $ show x, "!</p>"
    , "<p><a href='/pong'>pong</a></p>\n" ]

it is semi rack like syntax ;)

Benchmark results

I prepared results to show how big lie is pong test. Because in this type of test / showoff all you test is how fast really you can accept connections. Single thread will always win :). But lets look at results:

  • Multi threaded – 50 threads – Request rate: 858.8 req/s (1.2 ms/req) BOOO!!!!
  • Multi threaded – 4 threads ( on for each core ) – Request rate: 10020.8 req/s (0.1 ms/req) Vibe.d die!!! Yeah!!!!
  • Single threaded – default compilation – Request rate: 13584.1 req/s (0.1 ms/req) Mother of God!

Tested with httperf --uri=/ --port=8000 --num-calls=10000 --num-conns=20 httperf command.

Summary

You can test it on your own, i did it on latest 7.4.1 Ghc from haskell platform on OSX 10.7. And post reply with your results :) maybe i missed something. Code, scripts to build and run are in repository https://github.com/JakubOboza/haskell-warp-pong-test.

So how to test ?!

I think you should test your application in default environment so with db behind it, but anyone in this scenario can say it is testing performance of db. But everyday… users are really testing performance of our db ;>… or the weakest of the elements in chain. So if your db / rendering engine is performing at level of 50 req / sec fast app handler will not turn it into 5000 req / sec.

Comments