Observations

I started making mind notes of things that i understand about system i’m working with. I never had time to think about grammar of this and really get into it. The system has many parts but i thought it can be reproduced in for of language that is easy to parse and nice to work with. I think i might change my current solution into more LISP looking thing just to make it easier to parse.

Observation 1

Each campaign is a sequence of actions that happens one after another. So my initial thought was “this is simple”. We can represent it by list of actions like this

1

 [step] -> [step] -> [step]

But this is true only for simple campaigns and most of them are NOT like this :(

Observation 2

Most of the campaigns are built around key thing. This thing is making decisions! So if i can incorporate “if” i WIN ! Lets have a look

1
2
3
4

                     /> [step]
                   (No)
                    |
  [step] -> [if a < 5] - ( yes ) -> [step]

Yes! This was it, i think this is solution to all problems, ability to represent campaign as a sequence of steps mixed with if statements essentially abstract syntax tree of campaign.

Observation 3

It is not a tree…its more a graph. But AST will make it :] it is a programming language! So this was in my head and i did not had time to work on it… but today i decided to give a try and I made first impression of AST that we would need to have to make it work.

AST

I wrote simple grammar and made a parser of a very simple “ify” language. My language starts like this…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

-- Grammar
-- x is variable
-- a is artmetic thing aka number
-- n is actual number eg. 1
-- b is boolean
-- opa is arthmetic operator
-- opb is boolean operator
-- opr is comparison operator
-- S is statement
--  a   ::= x | n | - a | a opa a
--  b   ::= true | false | not b | b opb b | a opr a
--  opa ::= + | - | * | /
--  opb ::= and | or
--  opr ::= > | <
--  S   ::= x | x <- a | S1; S2 | ( S ) | if b then S1 else S2

This shows that the grammar is very simple, we can have assigments, operators to test and compare and ofc IF statement ;) this is our key to divnity!

Parser looks kinda ugly because i used parts of the code i wrote before and had i in different projects.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174

import System.Environment(getArgs)
import System.IO
import Control.Monad
import Text.ParserCombinators.Parsec
import Text.ParserCombinators.Parsec.Expr
import Text.ParserCombinators.Parsec.Language
import qualified Text.ParserCombinators.Parsec.Token as Token

--  Grammar
-- x is variable
-- a is artmetic thing aka number
-- n is actual number eg. 1
-- b is boolean
-- opa is arthmetic operator
-- opb is boolean operator
-- opr is comparison operator
-- S is statement
--  a   ::= x | n | - a | a opa a
--  b   ::= true | false | not b | b opb b | a opr a
--  opa ::= + | - | * | /
--  opb ::= and | or
--  opr ::= > | <
--  S   ::= x | x <- a | S1; S2 | ( S ) | if b then S1 else S2 

data AExpr = Var String
            | IntConst Integer
            | Neg AExpr
            | ABinary ABinOp AExpr AExpr
              deriving (Show)

data BExpr = BoolConst Bool
            | Not BExpr
            | BBinary BBinOp BExpr BExpr
            | RBinary RBinOp AExpr AExpr
             deriving (Show)

data ABinOp = Add
            | Subtract
            | Multiply
            | Divide
              deriving (Show)

data BBinOp = And | Or deriving (Show)
data RBinOp = Greater | Less deriving (Show)

data Stmt = Seq [Stmt]
           | Assign String AExpr
           | If BExpr Stmt Stmt
           | Ident String
             deriving (Show)

languageDef =
   emptyDef { Token.commentStart    = "/*"
            , Token.commentEnd      = "*/"
            , Token.commentLine     = "#"
            , Token.identStart      = letter
            , Token.identLetter     = alphaNum
            , Token.reservedNames   = [ "if"
                                      , "then"
                                      , "else"
                                      , "true"
                                      , "false"
                                      , "not"
                                      , "and"
                                      , "or"
                                      ]
            , Token.reservedOpNames = ["+", "-", "*", "/", "<-"
                                      , "<", ">", "and", "or", "not"
                                      ]
            }

lexer = Token.makeTokenParser languageDef

identifier = Token.identifier lexer -- parses an identifier
reserved   = Token.reserved   lexer -- parses a reserved name
reservedOp = Token.reservedOp lexer -- parses an operator
parens     = Token.parens     lexer -- parses surrounding parenthesis:
integer    = Token.integer    lexer -- parses an integer
semi       = Token.semi       lexer -- parses a semicolon
whiteSpace = Token.whiteSpace lexer -- parses whitespace

whileParser :: Parser Stmt
whileParser = whiteSpace >> statement

statement :: Parser Stmt
statement =   parens statement
           <|> sequenceOfStmt

sequenceOfStmt =
   do list <- (sepBy1 statement' semi)
      return $ if length list == 1 then head list else Seq list

statement' :: Parser Stmt
statement' =   ifStmt
            <|> assignStmt
            <|> identName

ifStmt :: Parser Stmt
ifStmt =
   do reserved "if"
      cond  <- bExpression
      reserved "then"
      stmt1 <- statement
      reserved "else"
      stmt2 <- statement
      return $ If cond stmt1 stmt2


identName :: Parser Stmt
identName =
   do  varName <- identifier
       return $ Ident varName

assignStmt :: Parser Stmt
assignStmt =
   do var  <- identifier
      reservedOp "<-"
      expr <- aExpression
      return $ Assign var expr

aExpression :: Parser AExpr
aExpression = buildExpressionParser aOperators aTerm

bExpression :: Parser BExpr
bExpression = buildExpressionParser bOperators bTerm

aOperators = [ [Prefix (reservedOp "-"   >> return (Neg             ))          ]
              , [Infix  (reservedOp "*"   >> return (ABinary Multiply)) AssocLeft]
              , [Infix  (reservedOp "/"   >> return (ABinary Divide  )) AssocLeft]
              , [Infix  (reservedOp "+"   >> return (ABinary Add     )) AssocLeft]
              , [Infix  (reservedOp "-"   >> return (ABinary Subtract)) AssocLeft]
               ]

bOperators = [ [Prefix (reservedOp "not" >> return (Not             ))          ]
              , [Infix  (reservedOp "and" >> return (BBinary And     )) AssocLeft]
              , [Infix  (reservedOp "or"  >> return (BBinary Or      )) AssocLeft]
              ]

aTerm =  parens aExpression
      <|> liftM Var identifier
      <|> liftM IntConst integer

bTerm =  parens bExpression
      <|> (reserved "true"  >> return (BoolConst True ))
      <|> (reserved "false" >> return (BoolConst False))
      <|> rExpression

rExpression =
   do a1 <- aExpression
      op <- relation
      a2 <- aExpression
      return $ RBinary op a1 a2

relation =   (reservedOp ">" >> return Greater)
          <|> (reservedOp "<" >> return Less)


parseString :: String -> Stmt
parseString str =
   case parse whileParser "" str of
     Left e  -> error $ show e
     Right r -> r

parseFile :: String -> IO Stmt
parseFile file =
   do program  <- readFile file
      case parse whileParser "" program of
        Left e  -> print e >> fail "parse error"
        Right r -> return r

main = do
  args <- getArgs
  ast <- parseFile(head args)
  putStrLn $ show ast

But it works… we can parse. Now i have to make it more production ready and less unstable ;]. This is first attempt at this idea. I need to expand grammar and think about it more then just few minutes. but i think it is a good start. Next step for me is to make better grammar and parser and move to building interpreter of this mini language.

Startup era

We live now in startup era and main focus of people building new products is to make them fast. Who is the first to build some solution is most likely to get the market and monetize it. So almost every father of startup is looking for a technology that will help him build thing fast. This is the main reason that Rails are so popular, they are not fast, they consume memory like crazy but they offer you out-of-box something that you can work on. It has all properties you are looking for and everything is there with rich documentation and easy to learn and use conventions. Rails are like a gift to people who wants to “prototype” application fast, but for 99% of solution out there they never leave prototype stage.

Going big!

Rails started small, the community was small and it was normal that not many people cared about finding security issues because it was more important to add support for everything and make the framework richer and richer. First thing that was the biggest helper in opening every door was raising popularity but this is also the biggest enemy of the framework. It started traction around the framework and people started to hack it, audit code and find things that can be exploited.

Natural point in life

Every framework has to go through this type of periods in its life, there is no bug free code. People will find bugs in code base. What is the best way to prevent this ? Have a great team of engineers that follows the trends. But… most of startup owners wants to cut the costs, they outsource the work and have periods of life of the product that simply nobody cares about it in technical way. Or it can be even worse, team can be focused so much on features because of the boss pressure that they don’t have time for it.

Building solution

Some of startup starters will be technical, more technical or not technical at all, in most cases it is not technical at all and this is a problem. This leads to building every startup base on frameworks like Rails or Django. Scenario looks the same every time. First team spends a lot of time building initial release and next it goes big so they don’t have a way to scale it in other way than spaming rails instances and changing database. So if something hits rails it hits whole platform and that hurts. Second scenario is that team is having some sort of engine and just a rails front end this is a smart approach because if something will go really wrong it only will kill front end and this is not bad! But how many teams build solutions in this way ? Not many, mostly polilingual team that know something more than ruby. What i experienced in my career is that people don’t wan to do some initial design decisions before start they just want to have product and “we will think about it later”. This from business point of view is ok but this “later” is often very early. Some startups are lucky enough to have engineer that are smart and know how to solve problems using background processing, caching and a bit of cheating (eg. like youtube do with vote count) so make everything work smooth but most of startups are created in a crazy way with big stress on speed of building.

Problems, security and future

People will always suggest things like YAML.safe_load in my personal opinion its not a solution but just a patch. Why not disable support for YAML, JSON, XML and any other type of request and make it explicit what you accept as form of request for actions. Trying to apply every possible parser to input is not often best thing to do.

Summary

I think problems like rails have now with security are not something we should cry about, it is just another step in becoming mature framework and problems like this can happen always with every framework. We have to embrace it and devise tactics to deal with it in a timely fashion so we will not be affected. Building software is not cheap, maintaining it is not cheap but… if you will hit right market you will get the money back.

This is how i see startup stage now.

Ports

Firt thought was to build small C program that will check stuff periodically or just “check stuff” in SHT1x and just print it out to output. So my first attempt was

For the example here i will use /proc/cpuinfo

1
2
3
4
5
6
7
8
9
10
11
12
13
14

-module(mycpuinfo).
-export([info/0]).

-define(CMD, "cat /proc/cpuinfo").

info() ->
    Port = open_port({spawn, ?CMD}, [{packet, 1}, use_stdio, exit_status, binary]),
    port_command(Port, <<>>),
    receive
  {Port, {data, Response}} ->
      {ok, {cpuinfo, Response}};
  _  -> error
    end.

Nothing super special, except that raspberry pi is not really cool with different packet sizes and it can for example not read whole input properly. I observed some issues with it. So i decided to explore more FFI.

erl_interface

The thing i found is called erl_interface and it is designed for FFI. This is it! What you do is you build process like thing in C and micro module in erlang that handles this. (It is just to make it look nice). But there are few glitches!

This is my module posting back “pong” on “ping” message

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#include "erl_interface.h"
#include "ei.h"

#define BUFFSIZE 100

int main(int argc, char** argv){

  int fd;
  unsigned char buf[BUFFSIZE];
  ErlMessage emsg;
  int msg_type;
  ETERM *fromp, *argp, *rsp;

  erl_init(NULL, 0);

  if(erl_connect_init(1, "cookie",  0) == -1){
    erl_err_quit("init failed");
  }

  if((fd = erl_connect("emu@raspberrypi")) < 0 ){
    erl_err_quit("Emus could won war with australia but you still can't connect to emu :(");
  }

  while(1){
    msg_type = erl_receive_msg(fd, buf, BUFFSIZE, &emsg);
    if(msg_type == ERL_TICK){
      /* Emu is checking australian defences, Polish people are safe! so I can ignore this */
    }
    else if(msg_type == ERL_ERROR){
      /* Huston we have an Emu! */
      break;
    }
    else {
      /* our pro msg */
      fromp = erl_element(1, emsg.msg);
      argp = erl_element(2, emsg.msg);
      rsp = erl_format("pong");
      erl_send(fd, fromp, rsp);


      erl_free_term(emsg.from);
      erl_free_term(emsg.msg);
      erl_free_term(fromp);
      erl_free_term(argp);
      erl_free_term(rsp);
    }
  }

  return 0;

}

First of all complexity goes p fast. You have to think about many things. Here i know that my host is called “emu@raspberrypi” and this is actually process that runs so if you will not remember about freeing memory you will fast learn what means “memory leak”.

But most important thing is how to build this and run.

Building

My make file looks like this

1
2

all:
  gcc -o lolpong -I/usr/lib/erlang/lib/erl_interface-3.7.7/include -L/usr/lib/erlang/lib/erl_interface-3.7.7/lib lolpong.c -lerl_interface -lei -pthread

It is important to remember about -pthread.

Running everything.

No how to make everything work… we need module

1
2
3
4
5
6
7
8

-module(lolpong).
-export([ping/0]).

ping() ->
    {any, 'c1@raspberrypi'} ! {self(), ping},
    receive
  R -> {ok, R}
    end.

` This c1 is for c extension 1 not super obvious :) cN will be for c extension N =).

And finally we need to spawn our node…

1

λ  erl -sname "emu" -setcookie "cookie"

This spawns node named emu@raspberrypi on my mini box with cookie “cookie” now … i said finally… but it was not final step.

Final step is to run out lolpong binary. It is important to run it after node is up because it will try to connect to this node.

1

./lolpong

Now we can run in our erlang shell check if everything works!

1
2

(emu@raspberrypi)1> lolpong:ping().
{ok,pong}

Works!

Summary

It is good fun, now i need to wait for the parts to assemble everything and build rest of application :)

Cheers!!

Making request

So first of all i used builtin inets for making requests. So first thing to do is to start inets!

1

inets:start().

Ok i generalized using httpc to really simple thing

1
2
3
4
5

post(URL, ContentType, Body) -> request(post, {URL, [], ContentType, Body}).
get(URL)                     -> request(get,  {URL, []}).

request(Method, Body) ->
    httpc:request(Method, Body, [], []).

So if you will a call you will get something like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

httpc:request(get, {"http://www.youtube.com/watch?v=JvxG3zl_WhU", []}, [], []).
{ok,{{"HTTP/1.1",200,"OK"},
     [{"cache-control","no-cache"},
      {"date","Tue, 22 Jan 2013 20:36:16 GMT"},
      {"server","gwiseguy/2.0"},
      {"content-length","221233"},
      {"content-type","text/html; charset=utf-8"},
      {"expires","Tue, 27 Apr 1971 19:44:06 EST"},
      {"set-cookie",
       "PREF=f1=50000000; path=/; domain=.youtube.com; expires=Fri, 20-Jan-2023 20:36:15 GMT"},
      {"set-cookie",
       "use_hitbox=d5c5516c3379125f43aa0d495d100d6ddAEAAAAw; path=/; domain=.youtube.com"},
      {"set-cookie",
       "VISITOR_INFO1_LIVE=99hU0tGtAng; path=/; domain=.youtube.com; expires=Thu, 19-Sep-2013 20:36:15 GMT"},
      {"set-cookie",
       "recently_watched_video_id_list=dda4a4415949369ee6eb305306b40e47WwEAAABzCwAAAEp2eEczemxfV2hV; path=/; domain=.youtube.com"},
      {"x-frame-options","SAMEORIGIN"},
      {"x-content-type-options","nosniff"},
      {"p3p",
       "CP=\"This is not a P3P policy! See http://support.google.com/accounts/bin/answer.py?answer=151657&hl=en-GB for more info.\""},
      {"x-xss-protection","1; mode=block"}],
     [60,33,68,79,67,84,89,80,69,32,104,116,109,108,62,60,104,
      116,109,108,32,108,97,110|...]}}

One more thing i added was extracting body of the response

1

  response_body({ok, { _, _, Body}}) -> Body.

So after calling it on response we can get body

1
2
3
4

  Response = httpc:request(get, {"http://www.youtube.com/watch?v=JvxG3zl_WhU", []}, [], []).
  Body = response_body(Response).
  io:format("~s~n",[Body]).
  <!DOCTYPE html><html lang="en"><head> ...

Parsing Json

So next thing to do was to parse json :) for this pupose use two libs mochijson2 or jiffy.

Mochijson2

So this one is easy to find and use. All you need to do is grab file from mochiweb project and compile it :) it has two method encode and decode.

1
2

mochijson2:decode(STUFF).
mochijson2:endcode(STUFF).

One thing worth mentioning is that it expects input in a bit structured format.

So typycaly it will use “struct” symbol for json “{}” objects.

1

{struct, [{name, <<"jakub">>}, {age, 27}, {ids, [1,2]}]}

So array in json will be list but object will have to be in tuple with struct. It is awkward at first but you can get used to it. but there is easier thing to use…

Jiffy

This! There is a good example on main page.

1
2
3
4
5
6
7

Eshell V5.8.2  (abort with ^G)
1> jiffy:decode(<<"{\"foo\": \"bar\"}">>).
{[{<<"foo">>,<<"bar">>}]}
2> Doc = {[{foo, [<<"bing">>, 2.3, true]}]}.
{[{foo,[<<"bing">>,2.3,true]}]}
3> jiffy:encode(Doc).
<<"{\"foo\":[\"bing\",2.2999999999999998224,true]}">>

This shows how nice it is to use :D no ‘struct’ pure love!

You can get it here https://github.com/davisp/jiffy

Summary

This is just note to self to not be checking up docs looking for this stuff again :).

New Discoveries

“Theory is the thing we all skip and later on regret skipping” (Yes i quoted my self, I heard someone did it before so its not hip). We learn a lot of stuff and some of the things simply slip through and gets forgotten. That is opening for “new discoveries” or “rediscoveries” of some solutions. I have a thesis that “if something is simple it will get its niche” (did it again). This would explain new hipe for Nodejs <nodejs.org>. I don’t want to judge technologies I did some stuff in node and i have small portion of experience but this text is not about this, its about technology behind stuff people think it’s brilliant and new nowadays.

Event loop | Evented solutions

Event loops exists long long and are nothing new. When someone says event loop first thing i think about is not nodejs or eventmachine or twistted but WinAPI. Yes old WinAPI and suddenly everybody realize that every GUI (Graphical User Interface) solution is evented. QT, WinAPI and many other even older solutions. This is nothing new, so lets take a trip through the cards of history to find other gems that we can rediscover and become new rocket scientists…or first tech archeologists.

Step One: Alternatives to sequencing

So most common thing people do is sequencing and even if we use brain in other way it is easy to us to think about and form sequences because it is something we understand. This is basic unit of work we think about. So most of people write small function that sequence some action and they build programs from small functions calling small functions in a really big sequence.

Alternative solution to do this is event machine, this solution will enable us to generate a lot of small event and if we will manage to force programmer to make them as small as possible and as fast as possible we can achieve a feeling of multitasking. How does it work ? we have one loop that picks events from event queue and process them and as long as nobody will make event that is blocking everything this will work like charm. Key to this solution is “non blocking”.

Achieving Non-blocking

To do this we need to have a way of “wait until will happen and if it happen’ed trigger event” thing. This can be done by many different solutions. For example for file descriptors (everything is a file) we have stuff like select, poll, epoll, kqueue. and this is not new its the 80’. Later on POSIX first official POSIX thing 1997. 15 years ago! Previously mentioned nodejs is representing this category (ofc from the fresh news!). Does it have flaws ? Yes! People call this async… its not async, not real async. Real async was also implemented in 80’s :D

Signals

Signals are also very old, basically this is similar concept (every concept is similar) You have a slot that handles signal and something that emits signals. It’s very handy and was implemented with big success in QT library. Does it has flaws ? basically same as event loop solutions. It’s a way of doing “message passing” pattern.

Message passing

Message passing really well implemented in Erlang is another way of achieving same result, imho most clean and good solution in the industry right now. Simply you cast messages and other entities receive them and react. YES THIS IS HOW PEOPLE WORK. Most of thins seems to be inspired by nature… but lets not go this route and start religious war.

Async

The real unix async calls. Yes unix has this built in.

1
2
3

#include <aio.h>
int aio_write(struct aiocb *aiocbp);
int aio_read(struct aiocb *aiocbp);

You call aio_write and write to a file will happen, sometime in future… BEFORE END OF THE WORLD, maybe!

Wow! This is fresh! nobody is using it nowadays! you should make new nodejs on it and call it modejs. Or wait ? Can you ? nobody really know how to use them and not be cluster f@cked after five seconds. Real async syscalls in unix. That is awesome and its older than your son! Lets dig more…

Reactive programming

This is the sh*t, real event programming, you have real reactor nothing is faked like in eventloop! But its nothing new… also implemented in 80’s. But what is the difference between reactive programming and event loop ? Not much because this is more conceptual thing around building your reactor. In reactor concept we think about data flows and reacting to changes in them.

We don’t have to research things that are found!

Map Reduce, Google, Smart people

So how mega smart people in Google work ? I don’t know I don’t work for google and I’m not smart but I assume they want to get sh*t done. And after reading Amazon Dynamo Paper http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf , Google Big Table paper http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/pl//archive/bigtable-osdi06.pdf and many other i think they developed it in kitchen. Yes in a kitchen.

They sat down, people from google and amazon i imagine 6 engineers took 1 apple (Steve Jobs was serving) and asked them self how to how to cut it into 6 parts so they can eat it. They took knife and sliced into 6 parts and split it to each other. Suddenly one stood up (Amazon guy with headphones in avatar) and said. Each part of apple is exclusive (patent trolls have everything exclusive nowadays) to each of us so we can eat them on our own. LETS MAKE DB USING THIS TECH AND WRITE A PAPER ON IT. Next guy said.. yeah we have limited number of apple parts so it is predictable and we form a RING from it… and thats how Big Table and Amazon dynamo was born. The Google cook who was slicing apple said, i sliced apple in parts one by one with one knife… AND HE WROTE MAP REDUCE PAPER.

They did not invented anything really new, they just approached problem simple. We can’t split and scale complex solutions so we will make everything simple. And it happens that most of things can be turned into linear or near linear problem.

Is Map Reduce really something new ?

In my opinion not. Map existed in lisp world probably since 70’s. Now someone took map and sequence of data that is exclusive and don’t depend on each other and said “What will happen if i will split this list into two parts and run map on each in two or more threads and then just sum results ?” and map reduce was born. Concept is dead simple. But its nothing new! I’m not taking here credit from google engineers. I think guys there are very smart and by using simple things in my eyes they are ^2 smart. But is it something ground breaking new ? No.

So maybe we should more often look into past. Maybe we can find more gems like this… i bet google has technology archeology department :D

Step Two: Type writer

I’m 80’ kid. I remember type writers and the sound so for me understanding how files in OS (Unix or in general) works was simple, but when recently i was explaining it to friend who is 90’ kid he did not got the concept of that well. Why ? Because he is from age where type writers did not exist. For me it was a bit shocking because i thought everybody knows exactly how they work. And when i said why we have \r and what it is i was for a moment happy that I’m very very old :D.

Queues, Interpreters and other stuff.

Technology archeologiest must be open to new things, read about everything and most of all, never stop thinking “this had to be already invented”. When I first suggested Queue as a solution to a problem some people looked at me like i would be snake. FIFO queues, something so basic… still some people don’t even think about them. This is dull example but basic data structures should be well known by everybody still… things that freezes blood in your veins happens.

Building DSL and interpreters is good fun, its not easy often I’m still learning a lot but this should not be “devil” to other programmers.

Old manuscripts

There are books that contains tones of cool knowledge, you can be technology archeologist like me! just venture into them and discover “Stuff”. First book i would recall here is Richard Stevens.

• 1990 – UNIX Network Programming – ISBN 0-13-949876-1
• 1992 – Advanced Programming in the UNIX Environment – ISBN 0-201-56317-7
• 1994 – TCP/IP Illustrated, Volume 1: The Protocols – ISBN 0-201-63346-9
• 1995 – TCP/IP Illustrated, Volume 2: The Implementation (z Garym R. Wrightem) – ISBN 0-201-63354-X
• 1996 – TCP/IP Illustrated, Volume 3: TCP for Transactions, HTTP, NNTP, and the UNIX Domain Protocols – ISBN 0-201-63495-3
• 1998 – UNIX Network Programming, Volume 1, Second Edition: Networking APIs: Sockets and XTI – ISBN 0-13-490012-X
• 1999 – UNIX Network Programming, Volume 2, Second Edition: Interprocess Communications – ISBN 0-13-081081-9

This guy is amazing, books are very good quality talking about topics that today are more than active in our community. 22 year old books!

• 1989 - Genetic Algorithms in Search, Optimization, and Machine Learning

Amazing book, I never was big fan of A.I. (maybe i should, that would give me chance to actually have some of I…) but this book opened my eyes on how simple this stuff is. How easy and how powerful. When i was at uni this was one of the books i enjoed the most.

• 1976 - Algorithms + Data Structures = Programs (Prentice-Hall Series in Automatic Computation) by Wirth.

Classic this guy guides you through Pascal and shows you how to program and at the end of book implements subset of language. I say 76’ FFS that good books will not be printed after 2006. 44 years ago published. C’mon!

• 1988 - C Programming Language (2nd Edition)

Next classic, especially section about making your own malloc(). C’mon 1988’ 24 years ago. Brilliant book.

And something fresh

• 2004 - The Art of C++ by Herbert Schildt

Really good book, also common pattern in books that i read… guy is implementing language at the end ( subset of C )

Yes this is good stuff. Old books containing knowledge that industry will discover in 6-10 years.

But what about current research ?

Current stuff like dependent types… i recon we will hear about them in industry after our deaths ;/ next week apocalypse ;/. Yes there are tones of things to rediscover… and it’s fun! lets do it!

Summary

I love when i hear about new stuff and suddenly after inspection it becomes clear it is just old thing on new cogs. I love this. I love technology archeology and i want to bring more examples of this and write about it on my blog.

Yes sending satelite in space is something new…. another event loop implementation is not.

Sorry for ranting that much :D… anyway nobody read this :D.

Lisp

Before i tried Clojure i had few “trips” to Lisp world but i was always stroked by amount of half finished vm’s and implementations that are far from being production ready. For example chicken, scsh, sketchy, mit-scheme, bigloo. Some of them are more some less production ready but none of them is ready for anything real. The most production ready lisp thing i saw in my life was elisp YES f*cking EMACS lisp. But I’m not any expert in lisp world!

Scheme

Scheme is a dialect of Lisp that implements subset of language, in scheme code and data has same form which is a massive game changer. Scheme has super duper hiper mega MACRO building capabilities. And it is probably its biggest problems…

Lisp Hacker’s

I remember when my friend was hacking in lisp, i remember when i was trying to hack stuff in scheme. So scheme is a trap! First stage of lisp/scheme hacker is to build scheme interpreter in scheme, when he finishes this he tries to implement scheme interpreter in his interpreter… Second trap is when he realizes that he wrote enough amount of scheme interpreters and he moves to implementing new features into his interpreter in interpreter in interpreter ( and yes ‘ready’ means that 8/10 runs stuff passes, some crash and half of features don’t even exits so called “junk features” ). So his dialect gets better than Haskell lazy evaluation, better etc… at this stage he has self bootstraping half working interpreter of his language that had no docs and nobody except his can use it and even if he can he must be on drags to make the lazy evaluation happen. Next stage of course is starting to do AI in scheme. So the point is that no shit gets done, its all conceptual work. If Da’vinci would program in Lisp he would not cut of his ear, he would cut of his head.

Life of ()

I wrote some borderline rant on lisp hackers and this was supposed to be micro tutorial on scheme. Well i took the Lisp way of “getting shit done” :D. If you don’t know that lisp bread and butter are ( and ) you should probably run away.

Naow!

So why Clojure ?

Clojure seems to be production ready stuff that can “get shit done”. It support concurrency (with STM - Software Transactional Memory) it is Scheme’y / Lispy and I have read that its hip in Open Source magazine. Thats enough to start learning yeah ?

No! But i want to learn it and write some stuff about it. It seems to be so different from other languages, RP notation and the whole community of Stallman like people.

Clojure runs on JVM ( Java Virtual Machine ) and beside Java being biggest troll language next to C++ ever! JVM is really good. It makes stuff really portable and its good for building stuff like backends for servers. Same goes with Scala using JVM, actually JVM is solid! ( coming from me Java hater ). Its not like some f*cktard writing in “Open Source Magazine” that C++ enables you to write portable code. No, writing os portable code in C++ is not easy, its hard and full of pain.

Getting s*it on your box

First thing you have to do is to install Clojure. This is the easy step. Just visit http://clojure.org/. I installed it using macports by running sudo port install clojure but for more into i would suggest going to:

• Minimum jar needed to start REPL http://clojure.org/getting_started
• Downloads for latest version http://clojure.org/downloads
• Brew / Macports search “clojure”
• In case that nothing else work, go to https://github.com/clojure/clojure and look for “build from source instructions”

Next you will want to install leiningen this is a build tool for Clojure. same procedure. either sudo port install leiningen or http://leiningen.org/ for more instructions.

Running repl and first code!

Enough of this sh*t lets do some stuff. First to run repl you need to do following.

1

clj

I mix it with cjl but thats because i’m a noob. This should prompt you with something like this

1
2
3

20:59 kuba@kuba:~ λ clj
Clojure 1.4.0
user=>

Success!!! We can now try to evaluate some simple stuff.

So now normal person would try to type something like 2 + 6 bad idea! Expressions in Lisp starts and ends with () so if you see things like ((((((((((((((((((((((((((((( or ))))))))))))))))))))))) you know something is happening :D (second one more commonly)

1
2
3
4
5
6

user=> 2 + 6
2
#<core$_PLUS_ clojure.core$_PLUS_@9d686c1>
6
user=> (+ 2 6)
8

RPN now i should quote link to wikipedia on Reverse Polish Notation. (I’m Polish so it is kind’a like looking at my back and trying to read sign on t-shirt). Here you go http://en.wikipedia.org/wiki/Reverse_Polish_notation Reverse polish notation simple words is a stack notation. Where you put operator on stack and then arguments. Or so called prefix notation. C, Java, Ruby are all infix notation eg. 2 + 6 while lisp / scheme is prefix notation aka (+ 2 6) on reality its so cool that you don’t have to repeat your self and you put operator, function only once and a list of arguments eg. (+ 1 2 3 4 5). So lets try it out.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

user=> (+ 1 2 3 4 5 6)
21
user=> (- 10 7)
3
user=> (- 10 7 1)
2
user=> (* 6 7)
42
user=> (* 6 7 2)
84
user=> (/ 4 5)
4/5
user=> (/ 4 5.0)
0.8
user=> (/ 4 5.0 2)

We can see that those examples work in REPL nice!. But wait there is something different between (/ 4 5) and (/ 4 5.0). Yes! Clojure has built in support for rational numbers so 4/5 is a rational 4/5 not rounded estimate of 4/5 while result of evaluating 4/5.0 is a floating point number. Lets see what happens if we will try to do 1/3

1
2
3
4

user=> (/ 1 3)
1/3
user=> (/ 1 3.0)
0.3333333333333333

Yeah the second one is not really accurate. Or is it ?

1
2
3
4

user=> (+ (/ 1 3) (/ 1 3))
2/3
user=> (+ (/ 1 3.0) (/ 1 3.0))
0.6666666666666666

Not really…. it should be something like 0.(6) :)

So this is how you invoke functions, built in functions. But what if you would want to comment something out ? because its a useful info about new scheme dialect you are building?

Comments

You do comments using ; everything after ; is getting ignored! For example:

1
2
3
4
5

user=> (* 1 1) ; most useful thing Clojure has to offer 
1
user=> (/ 1 1)
1
user=>  ; comment about lisp being full of ((((

defn

First thing that makes people happy usually while learning new language is the moment when guy that tries to explain whole concept helps them write first function! Yes thats what i will do now.

To define a function we use defn (old lisp grinders will connect it with def yes!) that defines a function for us, so to it we need to supply name of function, or the symbol that this function will be bind to, arguments and body like this.

1
2
3
4

user=> (defn multiply-by [base amount] (* base amount))
#'user/multiply-by
user=> (multiply-by 5 6)
30

Next we can invoke it eg. (multiply-by 3 4) its worth mentioning that you can use - in function names. But what is really happening ? this is two part thing. First we have def and second we have fn def binds symbol name with expression or value and fn return a function. So it is “like lambda” in fact it is lambda.

Lets go with example.

1
2
3
4
5
6
7
8
9
10

user=> (fn [base] (+ base 1) )
#<user$eval33$fn__34 user$eval33$fn__34@f5bfdbd>
user=> ((fn [base] (+ base 1) ) 5)
6
user=> (def troll-symbol 7)
#'user/troll-symbol
user=> (troll-symbol)
ClassCastException java.lang.Long cannot be cast to clojure.lang.IFn  user/eval41 (NO_SOURCE_FILE:25)
user=> (+ troll-symbol ((fn [base] (+ base 1) ) 5) )
13

This is slightly more complex but also should be easy. In first line we use fn to define function with one parameter called base this returns a function. In line two we do the same but this function is returned and passed and argument of 5 and evaluated to 6. In line three we define symbol troll-symbol with value of 7 we obviously can’t evaluate a value so we get exception! It’s all fine because its value. But we can use it like in last line, where we add to value of troll-symbol effect of evaluation of function that we defined passed with argument of 5.

Don’t be afraid :D. All we did here is explained what defn do, basically defn is just composition of def and fn. Lets you specify symbol that function returned by fn is bound to.

Defn simply saves us key strokes :) Like using Emacs :D this can lead to RSI. =)

1
2
3
4
5
6
7
8

user=> (def troll-symbol 8)
#'user/troll-symbol
user=> troll-symbol
8
user=> (def troll-foo (fn [base] (+ base 1)))
#'user/troll-foo
user=> (troll-foo 7)
8

This exactly shows what happens when you use defn, you get function from fn and bind it to some symbol. That’s it. Its good to have some play with all this brackets :) its fun! try defining something on your own :D

Hello world

So every tutorial has to have something like hello world program! In my tutorial there is space for Hello World also :D

All you need to do is to create a file called hello.clj and type this in!

1

(println "Hello World")

When you will run it using clj hello.clj you will see something like this

1
2

λ clj hello.clj
Hello World

Lambdas!

So i mentioned something called fn for creating functions. But sometime you need a function ad’hoc eg. when you pass it to map function this is done by using different syntax.

1
2

user=> (#(+ 1 %1) 9)
10

You can still use old syntax but this is just faster to type. Remember that %1, %2 … refer to arguments that are passed to this lambda.

Lists and Vectors

Like in all programming languages we have lists and vectors. Those are built in. Syntax for List is simply '() eg. '(1 2 3 4) and for vector aka “array” its [] eg. [1 2 3]. NO COMAS!

1
2
3
4

user=> [1 2 3 4 5]
[1 2 3 4 5]
user=> '(1 2 3 4)
(1 2 3 4)

Lets do something useful with them! print them out naow!

1
2
3
4
5
6
7
8
9
10
11
12
13
14

user=> (map #(println %1) '(1 2 3 4 5))
(1
2
nil 3
nil 4
nil 5
nil nil)
user=> (map #(println %1) [1 2 3 4 5])
(1
2
3
4
5
nil nil nil nil nil)

It is map… so it is ugly because result from println is nil so we printed out contents of vector and list but its ugly. Map is a high order function that takes function and list/array/sequance of arguments, applyies and evaluates funtion on each argument and stores results of this in a list that is returned at the end :) eg. (map (+ %1 1) (1 2 3)) will give [2 3 4].

Lets try to do it with doseq.

1
2
3
4
5

user=> (doseq [i [1 2 3]] (println i))
1
2
3
nil

Works!

Glossary

REPL - Read Eval Print Loop. Simply language console in which you can run code and see results without typing code into file and running interpreter on it.

Lisp - On of the biggest language trolls or only language worth learning. Still can’t decide

Jar - Java archive that contains code, can contain also jam. I like jam.

Summary

This is just the first part of the tutorial i hope you liked it, just covers the very basic of defining and evaluation functions.

Mac, Linux

In this text i use en0, en1 naming convention from OSX if you are linux user you should change it to eth0, eth1 w… check your network config using ifconfig. Basic knowledge required! =)

Tcpdump

Tcpdump is a tool that lets you dump network packets. This helps to debug networking issues, apis, communication or other stuff.

Basic options

Tcpdump basic options are

• -i ‘interface’ option lets you specify on which interface you will listen
• -nS lets you see basic information about packets
• -v, -vv, -vvv verbose mode
• -s 1514 lets you specify how much data from packet is displayed. In this case you see whole packet
• src, dst listening on specific things for source or destination
• net eg. 192.168.0.1/24 listening on all stuff in some network.
• port eg. port 3000 lets you listen on port

First example, getting info

First thing that people do often is to listen to everything that bounces en1 like this:

1

sudo tcpdump -nS -i en1

This is obviously bad idea, only good thing about its that i lets you see that “something is on” so you will be able to say that this device is actually working.

Example two, targeting host!

If you want to see all traffic that goes to some host, so something that is useful you should add host option.

1

sudo tcpdump -nS -i en1 host www.facebook.com

This will let you see if there are some packets going to and from <www.facebook.com>.

Example three, give me stuff targeting some port!

Lets say you want to see what generates curl to your own machine

1

sudo tcpdump -vvnS -i lo0 port 4000

and in other shell just type

1

curl http://localhost:4000

port is most fun option because it lets you see stuff that you are interested in.

Summary

Tcpdump is useful tool and i hope this text will let me not constantly forget its options.

Cheers!

Why ?

Recently in “professor Toons” :) computer club was a task that was ideal to solve as graph. It was one of this “find path” tasks. Yesterday just before going to bed I thought “Why not solve it in Erlang”. And this are my thoughts about erlang digraph: library.

Task

First of all http://erldocs.com/ is the way to go when working with documentation.

Second “the Task”. It had two parts, first was to load xml file with songs and second was to build graph and find playlist between two songs having in mind that for each song following has to start with the same letter that first ended.

example.

if you have song list: ABC CBA BAC ACB BBA

And you want to find playlist from ABC to BAC you will get ABC -> CBA -> ACB -> BAC. Easy

Digraph

Digraph API is quite nice but there are few things you have to have in mind.

• Digraph is imperative graph!
• You need to pass vertices to get_path!
• Graph with loads of edges can be quite big :D

To create empty graph you call :new/0

1

Graph = digraph:new().

To add node to graph you call :add_vertex/2

1
2

Data = {Value1, Value2, Value3},
digraph:add_vertex(Graph, Data).

To add edge to graph you call :add_edge/3

1

digraph:add_edge(Graph, VertexStart, VertexEnd).

To get all vertices from graph (random order) :vertices/1

1

Vertices = digraph:vertices(Graph).

And finally to get path you just call :get_path/3

1

Path = digraph:get_path(Graph, VertexStart, VertexEnd).

This path is a list of vertices in order. This API is just to remind me essential things about using the digraph from erlang stdlib. This graph is quite simple.

And for me for about 5000 vertices and a lot of edges it consumed 1.5 GB of ram. But when it was created it was super fast to use.

It was fun to play with.

Quick summary

I will make quick summary in form of short code sample!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

G = digraph:new(),
V1 = digraph:add_vertex(G, "Andrychow"),
V2 = digraph:add_vertex(G, "Krakow"),
V3 = digraph:add_vertex(G, "London"),
V4 = digraph:add_vertex(G, "Warsaw"),
V5 = digraph:add_vertex(G, "Paris"),

digraph:add_edge(G, V1, V2),
digraph:add_edge(G, V2, V4),
digraph:add_edge(G, V2, V5),
digraph:add_edge(G, V2, V3),
digraph:add_edge(G, V3, V4),
digraph:add_edge(G, V3, V2),
digraph:add_edge(G, V2, V1),

PathHome = digraph:get_path(G, V3, V1).

This gives you back ["London","Krakow","Andrychow"] so Win! we are at home!

Summary

It was fun to play a bit with digraph before going to bed. There is a tone of things i did not use and also look into digraph_utils for even more things :)

Many people use adds queue system to their products. Some of them do legendary things with it to make it extremely unreliable products :). Most of this solutions may seem trolling but they actually exists in some products.

Rolling out own queue system

First thing often people do is rolling out their own queue system. Is it bad ? no! it is great as long as you don’t have constraint that data can never be lost!

Can lose data in queue

If you use queue just to communicate between processes you can use something like unix name pipe. In reality this is just a file. Actually in Unix everything is a file and this is best ever design (if you neglect it you should die!).

1
2

mkfifo oldschool_pipe
gzip -9 -c < oldschool_pipe > out.gz &

And next you can use it to push stuff into it eg.

1

eacho "oldschoolllzzzz!!!!" > oldschool_pipe

but this is shell example, you could create it and just read/write it in your processes.

That is cool. And this is the last point where we will not see problems :)))

But we are programmers!

Yes we are programmers and most of us are young and full of energy nobody remembers 70’ i was born in 85 so i technically would be quite mad if i would remember 70’.

So how we approach problems so of us would create inside of their code queue.

In C it would be simply array wrapped with mutex’es but this is unreliable and its long to write and and and…

So what people do ? They try use READY products.

First big mistake

Use key-value store as queue. - Lets serialize array into XYZ and set it into key. - That’s good idea! Only one thing will write to it!

WRONG!

Such an assumption will provide you with insane amount of carnage in future. And even if i know “agile says XYZ now”… actually “agile” don’t say “take drugs an yolo because tomorrow you can die” but “TAKE RISKY THINGS FIRST” and this is risky thing. Should be implemented well.

What happens in this case ? Someone gets a great idea that product should scale adds another daemon and this f*cks up queue you lose messages.

Some dbms can handle this problem, but wrapping it into transaction will not solve the problem.

Scenario is: Process a)

• reads queue
• process is (makes a pop or push)
• saves the serialised queue

Now imagine process b) does the same. Everything is blazing fast and you get f*cked.

So DB system must know context. This is where RIAK shines, you get vector clocks and you know that you are f*cked. You can react but in 99% you don’t know how to resolve this issue but at least you would know… but some specialsits can disable this because handling vector clocks is a pain and you can get PERFORMANCE BOOST :))).

Redis list as queue

Redis is great tool to build a lot of stuff. And it has built in data structures. I think this is ground breaking because previous solutions like RDBMS most commonly use or other NoSQL solutions. Redis is great how to make queue within redis ?

1
2

lpush queue_name value -> push
rpop queue_name -> pop

example like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14

redis 127.0.0.1:6379> lpush queue 1
(integer) 1
redis 127.0.0.1:6379> lpush queue 2
(integer) 2
redis 127.0.0.1:6379> lpush queue 7
(integer) 3
redis 127.0.0.1:6379> rpop queue
"1"
redis 127.0.0.1:6379> rpop queue
"2"
redis 127.0.0.1:6379> rpop queue
"7"
redis 127.0.0.1:6379> rpop queue
(nil)

Cool ! Works great and any process can access it in atomic way isn’t it great ? best thing ever ?!

Actually it is very good. But there is one thing you just missed! Redis do a flush of keys every 60 sec if 10k keys did change by default. What does it mean ? You can get screwed if redis will instantly die!

How to fix this ? Visit http://redis.io/topics/persistence and see section “Append-only file”

1
2

- appendonly no
+ appendonly yes

Man you just did it, you lost some performance but you did it. Who would knew ? You just saved the world. How much we just cut performance ? “fsync every time a new command is appended to the AOF. Very very slow, very safe.” that is Ouch! Your boss could be unhappy even if this solution is actually the best, most simple and durable idea.

Can’t lose any data! Redis plus Mysql/Postgres

Persistence daemons worked out a new combo. You store each element of the queue like key-value store in SQL RDBMS and put its id on the queue in redis next you pop it up from the queue in redis process and updates its status in SQL RDBM. This is not so bad but it kills performance more than just turning on “appendonly yes”. Also it makes things hell more complicated and forces you to do updates in both system.

Is this system cure for cancer ? No! You have to have very good queue fail recovery / startup system. Simply empty list and make query

1

select * jobs where done = false

next you have to clean redis queue and push new data. Is this safe ? No you don’t know if few last jobs did finish or not. Eg. Mysql got f*cked but messages got processed. Yes this adds a lot more complications.

Also with this solution index on ID column makes its fast to make a select but slow to add or remove. And you want your queue to perform and yes mysql will do fsync.

Why not MongoDB

You can’t atomically pop stuff. Don’t think about pop/push/pushall on array in document! If you will have this idea check my gist https://gist.github.com/2071805 run it and see what happens :) what you get back.

RabbitMQ / ZeroMQ

When you will visit ZeroMQ page you will see

1
2
3
4
5
6
7
8
9
10

ØMQ \zeromq\:
 Ø  The socket library that acts as a concurrency framework.
 Ø  Faster than TCP, for clustered products and supercomputing.
 Ø  Carries messages across inproc, IPC, TCP, and multicast.
 Ø  Connect N-to-N via fanout, pubsub, pipeline, request-reply.
 Ø  Asynch I/O for scalable multicore message-passing apps.
 Ø  Large and active open source community.
 Ø  30+ languages including C, C++, Java, .NET, Python.
 Ø  Most OSes including Linux, Windows, OS X.
 Ø  LGPL free software with full commercial support from iMatix.

Nothing about consistency FASTERN THAN (this has to be good) TCP but can use TCP (i wonder if it can be faster than TCP even using TCP /trollface). Anyway you see a lot of stuff. I started some search on zeromq losing data and what i found http://zguide.zeromq.org/page:all#Missing-Message-Problem-Solver a nice image.

Problem resolution diagram

Big thing :)

If you will visit rabbitmq page http://www.rabbitmq.com/ you will see a lot of nice things like tutorial etc. Page is nice and has useful knowledge. Both solutions have client in Erlang (massive plus) and other languages. And even while setting up whole thing may be a pain i think this is a solid option both ZeromMQ and RabbitMQ.

Why do we use Queues ?

We use them to absorb traffic of messages and process their content by eg. workers / handlers etc. If we will make it unprocessable by more than one worker we ain’t doing our job properly.

What makes things hard.

• Locks if we use them, they will bite you back
• Many points where we store same data in different way
• Yes, locks will bite you back

So what is the best way to go ?

I think the best way to go is just to start a new movement called Unix Archeology because we seems to be reinventing the wheel too many times. But really

• Make a list of solution
• Ask your self if your idea is really good

I’m 100% sure that storing queues as serialized lists in memcached or keeping them as table in mysql/postgres and making loads of funky stuff to keep it running is not the way to go. It can seem like a good idea at start but it is not. Named pipe in file system can be better.

Loads of things can be brilliant queue choices eg. Redis, ZeroMQ, RabbitMQ or even named pipes but not serialized array in key-value store.

@antirez: “The Redis community is composed of 99% of people that, really, know their stuff, know the Redis internals and behaviour, and are * great *.”

@shanley: “@antirez I’ve never met a technical community where 99% of them were familiar with the internals of anything. Did you mean 9%?”

This sparked in my mind very quick review of the topics that we talk about in work and i realised that we talk about a lot about internals of redis and a bit about riak but this is different story :).

Spike

I just wanted to write a short post about first thing i ever picked when i was looking into internals of Redis. It is List :D i love lists.

So what i did was opening again github and picking up list header file to re-read.

https://github.com/antirez/redis/blob/d310fbedabd3101505b694f5c25a2e48480a3c2b/src/adlist.h

First thing that you notice is that code is simple and whole thing is implemented in 93 lines of header and 341 lines .c file. (with license etc lol).

Structure of List

In general List is just degenerated Tree. In Redis structure of it is simple. Whole description of the list is simply

1
2
3
4
5
6
7
8
9
10
11
12
13
14

typedef struct listNode {
    struct listNode *prev;
    struct listNode *next;
    void *value;
} listNode;

typedef struct list {
    listNode *head;
    listNode *tail;
    void *(*dup)(void *ptr);
    void (*free)(void *ptr);
    int (*match)(void *ptr, void *key);
    unsigned long len;
} list;

this knowledge lets us count how much space this will take on the heap and compare it with eg. set if we really need to. (list is imho most memory effective structure)

List iterator is important so its also worth having a peek at even if this is just internal implementation.

1
2
3
4

typedef struct listIter {
    listNode *next;
    int direction;
} listIter;

with this we can take a peek into .c file and check how you get iterator.

1
2
3
4
5
6
7
8
9
10
11

listIter *listGetIterator(list *list, int direction){
    listIter *iter;

    if ((iter = zmalloc(sizeof(*iter))) == NULL) return NULL;
    if (direction == AL_START_HEAD)
        iter->next = list->head;
    else
        iter->next = list->tail;
    iter->direction = direction;
    return iter;
}

And see that even if AL_START_HEAD is defined as 0 and AL_START_TAIL as 1 if we will use direction of 5 (lol) we will get tail :D I know that i’m bikesheding now.

Even without going any deeper you have a feeling how this works. Double linked list with (void) value. First thing i thought today was (this was stupid) “Wow why this is (void ) and not (char ) this would let compiler better type check it while compilation” but antirez wrote to me “@jakuboboza hint: grep listCreate .c” and that was the hint i needed. (void *) is more generic but list is used in many places in redis internals and i did not thought about it (lol)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

λ grep listCreate *.c
adlist.c:list *listCreate(void)
adlist.c:    if ((copy = listCreate()) == NULL)
aof.c:    server.aof_rewrite_buf_blocks = listCreate();
aof.c:    c->reply = listCreate();
aof.c:    c->watched_keys = listCreate();
bio.c:        bio_jobs[j] = listCreate();
multi.c:        clients = listCreate();
networking.c:    c->reply = listCreate();
networking.c:    c->io_keys = listCreate();
networking.c:    c->watched_keys = listCreate();
networking.c:    c->pubsub_patterns = listCreate();
object.c:    list *l = listCreate();
pubsub.c:            clients = listCreate();
redis-benchmark.c:    config.clients = listCreate();
redis.c:    server.clients = listCreate();
redis.c:    server.clients_to_close = listCreate();
redis.c:    server.slaves = listCreate();
redis.c:    server.monitors = listCreate();
redis.c:    server.unblocked_clients = listCreate();
redis.c:    server.ready_keys = listCreate();
redis.c:    server.pubsub_patterns = listCreate();
sentinel.c:    sentinel.scripts_queue = listCreate();
slowlog.c:    server.slowlog = listCreate();
sort.c:    operations = listCreate();
t_list.c:        list *l = listCreate();
t_list.c:            l = listCreate();
t_list.c:        server.ready_keys = listCreate();
ziplist.c:            ref = listCreate();

A lot of places :) lol.

In adlist.c we can also check how the list is created

1
2
3
4
5
6
7
8
9
10
11
12
13

list *listCreate(void)
{
    struct list *list;

    if ((list = zmalloc(sizeof(*list))) == NULL)
        return NULL;
    list->head = list->tail = NULL;
    list->len = 0;
    list->dup = NULL;
    list->free = NULL;
    list->match = NULL;
    return list;
}

This is just academic example :D I love it. This code is easy to understand and just pleasure to read.

why to even bother talking about internals ?

It is important to talk about them because if you read in documentation that

1
2
3

lpush is O(1)
rpush is O(1)
lpop is O(1)

you really want to check this out to get better understanding how stuff works under the hood. Even if this is trivial example.

It is worth talking about internals of tools that you use, you learn a lot and i think its truth what antirez said, this community is great!

Next

Thing to view is suggested by antires Dict!

antirez: “@jakuboboza it’s definitely a very simple implementation! Probably our most “on steroids” implementation of data structures is dict.c”

^_____^

I have in pipeline a lot texts about lua scripting in redis and using it to build some tools but can’t find time to finish this stuff ;/.

Auto failover

Every database wants to have auto failover mechanism. This is a great marketing pitch! haha :) Main thing about is that one of your server can go down and you still are operating as normal and when he will go up again everything is fine.. unless your routing server will go down :D ofc.

2.4.16 / 2.6

Since Redis 2.4 and 2.6 there was this idea of adding it. Antirez wrote a draft spec for it and implemented it as experimental feature. It is really well described here http://redis.io/topics/sentinel so i will just write a short note how did i setup it and how does it feel.

Setup

While preparing to this demo i did everything on master 0ee3f05518e081640c1c6f9ae52c3a414f0feaceso what i did was simply start “master” and “replica servers” with this configs (ofc turn daemonize to yes in production lol)

Standard Master setup with default script on port 6379 and replica with

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

daemonize no
timeout 0
loglevel notice
logfile stdout
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./replica_dir
slave-serve-stale-data yes
slave-read-only yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000

pidfile /var/run/redis-replica-7789.pid
port 7789
# replication config
slaveof 127.0.0.1 6379

slowlog-log-slower-than 10000
slowlog-max-len 1024
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

So i had master and slave running :) that was cool next thing i did was! configure and turn on the sentinel!

1
2
3
4
5
6
7
8
9
10
11

sentinel monitor mymaster 127.0.0.1 6379 1
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 900000
sentinel can-failover mymaster yes
sentinel parallel-syncs mymaster 1

sentinel monitor resque 127.0.0.1 7789 1
sentinel down-after-milliseconds resque 10000
sentinel failover-timeout resque 900000
sentinel can-failover resque yes
sentinel parallel-syncs resque 5

And i turned him on! with

1

redis-server sentinel-my.conf --sentinel

And stuff started to work :D

Carnage!!!

So i started easy

1
2
3
4

redis 127.0.0.1:6379> get "hello"
"hello"
redis 127.0.0.1:6379> set "hello" "lulz"
OK

Works! I killed master and connected on 26379 to sentinel master did query

1

 sentinel masters

and got

1
2
3
4
5
6
7

1)  1) "name"
    2) "resque"
    3) "ip"
    4) "127.0.0.1"
    5) "port"
    6) "7789"
    ...

Cool works great :D The only thing that worried me was that when i turned on master after failover (it took 8 sec) he did not pickup he is slave and he did not start replicating data.

when you do this…

You will see beefy

1
2
3
4
5

[36373] 26 Sep 21:15:03.441 # Error condition on socket for SYNC: Connection refused
[36373] 26 Sep 21:15:04.521 * Connecting to MASTER...
[36373] 26 Sep 21:15:04.521 * MASTER <-> SLAVE sync started
[36373] 26 Sep 21:15:04.521 # Error condition on socket for SYNC: Connection refused
[36373] 26 Sep 21:15:05.128 * MASTER MODE enabled (user request)

On the initial slave :) things just went from bad to good :D

Summary

This is cool new feature that you can have master-slave and auto failover server the only thing that driver have to do is if you get error while connecting / querying is to ask sentinel for new master connect and retry :) It is very basic but…

I like it!

THIS IS EXPERIMENTAL FEATURE and much more info about it you can finde here http://redis.io/topics/sentinel. Especially about pub/sub way of watching stuff / events while they occur.

Quick review

I used similar approach to store followers and following so i will just go fast through the keys and design.

1
2

  twtr:<user_id>:follows -> set of ids this user follows
  twtr:<user_id>:followers -> set of id's  that follows this user

What happens when i click “follow”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

redis 127.0.0.1:6379> SADD twtr:kuba:following amelia
(integer) 1
redis 127.0.0.1:6379> SADD twtr:amelia:followers kuba
(integer) 1
redis 127.0.0.1:6379> SADD twtr:kuba:following dan
(integer) 1
redis 127.0.0.1:6379> SADD twtr:dan:followers kuba
(integer) 1
redis 127.0.0.1:6379> SADD twtr:kuba:following ben
(integer) 1
redis 127.0.0.1:6379> SADD twtr:ben:followers kuba
(integer) 1
redis 127.0.0.1:6379> SMEMBERS twtr:kuba:following
1) "ben"
2) "amelia"
3) "dan"

So now i follow amelia, ben and dan.

What happens when amelia click followers!

1
2

redis 127.0.0.1:6379> SMEMBERS twtr:amelia:followers
1) "kuba"

This way for each user we can see set of people who follow him and those who he follows. Thats all we need like in tutorial.

Post

So i think it is not waste if we will decide to keep post in form of 2 keys

1
2
3

  twtr:<user_id>:post:<post_id> -> content text of post
  twtr:<user_id>:post:<post_id>:created_at -> creation time of post
  twtr:post_owner:<post_id> -> id of post creator.

Why this approach and not compacting all the things into pipe separate key? Both solutions seems to be ok, this just leaves little bit more flexibility. I know that it will generate 2 x times more pickups to redis then previous so you can consider doing

1

  twtr:<user_id>:post:<post_id> -> (timestamp|text)

Both solutions have a pros and cones, first one will require 2 x lookups and second one will require parsing data in app layer. Still i prefer first one.

Post id

This is hard topic, because in future we would want to scale ( lol ) . Generating post id is not easy task in this case. We could just use auto incrementing counter like this

1

  twtr:post_id:next -> auto incr

But to have something more flexible you should look at something like snowflake http://engineering.twitter.com/2010/06/announcing-snowflake.html.

Post list

For each user we will store a list of posts he wrote. Initially i thought that we could just pump everything into list. But this is not optimal. This is single point which will grow like crazy and we will be not able to decide how and when to archive parts that are not used. Eg. posts did by user 8 months ago are not really relevant today because if we will make assumption that on average person posts few times a week this 8 month old entry will be way forgotten. We want to archive it, also it will be healthier for memory to store short lists.

I see here two scenarios:

• user looks at his last few posts < 100
• user is infinite scrolling through all posts.

So this scenario reasonable seems to have list of lists in which we will have ordered post lists id’s. if we will use only LPUSH to add posts lists tot his list we will be able to to do easy LRANGE 0 to get newest lists.

1
2
3

  twtr:<user_id>:lists -> list of lits id's only LPUSH id and LRANGE 0 number.
  twtr:<user_id>:list_next -> auto incr counter for lists id's
  twtr:<user_id>:list:<list_id> -> list with 100 posts

So how do we get most recent posts ? we just LRANGE 0 2 to get most recent two lists and next we will merge them first + second. Both are LPUSH’ed so should be semi ordered. (we don’t really care about order). adding stuff to time line is bit tricky.

we need to do it like this LRANGE <key> 0 1 current list id, next we need to LLEN <key> to check size and if it is < SIZE ( for our example 100 ) we just LPUSH <key> <value> and job done, if size is > 100 we need to INCR <list counter> and LPUSH its result on list of lists and next we need to LPUSH <key> <value> on the new list.

And all of this in application layer. But this is the hardest bit to do. May seem to be complicated but if this seems to be not optimal you can add one more list

1

  twtr:<user_id>:list:current -> list of current 100 posts

This list is just the most current posts of particular user. How does this list work ? Algorithm is simple

• LPUSH new post id’s
• RPOP if SIZE > 100

This could be useful to reduce number of hits you get against redis.

Time line

Now the time line. Time line is exactly the same as user post list. we will just one “bit” about adding posts.

Algorithm here is when you add a post you have to pick all id’s of people who follow you. (example from top if you are adding post as amelia)

1

SMEMBERS twtr:<user_id>:followers

And you need to push your post id into their time line posts list. Thats all. Ofc one thing that we need to add are keys for time line

1
2
3
4

  twtr:<user_id>:timeline:lists -> list of lits id's only LPUSH id and LRANGE 0 number.
  twtr:<user_id>:timeline:list_next -> auto incr counter for lists id's
  twtr:<user_id>:timeline:list:<list_id> -> list with 100 posts
  twtr:<user_id>:timeline:current -> LPUSH, RPOP > SIZE current list cache

Summary

This is how i would approach building twitter like clone. Things like old lists can be easily archived into mysql, postgres or other thing and EXPIREed from redis. One thing in my design is common that i put a lot into keys this could be skiped but in my opinion it is not bad. IF you will use in form of user email md5 you can use it directly to access gravatar of that user.

On average you will need to do around 10-30 hits into redis to get data if you plan to do it in a “lazy way” you can minimize number of hits to around 10.

If you see problem with my design comment i want to know it!. The core of this design is that each user post data is stored into one redis instance. This is important because of access and race condition stories if you will have many redis instances. But achieving “sharding” in application layer is not hard. Only thing that i would care about is post id generator. This is single point of failure because i have a strong assertion that post_id is unique in whole system.

Cheers!

Upgrading octopress is a @!#!@pain! :>

What is this post about :)

Long time nothing new here so i will glue something together about stuff that we were talking about today with my friend Jarek. We talked about building backend for Todo app :). Yes simple todo app. How to build scalable backend. So my initial thought was “how i would design it in different databases”. (i’m taking only about data model)

Requirements

What we know:

• User has some sort of id. (number, email, hash of something)
• We need to be able to have different todo lists
• User can choose his todo list and see tasks ( obvious )
• User can tag tasks!
• User can query tasks in list by tags
• User can see all tags.

Design using Redis

How to do it with redis ? :)

First few facts i assumed at start. Single todo task has body and timestamps [created_at, updated_at] and base for key will be phrase “todoapp”.

So lets start with user and his list of todo lists :). This gives us first key

1
2
3

todo:<user_id>:todolist:next => auto incrementing counter for lists id
todo:<user_id>:todolists => [LIST]
todo:<user_id>:todolist:<todo_list_id>:name => list name

Here we have two keys, first is list id counter that we will bump to get new list counter :), second is list of todolists ids. Why do it this way ? Well people can add and remove todo lists.

Ok so how to create new list ?

1
2
3
4
5
6
7
8

redis 127.0.0.1:6379> INCR todo:kuba:todolist:next
(integer) 1
redis 127.0.0.1:6379> RPUSH todo:kuba:todolists 1
(integer) 1
redis 127.0.0.1:6379> LRANGE todo:kuba:todolists 0 -1
1) "1"
redis 127.0.0.1:6379> SET todo:kuba:todolist:1:name "things to do"
OK

Hey ! we just added id of our first list to list of our todo lists (lots of list word here!). Ok so now lets add a task.

list:

1
2

todo:<user_id>:todolist:<todo_list_id>:next => auto incrementing counter for tasks id
todo:<user_id>:todolist:<todo_list_id> => [LIST]

and task:

1
2
3

todo:<user_id>:task:<task_id> => content of task eg. "finish blog post"
todo:<user_id>:task:<task_id>:created_at => epoch time when it was created handled by app
todo:<user_id>:task:<task_id>:updated_at => epoch time when it was last updated handled by app

Ok so how to i add task to my list

1
2
3
4
5
6
7
8
9
10

redis 127.0.0.1:6379> INCR todo:kuba:todolist:1:next
(integer) 1
redis 127.0.0.1:6379> LPUSH todo:kuba:todolist:1 1
(integer) 1
redis 127.0.0.1:6379> SET todo:kuba:task:1 "finish blog post"
OK
redis 127.0.0.1:6379> SET todo:kuba:task:1:created_at  1343324314
OK
redis 127.0.0.1:6379> SET todo:kuba:task:1:updated_at  1343324315
OK

And we have our first task in. How do we get tasks from out todo list simple!

1
2
3
4
5
6

redis 127.0.0.1:6379> LRANGE todo:kuba:todolist:1 0 -1
1) "1"
redis 127.0.0.1:6379> GET todo:kuba:task:1
"finish blog post"
redis 127.0.0.1:6379> GET todo:kuba:task:1:created_at
"1343324314"

Ok so now we have very simple todo lists with tasks, well at least overview. Ofc you can use sets for todo lists or zsets but lets stay with lists to keep it simple for now.

How ro remove task from the list ?

1
2
3
4

redis 127.0.0.1:6379> LREM todo:kuba:todolist:1 -1 1
(integer) 1
redis 127.0.0.1:6379> LRANGE todo:kuba:todolist:1 0 -1
(empty list or set)

Good, now we can add tasks, remove tasks, same sotry with adding todo lists and removing todo lists. One last thing is to add tags!. Simply here each task will have list of tags and each tag will have list of tasks related with.

1
2

todo:<user_id>:task:<task_id>:tags => [LIST]
todo:<user_id>:tag:<tag>:tasks => [LIST]

So how we will add tags to tasks ? Simple!

1
2
3
4
5
6
7
8
9
10
11
12
13

redis 127.0.0.1:6379> LPUSH todo:kuba:task:1:tags "redis"
(integer) 1
redis 127.0.0.1:6379> LPUSH todo:kuba:task:1:tags "design"
(integer) 2
redis 127.0.0.1:6379> LPUSH todo:kuba:tag:redis:tasks 1
(integer) 1
redis 127.0.0.1:6379> LPUSH todo:kuba:tag:design:tasks 1
(integer) 1
redis 127.0.0.1:6379> LRANGE todo:kuba:tag:design:tasks 0 -1
1) "1"
redis 127.0.0.1:6379> LRANGE todo:kuba:task:1:tags 0 -1
1) "design"
2) "redis"

This example shows what we need to do to tag a task with something and how to peek tasks tagged with it. Why we have both lists ? To make it fast while searching. If user will click on particular tag like “redis” you want to get it O(1) time not O(N) after searching all keys. And same the other way, normal ui will pull task test, when it was created and tags to display so we want to have this data ready.

This is whole design for the todo app. We have 8 types of keys. Things like pagination, calculating time are all left to app layer. Important thing is that i scope everything to user key / id. This is because i want to isolate each user space easy. Each user in his own space will have short lists, there is no danger of “ultimate” non splittable lists.

Design using Mongodb

Well this case upfront is easier to grasp because for each list we can use single document or collection of documents lets talk about both solutions.

Todolist = Document

In this example we will use built in “array” operators

1
2
3
4
5
6
7

> db.todolists.save({name: "House work", tasks: []})
"ok"
> db.todolists.find({name: "House work"})
[
  {   "name" : "House work",   "_id" : {   "$oid" : "50118742cc93742e0d0b6f7c"   },
      "tasks" : [   ]   }
]

So lets add a task :)

1
2
3
4
5
6
7
8
9

db.todolists.update({name: "House work"},{$push:{"tasks":{"name":"finish blog post", "tags":["mongo"] } } })
"ok"
> > db.todolists.find({name: "House work"})
[
  {   "name" : "House work",   "_id" : {   "$oid" : "50118742cc93742e0d0b6f7c"   },
      "tasks" : [
       {   "name" : "finish blog post",   "tags" : [   "mongo" ]   }
      ]   }
]

this will create empty todo list with name “House work” of course this way we will not omit building sub lists of tags etc, we have to build in a same way like in redis but as part of document. The story is exactly the same like above in redis. Mongodb lets us query nested documents and this will enable us to skip some of the extra “lists” while doing search.

Lets try it out how to find out mongo tagged entries?

1
2
3
4
5
6
7
8

> db.todolists.find({"tasks.tags": { $in : ["mongo"] } })

[
  {   "name" : "House work",   "_id" : {   "$oid" : "50118742cc93742e0d0b6f7c"   },
      "tasks" : [
        {   "name" : "finish blog post",   "tags" : [   "mongo" ]   }
      ]   }
]

This way we can find whole todolist with task that contains tag “mongo” but after that we will have to work out from the document in app the task that we are interested int. Using it like this we will have a document with structure like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14

{
 user: "user id",
 name: "<name>",
 tasks: [
   {
    text : "todo text",
    tags : [
      "Tag1", "Tag2"
    ]
    created_at : Time,
    updated_at : Time
   }
 ]
}

Using redis we could wrapp stuff into MULTI command while using mongodb “array” command we are a bit cowboishing. They could remove wrong stuff in we will not be cautious :). (well same in redis!) Big plus of Mongodb is native time type!

Todolist = many documents

Using this approach we can leverage more of our stuff on mongodb search in this approach each task will be a different document. With structure like this

1
2
3
4
5
6

{
  todo_list: "todo list id",
  user: "user id",
  text: "todo text",
  tags: ["Tag1", "Tag2"]
}

This way we will have a lot of documents, more disk space consumption and still we will have to have second collection with with objects with structure like this

1
2
3
4
5
6

{
  todo_list: "todo list id",
  name: "todo list name",
  user: "user id",
  // tasks: [Tasks OBjectID Array] you could have this and remove todo_list id from tasks choice is yours :)
}

And this way we can use find tool very easy and get documents fast.

Summary

All of this solutions have some pros and cons, mongodb excels better when documents are bigger (limit is set on 16 mb per document) than loads of small documents (massive waste of space). Solution in redis is really fast and if you will implement lazy loading it will be very fast. You can adjust this designs to your situation by changing lists to sets etc. The place where redis OWNS mongodb in this context is “strucutres” and we use a lot of them to store data like this, lists sets, zsets. Implementing priority list in mongodb will be totally custom solution while in redis we can just use zset.

This is just my point of view on this. I will supply some code to cover it more in part two. This is next problem, i’m sure solution in mongodb using things like mongoid http://mongoid.org will be much more developer friendly then building things “rawly” in redis hiredis client.

btw i jsut wrote this from “top of my head” so it may contain typos and i’m sure keys, structures can be optimized :) This is just to open discussion with my friend :)

Cheers!

Example controller

How we use this ? Lets take a peek and simple example:

1
2
3
4
5
6
7
8
9

class VenuesController < ApplicationController

  respond_to :html, :xml, :json

  def index
    respond_with(@venues = Venue.all.page(params[:page]).per(20))
  end

end

Two things we will notice at start is respond_to :html, :xml, :json this is something like old merbs provides where we specify to which formats we want to respond. Second change is how we layout the action. All we have to do is to put into respond_with object we want to respond with.

What this buys for us ?

• If we have html request to actions like create, update or destroy we want to redirect on success
• if we have json, xml request to same type of “state changing” actions we want to render response with our format.

We achieve both of this with reponds_with in just one line. But lets take a peek at longer example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

class VenuesController < ApplicationController

  before_filter :authenticate_user!, :except => [:index, :show, :near]

  respond_to :html, :xml, :json

  def index
    respond_with(@venues = Venue.all.page(params[:page]).per(20))
  end

  def show
    respond_with(@venue = Venue.find_by(permalink: params[:id]))
  end

  def new
    repond_with(@venue = Venue.new)
  end

  def create
    @venue = Venue.create(params[:venue])
    respond_with(@venue, :location => venues_url)
  end


end

In this example we can see example of create action where we add :location => venues_url this will in case of format: html success redirect to this url.

Summary

Using this helps to write stuff fast and readable, you can still use plain old respond_to in action with old format.html syntax..

I hope you liked it :)

From this gist we can see that his vibe.d benchmark set pong test at 8425.85 req/sec (He used ab and i used httperf).

Warp

My first candidate in this competition of pong tests is haskell warp handler.

code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{-# LANGUAGE OverloadedStrings #-}

import Network.Wai
import Network.Wai.Handler.Warp
import Network.HTTP.Types (status200)
import Blaze.ByteString.Builder (copyByteString)
import qualified Data.ByteString.UTF8 as BU
import Data.Monoid
import Data.Enumerator (run_, enumList, ($$))

main = do
    let port = 8000
    putStrLn $ "Listening on port " ++ show port
    run port app

app req = return $
    case pathInfo req of
        ["pong"] -> pong
        x -> index x

pong = ResponseBuilder status200 [ ("Content-Type", "text/plain") ] $ mconcat $ map copyByteString
    [ "pong" ]

index x = ResponseBuilder status200 [("Content-Type", "text/html")] $ mconcat $ map copyByteString
    [ "<p>Hello from ", BU.fromString $ show x, "!</p>"
    , "<p><a href='/pong'>pong</a></p>\n" ]

it is semi rack like syntax ;)

Benchmark results

I prepared results to show how big lie is pong test. Because in this type of test / showoff all you test is how fast really you can accept connections. Single thread will always win :). But lets look at results:

• Multi threaded - 50 threads - Request rate: 858.8 req/s (1.2 ms/req) BOOO!!!!
• Multi threaded - 4 threads ( on for each core ) - Request rate: 10020.8 req/s (0.1 ms/req) Vibe.d die!!! Yeah!!!!
• Single threaded - default compilation - Request rate: 13584.1 req/s (0.1 ms/req) Mother of God!

Tested with httperf --uri=/ --port=8000 --num-calls=10000 --num-conns=20 httperf command.

Summary

You can test it on your own, i did it on latest 7.4.1 Ghc from haskell platform on OSX 10.7. And post reply with your results :) maybe i missed something. Code, scripts to build and run are in repository https://github.com/JakubOboza/haskell-warp-pong-test.

So how to test ?!

I think you should test your application in default environment so with db behind it, but anyone in this scenario can say it is testing performance of db. But everyday… users are really testing performance of our db ;>… or the weakest of the elements in chain. So if your db / rendering engine is performing at level of 50 req / sec fast app handler will not turn it into 5000 req / sec.

Custom template ?

I always end up looking at old projects and copying parts like gen_server and reusing them. I always knew rebar has option to write them but never had time to look at it. Today lol, i wanted to do some cleaning at home so every thing seems to be a good excuse to not do any cleaning :D.

What i need to know

Basic template is made from one or many .erl files written with mustache style { { } } code and .template file describing what to do with files.

Let’s build gen_server template

So my first file will be gen_server.erl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

%%% @author  { {author_name } } <{ {author_email} }>
%%% @copyright  { {copyright_year } }  { {author_name} }.
%%% @doc  { {description } }

-module( { {name } }).
-behaviour(gen_server).

-author(' { {author_name } } <{ {author_email } }>').

-export([start_link/1]).
-export([init/1, handle_call/3, handle_cast/2, terminate/2, handle_info/2, code_change/3, stop/1]).

% public api

start_link(_Args) ->
  gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

% state should be change with State that you will pass
init([]) ->
  {ok, state}.

stop(_Pid) ->
  stop().

stop() ->
  gen_server:cast(?MODULE, stop).

handle_call({method_name_and_params}, _From, State) ->
  Response = ok,
  {reply, Response, State};

handle_call(_Message, _From, State) ->
  {reply, error, State}.

handle_cast(_Message, State) -> {noreply, State}.
handle_info(_Message, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.
code_change(_OldVersion, State, _Extra) -> {ok, State}.

This is the template, i know a bit long i tried to cut out all comments, euint etc and narrow it to minimum. I posted it to show how much you can save :). Now you will need the transformation file. All this things in { { something } } will be replaced by things we will type in command line or defaults from template file.

1
2
3
4
5
6
7

{variables, [
  {name, "template"},
  {copyright_year, "2012"},
  {author_name, "jakub Oboza"},
  {author_email, "jakub.oboza@gmail.com"}
]}.
{template, "gen_server.erl", "src/{ {name} }.erl"}.

For me it looks like this, we have few default definitions and at the bottom. template! This is important part it says which file he has to copy where and what will be the name of new file. Now everything should be clear!

Injet it into rebar!

All you need now to do is symlink your template folders to ~/.rebar/templates and you can use them! (you can symlink your folder or just create one there :) )

Use!

1
2
3

λ rebar create template=gen_server name=example
==> tmp (create)
Writing src/example.erl

Viola!

important

When i was looking at this post i saw that {{ is converted in a wrong way by octopress so i added spaces between them! check repo for correct code!

My own templates

Today i started adding my own templates initially i have only gen_server and webmachine_resource but i will add more :). It is fun it is like building your own anti-boiler plate framework.

My templates repo: https://github.com/JakubOboza/rebar-templates

Hope this helps! Cheers!

First thing to achieve was to build a tool that will be able to work with logs in a way that will not limit it usage to small files. This is simple, i decided to build a lib that will enable to stream process logs by “entry”. Some of the

Initial idea

We will emit single “entry” so for logs that are build around “lines” we will emit line and for logs like rails we will emit whole entry. Line streaming is easy so few hours ago during watching 3 season of http://en.wikipedia.org/wiki/Metalocalypse i wrote this.

Future toons

Version 0.0.1 https://github.com/JakubOboza/future_toons This is my entry point to analytics on big files :). First thing was a benchmark. I said to my self that it can’t be more then 0.5sec on my box (i5 MBP) vs 100mb log file from nginx. I took a log from production and checked my initial code. It was 0.37 sec. That’s ok.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

function FutureToons(filename, callback, end_callback){
  // in case someone will call it wrong way ;)
  if(false === (this instanceof FutureToons)) {
    return new FutureToons(filename, callback);
  }
  events.EventEmitter.call(this);

  this.onEnd(end_callback);

  if(filename && callback){
    // if both present run instant
    this.onLine(callback);
    this.run(filename);
  }

}

// do not put methods between this line and initial definition
util.inherits(FutureToons, events.EventEmitter);

Code of whole thing is very simple. But while building node.js module there are few things that are useful. First of all you don’t need to implement your own way of inheritance you can use the one from utils. (Remember to put it just after function definition) because it overrides prototype :) you don’t wanna lose your “instance methods” don’t you.

Next nice thing to help users is check of instanceof just at top of “constructor” function. This prevents users from using it wrong. Well its easier to say it enables them to use it wrong and fixes their mistake.

1
2

var toons = require('future_toons');
new toons("example.js", function(line){});

and

1
2

var toons = require('future_toons');
toons("example.js", function(line){});

This will produce same output and act in same way.

How to use this ?

It is simple :) You need to know three things, where the file is, what do you want to do with each line and do you need to do something at the end.

basic example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

var toons = require('future_toons');

var on_each_line_callback = function(line){
  console.log("> " + line);  
};

new toons("example.txt", on_each_line_callback);
// this will auto trigger run and process it but you can delay it like this

var streamer = new toons();
streamer.onLine(on_each_line_callback);
// you can add function on end! also :)
streamer.onEnd(function(){console.log("lol")});
// run it naow!
streamer.run("example.txt");
// you can reuse it for many files if you want ;p

This example will show each line of the file prefixed with “>” symbol and at the end he will print out “lol”. This is most common case in real world :) you need to optimize “lol”. And for now thats the whole api.

Command line interface

Currently i added simple command line interface just to play with it. example usage:

1
2

λ time node bin/toons -f ~/Downloads/access.log -e "function(line){}" node bin/toons -f ~/Downloads/access.log -e "function(line){}"
0.38s user 0.07s system 101% cpu 0.440 total

It is a bit unsafe now so maybe i will remove it soonish.

Db vs File

Some people say they need to put logs into db, i always ask this people “why not just file? this db will have to write it to this file anyway :)”

Summary

if you will get email from new relic sales guy, ignore! don’t reply ever!!! Haha while writing this post i got new email from their sales. Code is on github i hope i will have some time to work on it and it will have rails production.log streaming support soonish.

Cheers!

Setup!

To setup replica node all you have to do is to add one line slaveOf in config @_@ of your new Redis instance. Sounds easy :). Lets think about most basic scenario.

Two nodes, master node and slave node. For purpose of this example you can just start redis using redis-server command without by default he will start on port 6379 and this is all we need to know to setup replication.

Configuration of replica node

To configure replica node all we need to do is to create place to store the db eg. mkdir replica_db and choose port eg. 7789. Last thing to do is to create config and point this node to the master. For me it looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

daemonize no
timeout 0
loglevel notice
logfile stdout
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./replica_dir
slave-serve-stale-data yes
slave-read-only yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000

pidfile /var/run/redis-replica-7789.pid
port 7789
# replication config
slaveof 127.0.0.1 6379

slowlog-log-slower-than 10000
slowlog-max-len 1024
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

Here the important thing really is slaveof 127.0.0.1 6379 where we set where is our master. port 7789 important if we are using few redis instances on one box and dir ./replica_dir be sure to not point this to master node db path if you do… you will suffer eternal flame.

Now just start the node redis-server redis-replica.conf and he will start syncing.

Checking if everything works

So by now we should have master running on default port and replica connected to it. Lets connect using redis-cli to master and set some keys eg. set name kuba. Now lets connect to our replica. If you followed the same configuration then me you can simply do ./redis-cli -p 7789 this will prompt you with regular command line interface. Now jsut type get name

1
2

redis 127.0.0.1:7789> get name
"jakub"

Bang works!

RO

Important information is that one master can have many replicas and each replica is read only! So you can connect to it and read from it if you want / need.

1
2
3
4
5
6
7
8
9

 λ ./redis-cli -p 7789
redis 127.0.0.1:7789> keys *
 1) "age"
 2) "name"
redis 127.0.0.1:7789> get name
"jakub"
redis 127.0.0.1:7789> set name "not jakub"
(error) READONLY You can't write against a read only slave.
redis 127.0.0.1:7789>

Summary

I never had deadly important data in redis :) But still its worth knowing how to setup this just in case something goes wrong you may want to have replica ready :).

On official site http://redis.io/topics/replication you can learn more about replication in redis.

Cheers!

npm

Npm stands for node package manager and it is something like gems in ruby, eggs in python or apt in debian. It lets you search, install and update your node or application modules. You can create node.js modules without npm but if you want to publish your module its better to do it this way. If you have node.js >0.6.10 you should have npm bundled with your instalation if not go to http://npmjs.org/ and follow the instructions. (for normal platforms all you have to do i run http://npmjs.org/install.sh)

scaffold

First thing to do is to initialize our module, to do this we need to create a directory for it and run npm init like this.

1
2

λ mkdir my_first_module && cd my_first_module
λ npm init

Next he will ask us few questions, we should answer ;). example output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

Package name: (my_first_module)
Description: my first module
Package version: (0.0.0)
Project homepage: (none) no-fucking-idea.com
Project git repository: (none)
Author name: Jakub Oboza
Author email: (none) jakub.oboza@gmail.com
Author url: (none) no-fucking-idea.com
Main module/entry point: (none)
Test command: (none) mocha -R landing lib/my_first_module.js
About to write to /private/tmp/my_first_module/package.json

{
  "author": "Jakub Oboza <jakub.oboza@gmail.com> (no-fucking-idea.com)",
  "name": "my_first_module",
  "description": "my first module",
  "version": "0.0.0",
  "homepage": "no-fucking-idea.com",
  "scripts": {
    "test": "mocha -R landing lib/my_first_module.js"
  },
  "dependencies": {},
  "devDependencies": {},
  "optionalDependencies": {},
  "engines": {
    "node": "*"
  }
}



Is this ok? (yes) yes

This will create for us package.json. Now we have fully functional module and we could stop now…. but

package.json

This file is description of our package. It is in form of json so it should be easy to read and change. Most of this fields don’t need a lot of description because keys are self explaining but things that we should look at is "scripts" where we defined "test" key. If we will now run npm test he will execute this command that is very useful.

Engines defined on what version of node.js our code will work leaving it to * is a bit hazard. You can set it to something specific if you want.

Important thing! If we will not specify entry point to our module by default it will be looking for index.js so for now lets leave it this way.

tests with mocha

If we want to write reasonable code that we want to rely on we should be doing massive testing. I think mocha is a very good library for this purpose! I strongly suggest installing it with flag -g so it will be accessible in global scope of npm

1

λ npm install -g mocha

Code

Ok but we don’t have any code yet ;/. Yes lets start coding.

1
2
3

λ mkdir lib
λ touch index.js
λ touch lib/my_first_module.js

Ok so we created lib directory with out module code, empty module file and index file. Now lets run the tests and see what happened.

1
2
3
4
5
6
7
8
9

λ npm test

> my_first_module@0.0.0 test /private/my_first_module
> mocha -R landing lib/my_first_module.js




  ✔ 0 tests complete (1ms)

Everything passed, that is expected because we don’t have any tests :) Now lets add to package.json one more thing development dependency for should.js it will enable us to use mocha in a bit rspec bdd style. Like this:

1
2
3

  "devDependencies": {
    "should": ">= 0.0.0"
  },

And again npm install -l to get everything installed locally.

First test

Initial mocha test for our module

1
2
3
4
5
6
7
8
9
10
11
12
13
14

function MyFirstFoo(a, b){

}


module.exports = MyFirstFoo

require('should')

describe('MyFirstFoo', function(){
  it("should be able to add", function(){
    MyFirstFoo(2, 3).should.be[5];
  });
});

First we define empty body of function next we have module.exports = this is to mark which functions will be visible outside this module when other clients will require this module. If you need more info about writing specs in mocha please read my earlier post http://no-fucking-idea.com/blog/2012/04/05/testing-handlebars-with-mocha/. Now lets run npm test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

 λ npm test

> my_first_module@0.0.0 test /private/tmp/my_first_module
> mocha -R landing lib/my_first_module.js
  -----------------------------------------------------------------------------------------------------------------------------------------------
  ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅✈
  -----------------------------------------------------------------------------------------------------------------------------------------------

  ✖ 1 of 1 tests failed:

  1) MyFirstFoo should be able to add:
     TypeError: Cannot read property 'should' of undefined
      at Context.<anonymous> (/private/tmp/lol/my_first_module/lib/my_first_module.js:13:21)
      at Test.run (/opt/local/lib/node_modules/mocha/lib/runnable.js:156:32)
      at Runner.runTest (/opt/local/lib/node_modules/mocha/lib/runner.js:272:10)
      at /opt/local/lib/node_modules/mocha/lib/runner.js:316:12
      at next (/opt/local/lib/node_modules/mocha/lib/runner.js:199:14)
      at /opt/local/lib/node_modules/mocha/lib/runner.js:208:7
      at next (/opt/local/lib/node_modules/mocha/lib/runner.js:157:23)
      at Array.0 (/opt/local/lib/node_modules/mocha/lib/runner.js:176:5)
      at EventEmitter._tickCallback (node.js:192:40)

...

Fails as expected so lets ad implementation to our function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

function MyFirstFoo(a, b){
  return a + b;
}


module.exports = MyFirstFoo

require('should')

describe('MyFirstFoo', function(){
  it("should be able to add", function(){
    MyFirstFoo(2, 3).should.be[5];
  });
});

And run npm test

1
2
3
4
5
6
7
8
9

λ npm test

> my_first_module@0.0.0 test /private/tmp/lol/my_first_module
> mocha -R landing lib/my_first_module.js
  -----------------------------------------------------------------------------------------------------------------------------------------------
  ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅✈
  -----------------------------------------------------------------------------------------------------------------------------------------------

  ✔ 1 tests complete (2ms)

We have landed safely ;) Now we are ready for development!

Summary

Creating good quality of code requires testing in node.js thats why i decided to join this two things and explain how to marry them both fast. More info can be found here http://howtonode.org/how-to-module.

I hope it helped a bit. Cheers!

Here is link to overview how redis cluster works http://redis.io/presentation/Redis_Cluster.pdf

redis.conf

Regular nodes can’t be part of cluster :( so you have to prepare separate redis configs for your cluster servers. Most important thing is to setup cluster-enabled and cluster-config-file I decided to name my config files redis-cluster-<port>.conf. I used ports 4444, 4445 4446 Here is my sample config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

daemonize yes
timeout 0
loglevel notice
logfile stdout
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./cluster_4444
slave-serve-stale-data yes
slave-read-only yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000

# Cluster
#
pidfile /var/run/redis-4444.pid
port 4444
cluster-enabled yes
cluster-config-file redis-cluster-4444.conf

slowlog-log-slower-than 10000
slowlog-max-len 1024
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

For each node i created directory cluster_<port> and that was the hardest part actually to do. With this all you have to do is to start ( for debug you can set daemonize to no) all nodes using redis-server path/to/redis-cluster-<port>.conf and then use magic ruby tool :)

redis-tribe.rb

In src/ directory of source you can find ruby script for creating and managing cluster. But first you need to have ruby installed with redis gem. i just did gem install redis but if you don’t have ruby you have to google how to install it etc (hint: get 1.9.2).

now you can run the script. ./redis-tribe.rb and see

1
2
3
4
5
6

λ ./redis-trib.rb
Usage: redis-trib <command> <arguments ...>

  create               host1:port host2:port ... hostN:port
  check                host:port
  reshard              host:port

To start cluster we will type “create” (useless comment)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

λ ./redis-trib.rb create 127.0.0.1:4444 127.0.0.1:4445 127.0.0.1:4446
Creating cluster
Connecting to node 127.0.0.1:4444: OK
Connecting to node 127.0.0.1:4445: OK
Connecting to node 127.0.0.1:4446: OK
Performing hash slots allocation on 3 nodes...
[FAIL] 5a2f6df453f1cd52bcb22c2afc45580283bcce87 127.0.0.1:4444 slots:0-1364 (1365 slots)
[FAIL] 35d107017bc726ece9b57e1ea2f21678555cf6a8 127.0.0.1:4445 slots:1365-2729 (1365 slots)
[FAIL] 76d06b0d3cb1b3829cb60574260dff2d06964cea 127.0.0.1:4446 slots:2730-4095 (1366 slots)
Can I set the above configuration? (type 'yes' to accept): yes
** Nodes configuration updated
** Sending CLUSTER MEET messages to join the cluster
Performing Cluster Check (using node 127.0.0.1:4444)
[FAIL] 5a2f6df453f1cd52bcb22c2afc45580283bcce87 127.0.0.1:4444 slots:0-1364 (1365 slots)
[FAIL] 35d107017bc726ece9b57e1ea2f21678555cf6a8 127.0.0.1:4445 slots:1365-2729 (1365 slots)
[FAIL] 76d06b0d3cb1b3829cb60574260dff2d06964cea 127.0.0.1:4446 slots:2730-4095 (1366 slots)
[OK] All 4096 slots covered.

Nice we have our cluster running :) now we can connect to any node and try it out.

1
2
3

λ ./redis-cli -h 127.0.0.1 -p 4445
redis 127.0.0.1:4445> set "jakub" "oboza"
(error) MOVED 198 127.0.0.1:4444

Sweet :D

Using this tool you can also reshard :D I did on my 15 keys worked :-F.

Smart clients

In redis doc we can read that you will require “smart client” to make it low latency. Yes, you can read from output that it was moved so you will have to cache where the key is now and reset temp cache when it will be moved (resharding)

Fire!

You can now test how it will behaves under fire by killing and restarting your nodes eg.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

[19008] 16 Apr 19:44:09.945 # Server started, Redis version 2.9.7
[19008] 16 Apr 19:44:09.946 * The server is now ready to accept connections on port 4444
[19008] 16 Apr 19:45:14.414 * Connecting with Node c20290a7b70a2a840a168c3309f00e3de1b1844d at 127.0.0.1:14446
[19008] 16 Apr 19:45:15.424 * Connecting with Node ab93647957ed4bb93fc43b1dc76202a6cdb94f49 at 127.0.0.1:14445
[19008] 16 Apr 19:59:10.047 * 1 changes in 900 seconds. Saving...
[19008] 16 Apr 19:59:10.047 * Background saving started by pid 19321
[19321] 16 Apr 19:59:10.080 * DB saved on disk
[19008] 16 Apr 19:59:10.248 * Background saving terminated with success
[19008] 16 Apr 20:08:29.837 * I/O error reading from node link: connection closed
[19008] 16 Apr 20:08:29.837 * I/O error reading from node link: connection closed
[19008] 16 Apr 20:08:30.056 * Connecting with Node 76d06b0d3cb1b3829cb60574260dff2d06964cea at 127.0.0.1:14446
[19008] 16 Apr 20:08:30.056 * I/O error writing to node link: Broken pipe
[19008] 16 Apr 20:08:30.525 * I/O error reading from node link: connection closed
[19008] 16 Apr 20:08:30.526 * I/O error reading from node link: connection closed
[19008] 16 Apr 20:08:31.063 * Connecting with Node 35d107017bc726ece9b57e1ea2f21678555cf6a8 at 127.0.0.1:14445
[19008] 16 Apr 20:08:31.064 * Connecting with Node 76d06b0d3cb1b3829cb60574260dff2d06964cea at 127.0.0.1:14446

Summary

Even if i think this is a great tool, and is unstable i saw after few minutes play that some things just don’t work as intended and some keys are not pushed. But it is pulled from unstable branch so i’m crossing my fingers for this project because it looks sweet! Go Go Antirez.