No F*cking Idea

Common answer to everything

Using Eredis, Redis With Erlang

| Comments

Recently i decided to move my blog from tumblr.com to octopress engine because it is just easier for me to maintain it and it looks nicer. The old blog is under http://no-fucking-idea.tumblr.com. My first post on new blog is dedicated to using redis with erlang.

Eredis

Wooga created a really nice (performance driven) redis driver for erlang. You can get it here https://github.com/wooga/eredis. It is really easy and nice.

Initial sample

On project page you can find simple examples how to use eredis. Examples there are all you need (but i need something for front page of my new blog so i will rewrite them and explain them :) ).

First you need to start your eredis application

initialization
1

{ok, Client} = eredis:start_link().

Client is the “connection / state” we will be using with rest of queries.

To query things with redis we will use q method from eredis module which takes “Connection / Client” state and list with params. This api is very simple here are two most basic examples of get and set. GET:

get
1

{ok, <<"OK">>} = eredis:q(Client, ["SET", "name", "kuba"]).

and SET:

set
1

{ok, <<"kuba">>} = eredis:q(Client, ["GET", "name"]).

From my point of view this is ideal candidate to be turned into gen_server behavior. We will pass “Connection / Client” as state and also we will build some “key” serialization methods around it to make it more durable and make our life easy if we will decide to refactor it later on.

Free Api wrapper

First thing i saw during development using Erlang is that you get free api if you follow simple patterns and encapsulate things into gen_server’s and applications.

example_db.erl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

-module(example_db).
-behaviour(gen_server).

-author("jakub.oboza@gmail.com").
-define(Prefix, "example").

-export([start_link/0]).
-export([init/1, handle_call/3, handle_cast/2, terminate/2, handle_info/2, code_change/3, stop/1]).
-export([get_script/2, save_script/3]).

% public api

start_link() ->
  gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

init([]) ->
  {ok, Redis} = eredis:start_link(),
  {ok, Redis}.

stop(_Pid) ->
  stop().

stop() ->
    gen_server:cast(?MODULE, stop).

%% public client api

get_script(Api, Method) ->
  gen_server:call(?MODULE, {get_script, Api, Method}).

save_script(Api, Method, Script) ->
  gen_server:call(?MODULE, {save_script, Api, Method, Script}).

%% genserver handles

handle_call({get_script, Api, Method}, _From, Redis) ->
  Response = eredis:q(Redis, [ "GET", get_key(Api, Method) ]),
  {reply, Response, Redis};

handle_call({save_script, Api, Method, Script}, _From, Redis) ->
  Response = eredis:q(Redis, ["SET", get_key(Api, Method), Script]),
  {reply, Response, Redis};

handle_call(_Message, _From, Redis) ->
  {reply, error, Redis}.

handle_cast(_Message, Redis) -> {noreply, Redis}.
handle_info(_Message, Redis) -> {noreply, Redis}.
terminate(_Reason, _Redis) -> ok.
code_change(_OldVersion, Redis, _Extra) -> {ok, Redis}.

%% helper methods

get_key(Api, Method) ->
  generate_key([Api, Method]).

generate_key(KeysList) ->
  lists:foldl(fun(Key, Acc) -> Acc ++ ":" ++ Key end, ?Prefix, KeysList).

% tests

-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").
-endif.


-ifdef(TEST).

generate_key_test() ->
  Key = generate_key(["one", "two", "three"]),
  ?assertEqual("example:one:two:three", Key).

server_test_() ->
  {setup, fun() -> example_db:start_link() end,
   fun(_Pid) -> example_db:stop(_Pid) end,
   fun generate_example_db_tests/1}.

generate_example_db_tests(_Pid) ->
  [
    ?_assertEqual({ok,<<"OK">>}, example_db:save_script("jakub", "oboza", <<"yo dwang">>) ),
    ?_assertEqual({ok,<<"yo dwang">>}, example_db:get_script("jakub", "oboza") )
  ].

-endif

Public Api

This code listing has two important parts first at top it starts at line 26. This is the public API which will be used by developer. This is this free api. Later on i will explain how to change redis to mongodb and probably to other db engines without doing changes in rest of our app. From my perspective this is awesome feature. In most cases when i had to make app scale problem of having code that was glues to one db engine was heavy.

eunit tests

At line 60. starts the declaration of tests, using rebar and eunit is very easy and it is always good to have test coverage in case of refactoring. I’m a fan of test driven development so i like to cover in tests first things that i will use or i think they might be error prone. Here is used “test generators” to just to show how to write tests for gen_server.

Rebar

Before i will explain more i need to say little about rebar. It is a command line tool that was developed by basho to help create app. it is by far the best thing i found learning erlang to help me do boring stuff and eliminate a lot of rage writing app.src files. To get rebar simply do (you can always go to https://github.com/basho/rebar to get most up to date informations about building it)

1
2
3

λ  git clone git://github.com/basho/rebar.git
λ  cd rebar
λ  ./bootstrap

I use my own set of zsh scripts so all i did to add it to my path was to edit .furby file in my home dir. I strongly suggest also adding it to $PATH just to make your life easier.

Back to example_db!

To create app using rebar you just need to

1
2
3
4
5
6

λ mkdir example_db
λ rebar create-app appid=example_db
==> example_db (create-app)
Writing src/example_db.app.src
Writing src/example_db_app.erl
Writing src/example_db_sup.erl

This command created src folder with scaffold of application OTP pattern and supervisor thats almost all we need :). Now you can compile it using rebar compile and run tests using rebar compile eunit in out app currently we will see

rebar compile eunit
1
2
3
4
5
6
7
8

λ rebar compile eunit
==> example_db (compile)
Compiled src/example_db_app.erl
Compiled src/example_db_sup.erl
==> example_db (eunit)
Compiled src/example_db_app.erl
Compiled src/example_db_sup.erl
  There were no tests to run.

Nothing to do because its empty. Lets add our db module. But before this we need to add dependencies for eredis module. Lets create rebar.config file and add it.

1
2
3
4
5
6
7
8
9
10
11
12

λ emacs rebar.config
λ cat rebar.config
%%-*- mode: erlang -*-

{erl_opts, []}.
{cover_enabled, true}.

{deps,
  [
    {eredis, ".*", {git, "https://github.com/wooga/eredis.git", "HEAD"}}
  ]
}.

Now just run rebar get-deps to get all dependencies downloaded. After adding our example_db.erl into src directory we can run rebar compile eunit to compile and run tests. We have added {cover_enabled, true} in rebar.conf so also test code coverage will be generated for us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

λ rebar compile eunit
==> eredis (compile)
==> example_db (compile)
Compiled src/example_db.erl
==> eredis (eunit)

=ERROR REPORT==== 28-Mar-2012::22:19:35 ===
** Generic server <0.263.0> terminating
** Last message in was {tcp,#Port<0.4516>,
                            <<"*3\r\n$7\r\nmessage\r\n$3\r\nfoo\r\n$2\r\n12\r\n">>}
** When Server state == {state,"127.0.0.1",6379,<<>>,100,#Port<0.4516>,
                               {pstate,undefined,undefined},
                               [<<"foo">>],
                               {#Ref<0.0.0.4058>,<0.180.0>},
                               {[{message,<<"foo">>,<<"11">>,<0.263.0>},
                                 {message,<<"foo">>,<<"10">>,<0.263.0>},
                                 {message,<<"foo">>,<<"9">>,<0.263.0>},
                                 {message,<<"foo">>,<<"8">>,<0.263.0>},
                                 {message,<<"foo">>,<<"7">>,<0.263.0>},
                                 {message,<<"foo">>,<<"6">>,<0.263.0>},
                                 {message,<<"foo">>,<<"5">>,<0.263.0>},
                                 {message,<<"foo">>,<<"4">>,<0.263.0>},
                                 {message,<<"foo">>,<<"3">>,<0.263.0>}],
                                [{message,<<"foo">>,<<"2">>,<0.263.0>}]},
                               10,exit,need_ack}
** Reason for termination ==
** max_queue_size
  All 53 tests passed.
Cover analysis: /private/tmp/example_db/deps/eredis/.eunit/index.html
==> example_db (eunit)
Compiled src/example_db.erl
  All 3 tests passed.
Cover analysis: /private/tmp/example_db/.eunit/index.html

All seems to be fine! lets create file called start.sh to test it out

1
2

λ cat start.sh
erl -pa ebin -pa deps/*/ebin

and make it executable with chmod +x start.sh

And lets rock ‘n’ roll

1
2
3
4
5
6
7
8
9
10

 λ ./start.sh
Erlang R15B (erts-5.9) [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9  (abort with ^G)
1> example_db:start_link().
{ok,<0.33.0>}
2> example_db:save_script("example", "script", "puts '2+2'").
{ok,<<"OK">>}
3> example_db:get_script("example", "script").
{ok,<<"puts '2+2'">>}

Have fun :) Hope it was useful. You can download code for this blog post here https://github.com/JakubOboza/example_db-code

Huh ^___^

that was my first post on new blog ;)

Comments