This blog post is start of series on building ORM/ODM libraries for Mongodb in different languages. I am a big fan of mongoid http://mongoid.org great Ruby ODM for mongodb. But i am even bigger fan of Mongodb http://www.mongodb.org/. Some time ago i saw post by my friend from http://www.lunarlogicpolska.com/ about virtus http://solnic.eu/2011/06/06/virtus—attributes-for-your-plain-ruby-objects.html (it’s really worth reading) and i thought “it’s nice”. Today i have spent two hours to cook this starter project because i want to learn more about virtus and building ODM’s ORM’s is not so common topic across web.

Aim

Building fully featured ODM in two hours from scratch is more then you can expect from me. Goal if first part is to make something that will map ruby PORO’s to mongodb collections, be able to save them, find one or many in database and destroy. This looks like a lot but we will do it slowly and the implementation will be very basic. Our ODM will be named “Muppet”. Name describes the project :).

Virtus

Virtus handles properties for Plain Old Ruby Objects and this is all we need to have. This eliminates a lot of boilerplate code we would have to write to make anything work. PORO Powah!

Mongo

mongo is a Mongodb native ruby driver. It has really good api and is easy to use. For most things in this post i just proxied calls to it :).

Database!

Be sure to have mongodb working ;).

First step: Building a gem

All about building gem you can find in my previous blog post here http://no-fucking-idea.com/blog/2012/04/11/building-gem-with-bundler/ Be sure to add rspec :) i used it to describe tests for this project.

Second step: Describing api

I like do develop things in TDD/BDD style so first thing for me description of api i wanted to implement during this tutorial. All this specs are stripped to minimum to enhance readability.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86

require 'spec_helper'

class User
  include Virtus
  include Muppet

  attribute :name, String
  attribute :age, Integer
  attribute :birthday, DateTime
end

describe Muppet do

  describe "#configuration" do

    it "should be configurable" do
      lambda do
        Muppet.configure \{ \{
          host: "localhost",
          port: 27017,
          database: "test"
        }}
        Muppet.config.should be_a(Hash)
        Muppet.config[:host].should eql("localhost")
      end.should_not raise_error
    end

    it "should by default point to localhost:27017" do
      lambda do
        Muppet.config[:host].should eql("localhost")
        Muppet.config[:port].should eql(27017)
      end
    end

  end

  describe "#connection" do

    it "should connect to mongodb" do
      lambda do
        Muppet.connect!
      end.should_not raise_error
    end

  end

  describe "#inserting" do

    it "should insert values to database" do
      pending
      lambda do
        user = User.new(name: "Jakub", age: 27 )
        user.save.should be_true
      end.should change(User, :count)
    end

  end

  describe "#quering" do

    it "should query all documents from collection" do
      pending
      User.find_one.should be_a(User)
    end

    it "should query first document from collection" do
      pending
      users = User.find
      users.should be_a(Array)
      users[0].should be_a(User)
    end

  end

  describe "#destroy" do

    it "should destroy document" do
      pending
      lambda do
        User.find_one.destroy
      end.should change(User, :count).by(-1)
    end

  end

end

I defined them in order how i wanted to implement this. First configuration and connection to db. Next inserting and querying and destroying as last thing. Also at top i have simple User object with virtus and muppet included.

Third step: layouting Muppet!

At this moment we should have ready specs, that fail hard throwing errors on unknown tokens. Its good. What i did is simple lib/muppet.rb defines the module we will include into out PORO’s.

1
2
3
4
5
6
7

require "mongo"
require "muppet/version"
require "muppet/setup"

module Muppet
  extend Muppet::Setup
end

in lib/muppet/ we will have components of our project. I we already know we will start with setup as defined in muppet.rb so lets create file lib/muppet/setup and configuration and connection to mongodb.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

module Muppet
  module Setup
    @@configuration = {
      host: "localhost",
      port: 27017,
      database: "test"
    }

    def configure(&user_config)
      @@configuration.merge!(user_config.call)
    end

    def config
      @@configuration
    end

    def connect!
      @@connection = Mongo::Connection.new(@@configuration[:host],
        @@configuration[:port])
      @@database = @@connection.db(@@configuration[:database])
    end

    def database
      @@database
    end

    def connection
      @@connection
    end

  end
end

Here i defined default configuration and few method to access vital part of our config like database , connection. Most important part is connect! this method uses mongo gem to establish connection to mongodb store it into connection variable and set the database we will be working on. I wanted to make few things explicit so i used some name redundancy. (later on i learned that i did not even need config method)

With this working we can run out rspecs and if mongodb is up we should see all green and few yellows! Good it works!

Forth step: support for quering and inserting

Now lets remove pending marks from specs that are in “describes” quering and inserting. This will be the heart of our ODM. We will define how he should save and load object from database. Before this we will have to update out lib/muppet.rb to include things we will use.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

require "mongo"
require "muppet/version"
require "muppet/setup"
require "muppet/document"

module Muppet
  extend Muppet::Setup
  include Muppet::Document::InstanceMethods

  def self.included(base)
    base.extend( Muppet::Document::ClassMethods )
  end

end

Ok many new things. I created lib/muppet/document.rb module with the stuff we will put into class and instance definitions in the moment of inclusion. As we can see in definition of User in our test cases we will include Muppet so all the instance methods like (save, destroy) will have to be defined in separated module then class methods like (find_one, find). In document.rb we can see how it is implemented.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

module Muppet
  module Document
    module ClassMethods

      def collection_name
        self.to_s
      end

      def collection
        Muppet.database.collection(collection_name)
      end

      def find_one(opts = {})
        result = collection.find_one(opts)
        return nil unless result
        self.new(result)
      end

      def find(opts = {})
        results = collection.find(opts)
        results.map{|result| self.new(result) }
      end

      def count
        collection.count
      end

    end

    module InstanceMethods

      def save
        self.class.collection.insert(self.attributes)
      end

      def destroy
        self.class.collection.remove(self.attributes)
      end

    end

  end
end

I could not think of a way to implement it in a more simple way. We have to “sections” first is class methods where we define

• collection_name this method says to us what is the collection name (we can override it in model)
• collection uses database to return this collection object for us to use.
• find_one, find, count are methods that we proxy in a dirty explicit way (but without using method missing) things to mongo native driver.

Only thing we do here is to wrap things that we get back into self.new object. So we can mimic AR/Mongoid api and when doing User.find we will get back array of all users. not array of hashes :).

Instance methods are only two save and destroy save is raw and simple, takes all attributes from using virtues and saves them using mongo native driver. destroy acts in the same way.

Now we can run the specs and see all is green. We have a basic ODM where we can add documents, map and query them to PORO’s and remove.

Summary

I really enjoyed writing this code and blog post :). You can find code for this blog post here https://github.com/JakubOboza/muppet

This code is buggy and even specs needs to be enhanced but this is a good start for building new features on top of it. In future parts i want to implement, updating, proxy objects, relations, embedded objects, dirty tracking(probably using https://github.com/solnic/virtus-dirty_tracking) and few other mechanism that will enable us to make a fully functional ODM from “Muppet”

-Cheers

Building a new gem ( ruby library ) with bundler is easy task. I found a lot of tutorial on this topic in the web but none of them was covering more then just generating scaffold and packing up the gem. I will try to uncover a bit more. So at the end of this post you will be able to generate new gem, build it and it will have ready support for rspec.

First of all you need bundler. To install bundler just type gem install bundler. Probably most of you guys have it already :) but just in case.

Generating new gem

To generate gem scaffold all you have to do is call bundle gem <gem name> like here

1
2
3
4
5
6
7
8
9
10

bundle gem doctor_toons
      create  doctor_toons/Gemfile
      create  doctor_toons/Rakefile
      create  doctor_toons/LICENSE
      create  doctor_toons/README.md
      create  doctor_toons/.gitignore
      create  doctor_toons/doctor_toons.gemspec
      create  doctor_toons/lib/doctor_toons.rb
      create  doctor_toons/lib/doctor_toons/version.rb
Initializating git repo in /private/tmp/doctor_toons

This will create directory named doctor_toons with:

• Gemfile where we specify production and development gems
• Rakefile with our rake tasks, you need to remember that rake is not part of Gemfile if you are using ruby < 1.9.2
• License file with license, GPL, BSD, MIT, LOL choose one and put it in by default MIT.
• it initializes git repo inside this directory and adds .gitignore with default ignores

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

λ cat .gitignore
*.gem
*.rbc
.bundle
.config
.yardoc
Gemfile.lock
InstalledFiles
_yardoc
coverage
doc/
lib/bundler/man
pkg
rdoc
spec/reports
test/tmp
test/version_tmp
tmp

• lib directory for your code !!!
• < name >.gemspec file with whole gemspec for our gem.

First thing we need to do is set gems we will be using while building our lib. Initial Gemfile looks like this

1
2
3
4
5

λ cat Gemfile
source 'https://rubygems.org'

# Specify your gem's dependencies in doctor_toons.gemspec
gemspec

All production needed gems needs to be added before gemspec. So if we will want to add rspec as development dependency we need to put it after gemspec term in Gemfile. Eg. if we want to use mongo 1.6.2 as production dependency and rspec 2.9.0 as our development dependency we should do it like this.

1
2
3
4
5
6
7
8
9

source 'https://rubygems.org'

gem 'mongo', '~> 1.6.2'
gem 'bson',  '~> 1.6.2'

# Specify your gem's dependencies in doctor_toons.gemspec
gemspec

gem 'rspec', '~> 2.9.0'

Now we can simply bundle all dependencies for our gem by hitting. bundle install you should see something like this

1
2
3
4
5
6
7
8
9
10
11
12

λ bundle install
Fetching gem metadata from https://rubygems.org/....
Using bson (1.6.2)
Using diff-lcs (1.1.3)
Using doctor_toons (0.0.1) from source at /private/tmp/doctor_toons
Using mongo (1.6.2)
Using rspec-core (2.9.0)
Using rspec-expectations (2.9.1)
Using rspec-mocks (2.9.0)
Using rspec (2.9.0)
Using bundler (1.1.0)
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

Building gem!

Ok now we are ready for next step, building gem! To do it we call gem build < name >.gemspec so lets do it

1
2
3

λ gem build doctor_toons.gemspec
ERROR:  While executing gem ... (Gem::InvalidSpecificationException)
    "FIXME" or "TODO" is not a description

Error. This means that we need to supply few things before we will build this gem. First of all we need to emacs < name >.gemspec to edit the file with gemspec and set few things. By default this file should look like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# -*- encoding: utf-8 -*-
require File.expand_path('../lib/doctor_toons/version', __FILE__)

Gem::Specification.new do |gem|
  gem.authors       = ["JakubOboza"]
  gem.email         = ["jakub.oboza@gmail.com"]
  gem.description   = %q{TODO: Write a gem description}
  gem.summary       = %q{TODO: Write a gem summary}
  gem.homepage      = ""

  gem.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
  gem.files         = `git ls-files`.split("\n")
  gem.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
  gem.name          = "doctor_toons"
  gem.require_paths = ["lib"]
  gem.version       = DoctorToons::VERSION
end

So lets edit description and summary and build again our gem. gem build doctor_toons.gemspec

1
2
3
4
5
6

λ gem build doctor_toons.gemspec
WARNING:  no homepage specified
  Successfully built RubyGem
  Name: doctor_toons
  Version: 0.0.1
  File: doctor_toons-0.0.1.gem

Yeah now we can build our gem :) Lets have a look at gemspec gem spec doctor_toons-0.0.1.gem before we will be adding rspec to it :)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

 λ gem spec doctor_toons-0.0.1.gem
--- !ruby/object:Gem::Specification
name: doctor_toons
version: !ruby/object:Gem::Version
  version: 0.0.1
  segments:
  hash:
platform: ruby
authors:
- JakubOboza
autorequire:
bindir: bin
cert_chain: []
date: 2012-04-11 00:00:00.000000000Z
dependencies: []
description: He sleeps with a gun, but he loves his son
email:
- jakub.oboza@gmail.com
executables: []
extensions: []
extra_rdoc_files: []
files:
- .gitignore
- Gemfile
- LICENSE
- README.md
- Rakefile
- doctor_toons.gemspec
- lib/doctor_toons.rb
- lib/doctor_toons/version.rb
homepage: ''
licenses: []
post_install_message:
rdoc_options: []
require_paths:
- lib
required_ruby_version: !ruby/object:Gem::Requirement
  none: false
  requirements:
  - - ! '>='
    - !ruby/object:Gem::Version
      version: '0'
      segments:
      hash:
required_rubygems_version: !ruby/object:Gem::Requirement
  none: false
  requirements:
  - - ! '>='
    - !ruby/object:Gem::Version
      version: '0'
      segments:
      hash:
requirements: []
rubyforge_project:
rubygems_version: 1.8.15
signing_key:
specification_version: 3
summary: Psycho dad
test_files: []

Everything is fine.

Adding testing support

Before we will be wiriting any library code its nice to add support for testing framework. I like name of rspec and rspec as a lib. It lets you define in form of tests specification for your library.

Rspec

Adding rspec to our project is easy. All we need to do it create directory called spec and file inside of it called spec_helper.rb like this

1
2

λ mkdir spec
λ touch spec/spec_helper.rb

Now we need to edit spec_helper.rb and add few things.

1
2
3
4
5
6
7
8

require 'rubygems'
require 'bundler/setup'
# our gem
require 'doctor_toons'

RSpec.configure do |config|

end

This is bare minimum spec_helper.rb file we can have. All we do is require on rubygems, bundler setup and our gem doctor_toons :). To make it easy to use we should expand our rake with a spec task! like this

1
2
3
4
5
6
7
8

#!/usr/bin/env rake
require "bundler/gem_tasks"

require 'rspec/core/rake_task'

RSpec::Core::RakeTask.new('spec')

task :default => :spec

This will add rspec task called rake spec that will run all our specs :) Now last thing we need to add first initial spec to our spec folder so we will be able to test it.

1
2
3
4
5
6
7
8
9
10
11

require 'spec_helper'

describe DoctorToons do

  it "should rock" do
    lambda do
      DoctorToons.rock()
    end.should_not raise_error
  end

end

Remember spec file names has to end with _spec.rb

Now if you will run rake spec you should see something like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

Users/kuba/.rvm/rubies/ruby-1.9.2-p290/bin/ruby -S rspec ./spec/doctor_toons_spec.rb
F

Failures:

  1) DoctorToons should rock
     Failure/Error: lambda do
       expected no Exception, got #<NoMethodError: undefined method `rock' for DoctorToons:Module>
     # ./spec/doctor_toons_spec.rb:6:in `block (2 levels) in <top (required)>'

Finished in 0.00218 seconds
1 example, 1 failure

Failed examples:

rspec ./spec/doctor_toons_spec.rb:5 # DoctorToons should rock
rake aborted!
/Users/kuba/.rvm/rubies/ruby-1.9.2-p290/bin/ruby -S rspec ./spec/doctor_toons_spec.rb failed

Tasks: TOP => spec
(See full trace by running task with --trace)

Wooow works :) now you can implement your library to rock the world!

Few useful things

If you want to have colorful output like me :). You can add .rspec file in top dir of project with content like this

1
2

--color
--format documentation

First line adds color output and second line is format out spec output, it is nice because its the most verbose output. You can easy see where spec failed. When you are on early stage of the project and you have few describes like 50-100 it often helps to develop faster.

Summary

This is just intro post about my upcoming series on building tools that you need :). Fingers crossed :)

-Cheers!

Mocha and Handlebars are two great things i use. Mocha is a testing library which can be used for backend (node.js) and fronend testing. On frontend its only dependency is jQuery. Handlebars is templating language that can be used for frontend javascript partials or even for backend (node.js) layouts. What ever you want! :).

Mocha

When i started working with express.js sinatra like framework i took a peek at other projects that people from Vision Media do. One of them was mocha and at that time i needed something to test my backend code. It is good to use such a tool while learning new thing. This was you can document your failures :). As i found it is also great tool to test front end code.

Chai.js / Should.js

Because Mocha is not shipped with everything i like i decided to use should.js https://github.com/visionmedia/should.js on backend (node.js) and chai.js on frontend to make my tests suits more rspec like. This way we will be able to write our specs like this.

1
2
3
4
5
6
7
8
9
10

(function() {

  describe('Application', function() {
    it('should exists', function() {
       var app = new Application();
       app.should.be.an.instanceof(Application);
    });
  });

}).call(this);

I really like this app.should.be.an.instanceof(Application); syntax. But if you don’t like it you can use more jasmine like syntax all info can be found here http://visionmedia.github.com/mocha/.

Handlebars

Second very useful thing that i never skip in my project is Handlebars, easy to use and clean language to build templates in javascript. Simply it is mustache on steroids. But lets dive straight into some example.

1
2
3
4
5
6
7

 <script id="list-entry" type="text/x-handlebars-template">
    <li  class= >
        <span></span> -> 
      <strong></strong>:&nbsp;
      <span></span>
  </li>
 </script>

This sample show how to in easy way embed handlebars into your html code. Important thing is that only code in is getting interpolated. so in this example ` class= ` mans that if in rendering context we have class variable render `class=value_of_class_variable_in_context` as part of li element. will render contents of this var from context. If there is no variable with this name or it is undefined it will not be rendered. thats the important bit that can make some debugging harder. Eg. underscore.js templates explode if you don’t have any param that you use inside of them.

Both tools in action

Ok so lets try it out and write some code in bdd style using handlebars. I have prepared initial setup for this, it contains lib directory with our javascripts, spec directory with out specs and support directory with our mocha.js, chai.js etc. Our test runner is single index.html file we can open in browser. All the code can be downloaded here https://github.com/JakubOboza/handlebars-and-mocha

Step one

Lets write a spec that makes checks if out app loads

1
2
3
4
5
6
7
8
9
10

(function() {

  describe('Application', function() {
    it('should exists', function() {
       var app = new Application();
       app.should.be.an.instanceof(Application);
    });
  });

}).call(this);

Obviously we have here a failing spec. Lets write some implementation to satisfy out spec.

1
2
3

function Application(template_id){

}

Bang everything works fine :). Lets make it do something useful, render some templates! First lets write a spec for it –>

1
2
3
4

it('should render handlebars template', function(){
  var app = new Application("#test-template");
  app.render({name: "kuba"}).should.equal("<p>kuba</p>");
});

Now lets implement this, our template will look like this

1
2
3

 <script id="test-template" type="text/x-handlebars-template">
    <p></p>
 </script>

It is very basic, hard to make mistake ;). And javascript code

1
2
3
4
5
6
7
8

function Application(template_id){
  var source = $(template_id);
  this.template = Handlebars.compile(source.html());
}

Application.prototype.render = function(params){
  return this.template(params);
};

Works! Like a charm! I really enjoy writing tests for javascript this way. It is much more like rspec. At least for me it is much more useful then jasmine

Why not jasmine

I prefer syntax of Mocha. On this few examples i think i show how to start using it and how fun it is. For people with rspec background this should be very easy tool to pick up. More about Mocha can be found on official project page http://visionmedia.github.com/mocha/.

Example code for this post

Can be found here https://github.com/JakubOboza/handlebars-and-mocha just open index.html :)

Cheers!

Mongodb has support for running Map Reduce queries besides having regular sql like query interface. In documentation we can read that it is not the best idea to use it as a regular interface but it is very good for generating things in backgrounds like preparing reports or caching some data. I will try to show simple example how to create a useful map reduce query and execute it.

Javascript

Map reduce queries in Mongodb are written in javascript. All you have to do is to prepare two regular javascript functions

1
2
3
4

var map = function(){
    /* emit values for each document */
  emit();
}

In map function you have to emit key –> values from document, eg. for each document emit urls and counts of them.

1
2
3
4

var reduce = function(key, value){
    /* reduce emited values into result */
  return {result1: one, result2: two};
}

In reduce function you simple gather results and sum them up. It is easier to think about if you will imagine that reduce is something like fold or inject (depending on background) on emitted values from mapping function.

Running scripts

Mongo db has a really nice interface for running scripts. Lets examine a simple example

1

mongo localhost:27017/canis_production generate_report.js

This will run generate_report.js script on database canis_production on db node localhost:27017. You don’t need to do it, but its easiet to write it into file then type each time functions ;).

Example Map reduce query

Now this is a simple mapReduce that actually do something. It is emitting for each document url field and value 1. Reducer is adding values for the same key so this way we will know how many occurrences of each url we have across whole collection.

1
2
3
4
5
6
7
8

var map = function(){
  emit(this.url, 1);
}
var reduce = function(key, values){
  var res = 0;
  values.forEach(function(v){ res += 1});
  return {count: res};
}

this is have we defined out map reduce functions now all we need to do is just runt he query.

1

db.sites.mapReduce(map, reduce, { out: "mapped_urls" });

To run mapReduce we are using mapReduce function on collection (this example uses collection named “sites”), first argument is map function, second is reduce function and third is option but very useful, it is output collection where results will be stored in form of documents. This option lets us run the query at eg. night and see results in the morning :).

Lets test it

First some sample data

1
2
3
4
5

> db.sites.insert({url: "www.google.com", date: new Date(), trash_data: 5 });
> db.sites.insert({url: "www.no-fucking-idea.com", date: new Date(), trash_data: 13 });
> db.sites.insert({url: "www.google.com", date: new Date(), trash_data: 1 });
> db.sites.insert({url: "www.no-fucking-idea.com", date: new Date(), trash_data: 69 });
> db.sites.insert({url: "www.no-fucking-idea.com", date: new Date(), trash_data: 256 });

now functions and query

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

> var map = function(){
...   emit(this.url, 1);
... }
> var reduce = function(key, values){
...   var res = 0;
...   values.forEach(function(v){ res += 1});
...   return {count: res};
... }
> db.sites.mapReduce(map, reduce, { out: "mapped_urls" });
{
  "result" : "mapped_urls",
  "timeMillis" : 75,
  "counts" : {
      "input" : 5,
      "emit" : 5,
      "reduce" : 2,
      "output" : 2
  },
  "ok" : 1,
}

And results

1
2
3

> db.mapped_urls.find({})
{ "_id" : "www.google.com", "value" : { "count" : 2 } }
{ "_id" : "www.no-fucking-idea.com", "value" : { "count" : 3 } }

Worked perfect ;)

Docs

More information on map reduce interface you can find in documentation for mongodb http://www.mongodb.org/display/DOCS/MapReduce

Caching

With ruby and rails we often want to have caching of static content so we will try to reduce requests that has to come through rails stack when possible. Nginx in front is a great tool and we can use its abilities to add caching easy.

App

lets imagine we have a simple sinatra app. For purpose of this post we will have app with one method /ohhai that shows current time. This is a great way to test if out caching is working fine. Code of the app is really simple:

1
2
3
4
5
6

class ExampleStaleApp < Sinatra::Base
  get "/ohhai" do
    sleep(5)
    Time.now.to_s
  end
end

Also to start it easy i have created a config.ru rackup file describing how to start app (in repo there is start.sh script ;)

1
2
3
4
5
6
7
8

require 'rubygems'
require 'bundler'

Bundler.require

require './app'

run ExampleStaleApp

If you are using code from my repo https://github.com/JakubOboza/003-nginx-cache-stale-example to start it all you need to do is

1
2
3
4

λ git clone https://github.com/JakubOboza/003-nginx-cache-stale-example
λ cd 003-nginx-cache-stale-example
λ bundle install
λ ./start.sh

I configured the app with my own path/uri and to look for upstream server on port 6677 so you need to change it if you are using different settings.

Caching

Our app is running now. Lets add caching, for this we will need to add nginx frontend config. In most cases i create a single nginx config for each server in sites-available directory and symlink it in sites-enabled (like apache do by default). I like this setting, it helps a lot to maintain more then one site which is common in development environment and also common on shared applications servers.

Nginx config file

I will show complete nginx config file for this example and explain each bit one by one.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

upstream sinatra_rackup{
  server 0.0.0.0:6677;
}

proxy_cache_path  /tmp/cache levels=1:2 keys_zone=my-test-cache:8m max_size=5000m inactive=300m;

server {
    listen 80;
    server_name example_stale.local;
    root /Users/kuba/Workspace/Ruby/example_stale/public;

    access_log  /var/log/nginx/example.stale.access.log;

    location / {
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_cache my-test-cache;
      proxy_cache_valid  200 302  1m;
      proxy_cache_valid  404      60m;
      proxy_cache_use_stale   error timeout invalid_header updating;
      proxy_redirect off;

      if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
      }
      if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
      }
      if (!-f $request_filename) {
        proxy_pass http://sinatra_rackup;
        break;
      }
    }

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
      root html;
    }
}

It isn’t even so long ;)

upstream

1
2
3

upstream sinatra_rackup{
  server 0.0.0.0:6677;
}

in this part we create description of our upstream. In other words where our application server will be listening. It is easy to just use port but you can configure it to use unix.socket if you want to gain on performance.

global cache config

1

proxy_cache_path  /tmp/cache levels=1:2 keys_zone=my-test-cache:8m max_size=5000m inactive=300m;

This directive sets the place where cache is stored and sets the zone name and how big it can be. We will refer tot his zone later on in proxy pass cache config.

app cache config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

server {
    listen 80;
    server_name example_stale.local;
    root /Users/kuba/Workspace/Ruby/example_stale/public;

    location / {
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_cache my-test-cache;
      proxy_cache_valid  200 302  1m;
      proxy_cache_valid  404      60m;
      proxy_cache_use_stale   error timeout invalid_header updating;
      proxy_redirect off;

      proxy_pass http://sinatra_rackup;
    }

}

Here we configure server name for out application, port to listen on for it and root directory (this is fixed with my mac so you should change it). Whole magic happens in location description. here you have few important things for us.

1

proxy_cache my-test-cache;

Sets which cache zone we will use. We use same cache zone we defined in proxy_cache_path with keys_zone. Next we are setting for how long it should be cached. I used 1 minute for 200 and 302 status. this lets us see on our example app how this works each minute we see new time :). This is awesome! Next you can set different caching time expiry for other status. Here we are refreshing 404 status cache each hour (it could be days :) ).

Last but not the least is serving stale content if upstream is dead.

1

proxy_cache_use_stale error timeout invalid_header updating;

This sets up for us config that will enable serving stale content if upstream is dead. This is nice if you want to provide some content in case when your backend is dead.

Test

You can test it now. Or wait… with my config you have to add entry to /etc/hosts

1

127.0.0.1 example_stale.local

Now you can go to example_stale.local/ohhai (or just curl example_stale.local/ohhai) and see how our cache works. Even more now you can kill your app server and still see cache being served correctly.

Results

First request Next requests

–> http://www.youtube.com/watch?v=lgoXUzIwXk0

Cheers

How you can use it? Depends on your app architecture, but for every bit of content that you create which is “static” it is great thing to have. I like this feature of nginx and i hope this post will help you ;).

Intro

Rebar is a great command line tool for building your Erlang apps. It was developed by guys from basho http://basho.com/. If you want to build Erlang app or module you can skip a lot of config / boilerplate code but using rebar.

Wait what? How do i get it ?

To obtain rebar all you have to do is clone source using

1

λ git clone git://github.com/basho/rebar.git

after obtaining source go into directory and bootstrap it.

1
2

λ cd rebar
λ ./bootstrap

This will build rebar script if everything is successful. Last thing i suggest is adding this directory to your path.

1

λ export PATH=/path/to/rebar:$PATH

So you will be able to use it like other command lines tools from “global namespace”. Now you should have working rebar installation. Just to test that everything is ok you can run rebar like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

λ rebar
No command to run specified!
Usage: rebar [-h] [-c] [-v <verbose>] [-V] [-f] [-D <defines>] [-j <jobs>] [-C <config>] [-p] [var=value,...] <command,...>

  -h, --help      Show the program options
  -c, --commands  Show available commands
  -v, --verbose       Verbosity level (-v, -vv, -vvv, --verbose 3). Default: 0
  -V, --version       Show version information
  -f, --force     Force
  -D          Define compiler macro
  -j, --jobs      Number of concurrent workers a command may use. Default: 3
  -C, --config        Rebar config file to use
  -p, --profile       Profile this run of rebar
  var=value        rebar global variables (e.g. force=1)
  command        Command to run (e.g. compile)

How do i use it @_@ ?

Two most important things you can generate using rebar are applications and nodes. To generate application you just need to create app directory and run rebar create-app command like this.

1
2
3
4
5
6
7

mkdir furby
λ cd furby
λ rebar create-app appid=furby
==> furby (create-app)
Writing src/furby.app.src
Writing src/furby_app.erl
Writing src/furby_sup.erl

This has created application scaffold with ready to go supervisor. This is ready to go! to compile it just run rebar compile

Me gusta

This is all fine but that don’t eliminate a lot, sweet things are behind the corner :).

eunit

Rebar enables you to use easy eunit testing framework within your code. Like we did it here http://no-fucking-idea.com/blog/2012/03/23/using-eredis-in-erlang/. To do it just run rebar compile eunit .

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

λ rebar compile eunit
zsh: correct 'eunit' to '.eunit' [nyae]? n
==> eredis (compile)
==> example_db (compile)
==> eredis (eunit)

=ERROR REPORT==== 29-Mar-2012::19:13:21 ===
** Generic server <0.259.0> terminating
** Last message in was {tcp,#Port<0.4549>,
                            <<"*3\r\n$7\r\nmessage\r\n$3\r\nfoo\r\n$2\r\n12\r\n">>}
** When Server state == {state,"127.0.0.1",6379,<<>>,100,#Port<0.4549>,
                               {pstate,undefined,undefined},
                               [<<"foo">>],
                               {#Ref<0.0.0.3990>,<0.176.0>},
                               {[{message,<<"foo">>,<<"11">>,<0.259.0>},
                                 {message,<<"foo">>,<<"10">>,<0.259.0>},
                                 {message,<<"foo">>,<<"9">>,<0.259.0>},
                                 {message,<<"foo">>,<<"8">>,<0.259.0>},
                                 {message,<<"foo">>,<<"7">>,<0.259.0>},
                                 {message,<<"foo">>,<<"6">>,<0.259.0>},
                                 {message,<<"foo">>,<<"5">>,<0.259.0>},
                                 {message,<<"foo">>,<<"4">>,<0.259.0>},
                                 {message,<<"foo">>,<<"3">>,<0.259.0>}],
                                [{message,<<"foo">>,<<"2">>,<0.259.0>}]},
                               10,exit,need_ack}
** Reason for termination ==
** max_queue_size
  All 53 tests passed.
Cover analysis: /private/tmp/example_db/deps/eredis/.eunit/index.html
==> example_db (eunit)
  All 3 tests passed.
Cover analysis: /private/tmp/example_db/.eunit/index.html

coverage

Also if you will fiddle with rebar.config and set some variables like this:

1
2
3
4
5

λ cat rebar.config
%%-*- mode: erlang -*-

{erl_opts, []}.
{cover_enabled, true}.

you can get test coverage generated in .eunit folder. but this is just the beginning. lets look at it.

1
2
3
4
5
6
7
8
9
10
11
12

example_db/.eunit λ cat example_db.COVER.html
<html>
<head><title>.eunit/example_db.COVER.html</title></head><body bgcolor=white text=black>
<pre>
File generated from /private/tmp/example_db/.eunit/example_db.erl by COVER 2012-03-29 at 19:13:21

****************************************************************************

        |  -module(example_db).
        |  -behaviour(gen_server).
        |
...(and more)

dependencies

Last thing i want to mention is dependencies, i love this feature from rebar. You can add dependencies and rebar will do all the magic for you :). just open your rebar.config and add thme like this:

1
2
3
4
5
6
7
8
9
10

%%-*- mode: erlang -*-

{erl_opts, []}.
{cover_enabled, true}.

{deps,
  [
    {eredis, ".*", {git, "https://github.com/wooga/eredis.git", "HEAD"}}
  ]
}.

Whenever you will type rebar get-deps he will download all dependencies and install them into deps directory. This makes developing applications using things like mochiweb really easy!

Summary

I love this tool, it makes learning and development in Erlang much easier and more rewarding experience. I hope this help you a bit :). Cheers!

Recently i decided to move my blog from tumblr.com to octopress engine because it is just easier for me to maintain it and it looks nicer. The old blog is under http://no-fucking-idea.tumblr.com. My first post on new blog is dedicated to using redis with erlang.

Eredis

Wooga created a really nice (performance driven) redis driver for erlang. You can get it here https://github.com/wooga/eredis. It is really easy and nice.

Initial sample

On project page you can find simple examples how to use eredis. Examples there are all you need (but i need something for front page of my new blog so i will rewrite them and explain them :) ).

First you need to start your eredis application

1

{ok, Client} = eredis:start_link().

Client is the “connection / state” we will be using with rest of queries.

To query things with redis we will use q method from eredis module which takes “Connection / Client” state and list with params. This api is very simple here are two most basic examples of get and set. GET:

1

{ok, <<"OK">>} = eredis:q(Client, ["SET", "name", "kuba"]).

and SET:

1

{ok, <<"kuba">>} = eredis:q(Client, ["GET", "name"]).

From my point of view this is ideal candidate to be turned into gen_server behavior. We will pass “Connection / Client” as state and also we will build some “key” serialization methods around it to make it more durable and make our life easy if we will decide to refactor it later on.

Free Api wrapper

First thing i saw during development using Erlang is that you get free api if you follow simple patterns and encapsulate things into gen_server’s and applications.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

-module(example_db).
-behaviour(gen_server).

-author("jakub.oboza@gmail.com").
-define(Prefix, "example").

-export([start_link/0]).
-export([init/1, handle_call/3, handle_cast/2, terminate/2, handle_info/2, code_change/3, stop/1]).
-export([get_script/2, save_script/3]).

% public api

start_link() ->
  gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

init([]) ->
  {ok, Redis} = eredis:start_link(),
  {ok, Redis}.

stop(_Pid) ->
  stop().

stop() ->
    gen_server:cast(?MODULE, stop).

%% public client api

get_script(Api, Method) ->
  gen_server:call(?MODULE, {get_script, Api, Method}).

save_script(Api, Method, Script) ->
  gen_server:call(?MODULE, {save_script, Api, Method, Script}).

%% genserver handles

handle_call({get_script, Api, Method}, _From, Redis) ->
  Response = eredis:q(Redis, [ "GET", get_key(Api, Method) ]),
  {reply, Response, Redis};

handle_call({save_script, Api, Method, Script}, _From, Redis) ->
  Response = eredis:q(Redis, ["SET", get_key(Api, Method), Script]),
  {reply, Response, Redis};

handle_call(_Message, _From, Redis) ->
  {reply, error, Redis}.

handle_cast(_Message, Redis) -> {noreply, Redis}.
handle_info(_Message, Redis) -> {noreply, Redis}.
terminate(_Reason, _Redis) -> ok.
code_change(_OldVersion, Redis, _Extra) -> {ok, Redis}.

%% helper methods

get_key(Api, Method) ->
  generate_key([Api, Method]).

generate_key(KeysList) ->
  lists:foldl(fun(Key, Acc) -> Acc ++ ":" ++ Key end, ?Prefix, KeysList).

% tests

-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").
-endif.


-ifdef(TEST).

generate_key_test() ->
  Key = generate_key(["one", "two", "three"]),
  ?assertEqual("example:one:two:three", Key).

server_test_() ->
  {setup, fun() -> example_db:start_link() end,
   fun(_Pid) -> example_db:stop(_Pid) end,
   fun generate_example_db_tests/1}.

generate_example_db_tests(_Pid) ->
  [
    ?_assertEqual({ok,<<"OK">>}, example_db:save_script("jakub", "oboza", <<"yo dwang">>) ),
    ?_assertEqual({ok,<<"yo dwang">>}, example_db:get_script("jakub", "oboza") )
  ].

-endif

Public Api

This code listing has two important parts first at top it starts at line 26. This is the public API which will be used by developer. This is this free api. Later on i will explain how to change redis to mongodb and probably to other db engines without doing changes in rest of our app. From my perspective this is awesome feature. In most cases when i had to make app scale problem of having code that was glues to one db engine was heavy.

eunit tests

At line 60. starts the declaration of tests, using rebar and eunit is very easy and it is always good to have test coverage in case of refactoring. I’m a fan of test driven development so i like to cover in tests first things that i will use or i think they might be error prone. Here is used “test generators” to just to show how to write tests for gen_server.

Rebar

Before i will explain more i need to say little about rebar. It is a command line tool that was developed by basho to help create app. it is by far the best thing i found learning erlang to help me do boring stuff and eliminate a lot of rage writing app.src files. To get rebar simply do (you can always go to https://github.com/basho/rebar to get most up to date informations about building it)

1
2
3

λ  git clone git://github.com/basho/rebar.git
λ  cd rebar
λ  ./bootstrap

I use my own set of zsh scripts so all i did to add it to my path was to edit .furby file in my home dir. I strongly suggest also adding it to $PATH just to make your life easier.

Back to example_db!

To create app using rebar you just need to

1
2
3
4
5
6

λ mkdir example_db
λ rebar create-app appid=example_db
==> example_db (create-app)
Writing src/example_db.app.src
Writing src/example_db_app.erl
Writing src/example_db_sup.erl

This command created src folder with scaffold of application OTP pattern and supervisor thats almost all we need :). Now you can compile it using rebar compile and run tests using rebar compile eunit in out app currently we will see

1
2
3
4
5
6
7
8

λ rebar compile eunit
==> example_db (compile)
Compiled src/example_db_app.erl
Compiled src/example_db_sup.erl
==> example_db (eunit)
Compiled src/example_db_app.erl
Compiled src/example_db_sup.erl
  There were no tests to run.

Nothing to do because its empty. Lets add our db module. But before this we need to add dependencies for eredis module. Lets create rebar.config file and add it.

1
2
3
4
5
6
7
8
9
10
11
12

λ emacs rebar.config
λ cat rebar.config
%%-*- mode: erlang -*-

{erl_opts, []}.
{cover_enabled, true}.

{deps,
  [
    {eredis, ".*", {git, "https://github.com/wooga/eredis.git", "HEAD"}}
  ]
}.

Now just run rebar get-deps to get all dependencies downloaded. After adding our example_db.erl into src directory we can run rebar compile eunit to compile and run tests. We have added {cover_enabled, true} in rebar.conf so also test code coverage will be generated for us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

λ rebar compile eunit
==> eredis (compile)
==> example_db (compile)
Compiled src/example_db.erl
==> eredis (eunit)

=ERROR REPORT==== 28-Mar-2012::22:19:35 ===
** Generic server <0.263.0> terminating
** Last message in was {tcp,#Port<0.4516>,
                            <<"*3\r\n$7\r\nmessage\r\n$3\r\nfoo\r\n$2\r\n12\r\n">>}
** When Server state == {state,"127.0.0.1",6379,<<>>,100,#Port<0.4516>,
                               {pstate,undefined,undefined},
                               [<<"foo">>],
                               {#Ref<0.0.0.4058>,<0.180.0>},
                               {[{message,<<"foo">>,<<"11">>,<0.263.0>},
                                 {message,<<"foo">>,<<"10">>,<0.263.0>},
                                 {message,<<"foo">>,<<"9">>,<0.263.0>},
                                 {message,<<"foo">>,<<"8">>,<0.263.0>},
                                 {message,<<"foo">>,<<"7">>,<0.263.0>},
                                 {message,<<"foo">>,<<"6">>,<0.263.0>},
                                 {message,<<"foo">>,<<"5">>,<0.263.0>},
                                 {message,<<"foo">>,<<"4">>,<0.263.0>},
                                 {message,<<"foo">>,<<"3">>,<0.263.0>}],
                                [{message,<<"foo">>,<<"2">>,<0.263.0>}]},
                               10,exit,need_ack}
** Reason for termination ==
** max_queue_size
  All 53 tests passed.
Cover analysis: /private/tmp/example_db/deps/eredis/.eunit/index.html
==> example_db (eunit)
Compiled src/example_db.erl
  All 3 tests passed.
Cover analysis: /private/tmp/example_db/.eunit/index.html

All seems to be fine! lets create file called start.sh to test it out

1
2

λ cat start.sh
erl -pa ebin -pa deps/*/ebin

and make it executable with chmod +x start.sh

And lets rock ‘n’ roll

1
2
3
4
5
6
7
8
9
10

 λ ./start.sh
Erlang R15B (erts-5.9) [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9  (abort with ^G)
1> example_db:start_link().
{ok,<0.33.0>}
2> example_db:save_script("example", "script", "puts '2+2'").
{ok,<<"OK">>}
3> example_db:get_script("example", "script").
{ok,<<"puts '2+2'">>}

Have fun :) Hope it was useful. You can download code for this blog post here https://github.com/JakubOboza/example_db-code

Huh ^___^

that was my first post on new blog ;)