hubertlepicki.com

Amazon SimpleDB is schema-free, non-relational database engine developed by Amazon and available as paid service as one of Amazon Web Services. It’s simple to use, fast and, combined with EC2 gives you great scalability out of the box.

One of biggest problems I had to face, when developing applications that use SimpleDB backend, is that I couldn’t run SimpleDB locally. No one can. It’s not available to purchase or download, you can’t do “apt-get install amazon-simpledb”. Using remote database for development is not a thing you want to do. It’s slow when used from outside of EC2 network. You probably need separate account for each of your developers. This sucks even more if you realize that you pay for running your tests… Hey, we want to run them as much as we can in BDD style – don’t want to be bothered by CPU usage it generates.

Solution to my troubles appeared to be M/Gateway’s M/DB. This is free and open source database, compatible with most recent (2009-04) API of SimpleDB, and you can run it on your own hardware. While it is possible to compile it from source, you probably need to pay developers for support on info how to do it ;). You won’t find also any packages for Ubuntu or Debian on the Internet (but if you do, please let me know!). The easiest way to get it up and running locally is to grab VMWare image and run it locally. Images and documentation how to set it up can be found on M/DB’s site. I run it on VirtualBox, which is my favourite among desktop virtualisation technologies, and have customized image a bit, but it’s equally easy to use it it VMWare. I’ll try to write detailed instructions on how to set up M/DB in VirtualBox some time next week on this blog. You also need to create 2 databases on your M/DB instance, and save access key id an secret access keys of those.

When you have M/DB installed locally, you need M/DB adapter for Ruby. Hey, didn’t I say it’s compatible with Amazon SimpleDB? So, existing adapters should work. Sort of.

The one I am using is Appoxy’s AWS combined with Appoxy’s SimpleRecord. First library is fork of abandoned RightScale’s AWS. I have made simple modifications to it, and now, you get M/DB support out of the box in master Git branch. You’ll need to manually build and install gem until nev version is released (which I show below). Second library is ActiveRecord pattern implemented on top of SimpleDB. Not as big, powerful and famous as Rails’ ActiveRecord, but good enough to provide object-oriented abstraction for your data for daily use.

To get started with these, install required gems:

# Until we have appoxy-aws version 1.11.36, we need to build gem manually:

$ git clone git://github.com/appoxy/aws.git
Initialized empty Git repository in /home/hubert/aws/.git/
remote: Counting objects: 377, done.
remote: Compressing objects: 100% (296/296), done.
remote: Total 377 (delta 198), reused 116 (delta 53)
Receiving objects: 100% (377/377), 167.74 KiB | 108 KiB/s, done.
Resolving deltas: 100% (198/198), done.
$ cd aws
$ gem build aws.gemspec 
WARNING:  no rubyforge_project specified
WARNING:  description and summary are identical
  Successfully built RubyGem
  Name: aws
  Version: 1.11.36
  File: aws-1.11.36.gem
$ sudo gem install aws
Successfully installed aws-1.11.36
1 gem installed
Installing ri documentation for aws-1.11.36...
Installing RDoc documentation for aws-1.11.36...

# in in future we'll just do:
# gem install appoxy-aws --source http://gems.github.com

# install simple-record
$ gem install appoxy-simple_record --source http://gems.github.com

Create your new M/DB Rails project:

$ rails mdb && cd mdb

Put these two lines in your config/environment.rb:

  config.gem "aws", :lib => "right_aws"
  # uncomment line below when version 1.11.36 is ready on github, and remove line above
  # config.gem "appoxy-aws", :lib => "right_aws", :source => "http://gems.github.com"
  config.gem "appoxy-simple_record", :lib => "simple_record", :source => "http://gems.github.com"

Now, you need to connect to your local M/DB instance when starting Rails. Edit your config/environment.rb to contain:

SimpleRecord.establish_connection(
  YOUR_KEY_ID_HERE,
  YOUR_SECRET_ACCESS_KEY_HERE,
  :mode => :per_thread,
  :server => IP_OF_YOUR_VIRTUAL_MACHINE,
  :protocol => "http",
  :port => 80,
  :signature_version => 2,
  :service => "/mdb/request.mgswi"
)

I advise you use app_config or AmberBit Config to use different keys for development and test environments, which will result in using different databases.

In order to check if your database is set up properly, let’s create a model “Brick”:

class Brick < SimpleRecord::Base
  has_attributes :weight, :color
end

There’s no migrations. To create table, for bricks, start up console and type:

$ ./script/console

Loading development environment (Rails 2.3.4)
>> Brick.create_domain
=> {:box_usage=>"0.0010000000", :request_id=>"eb4e3424-6371-89c4-330c-0c808d09ec52"}
>>

Hooray! Now, let’s see some more operations:

>> Brick.create(:weight => "1kg", :color => "red")
=> #, @attributes={"weight"=>["1kg"], "id"=>"1a9d944c-afaa-11de-9808-00a0d1a202b3", "color"=>["red"], "updated"=>[Sat, 03 Oct 2009 01:19:59 +0200], "created"=>[Sat, 03 Oct 2009 01:19:59 +0200]}, @new_record=false>
>> Brick.create(:weight => "0.5kg", :color => "brown")
=> #, @attributes={"weight"=>["0.5kg"], "id"=>"23111360-afaa-11de-9808-00a0d1a202b3", "color"=>["brown"], "updated"=>[Sat, 03 Oct 2009 01:20:13 +0200], "created"=>[Sat, 03 Oct 2009 01:20:13 +0200]}, @new_record=false>
>> Brick.find(:all).first.color
=> "red"
>> Brick.find(:all).last.weight
=> "0.5kg"

Cool, huh?

References

• Appoxy’s AWS
• Appoxy’s SimpleRecord
• M/DB
• Amazon SimpleDB

I have been using Ruby on Rails since the 1.1 release, and there have been more and less successful releases since. Rails 1.2 was amazing and brought us REST - most of us never looked back. However, the 1.2 series was still very buggy and was breaking backward compatibility several times between minor releases.

I wasn’t too much excited about Rails 2.0, particularly because it wasn’t backward-compatible and most of plugins had to wait for several months to be rewritten to work with 2.0. However, since then, Rails is evolving quite nicely and version 2.3 seems to be best you could ever get in Ruby web applications development. And here is why:

• nested model mass-assignment significantly simplifies building complicated forms. At the time of writing, it’s still a bit buggy when it comes to validation (in RC1), but that is about to change before final release.

• http digest authentication which is great news for everyone who build web services and need a bit more security in place.

• metal, using which we can finally bypass routing monster and save some milliseconds for “mission-critical” requests.

• overall performance boost, as pointed out by one of my friends. I didn’t see any benchmarks yet but overall feeling, especially with environment startup and running rake tasks is astonishing.

Now, let’s just wait for 3.0 revolution to come…

This might be useful for these present at today’s BGUL meeting, but not only. Basically, I have demonstrated how to develop simple script useful for sysadmins, that checks if web pages are up and running, and stores results in MySQL database.

Additional cleanup, error handling and comments added (in Polish, sorry).

Full code is here

#!/usr/bin/ruby
#
# G?�wny plik programu
# utuchamianie:
# $ ruby main.rb
# lub
# $ ./main.rb

# Do??czamy nasze biblioteki
require 'config'
require 'site_test'

# i bibliotek? do obs?ugi MySQL
require 'mysql'

begin
  # Najpierw ??czymy si? z baz? danych
  connection = Mysql::init()
  # wpiszcie swoje dane poni?ej (has?o tu jest puste, ale jak trzeba to nie zapomnijcie)
  connection.connect("localhost", "root", "", "tester")

  for url in Config.instance.urls # w p?tli sprawd? wszystkie strony
    # przetestuj URL
    puts "Testuje... " +url
    test = SiteTest.new(url)
    puts Time.new.strftime('%Y-%m-%d %H:%M:%S')
    connection.query("INSERT into site_tests VALUES ('', '#{Time.new.strftime('%Y%m%d%H%M%S')}', '#{url}', '#{test.code}', '#{test.message}')")
  end

rescue Mysql::Error => e # odpowiednik sekcji "catch" z Javy/C++
  puts "Oops, mamy error bazy danych: "+e
ensure # wykonaj zawsze -> patrz "finally" w javie
  connection.close()
end

require "yaml"
require "singleton"

# Daje dost?p do danych zapianych w pliku konfiguracyjnym
class Config
  include Singleton

  attr_reader :urls, :timeout

  def initialize
    # Pliki YAML przegl?da si? podobnie do plik�w XML przy pomocy XPath
    options = YAML::parse( File.open( "configuration.yml" ) )
    @urls = options.select("/urls/*").collect { |el| el.value }
    @timeout = options.select("/timeout")[0].value.to_i # konwersja stringa do integera
  end
end

require 'net/http'
require 'uri'
require 'config'

# Klasa ta przy inicjalizacji testuje dost?pno?? strony, zapisuj?c we w?a?ciwo?ciach "code" i "message" status
class SiteTest

  attr_accessor :code, :message

  def initialize(url) # konstruktor z parametrem
    begin
      adres = URI.parse(url)
      path = "/"
      path = adres.path if adres.path != ""
      req = Net::HTTP::Get.new(path)
      res = Net::HTTP.start(adres.host, adres.port) {|http|
        http.read_timeout = http.open_timeout = Config.instance.timeout
        http.request(req)
      }

      @code = res.code
      @message = res.message
    rescue Timeout::Error => e
      @code = 0
      @message = "Timeout of #{Config.instance.timeout} seconds exceeded"
    rescue SocketError =>e
      @code = -1 # tak oznaczmy sobie "inny b??d", np. nie ma takiego hosta
      @message = e.to_s
    end

  end
end

# To jest zwyk?y plik YAML
urls:
    - http://www.wp.pl
    - http://slashdot.org
    - http://gazeta.pl
    - http://bgul.org
    - http://microsoftruby.com

timeout: 10