A gem is a simple way to distribute functionality, it can be a small plugin, a Ruby library or sometimes a whole program. Thanks to RubyGems, a gem hosting service, developers have a wide range of gems at their disposal allowing them to easily add functionality to their applications.

But what if there is no gem available that will suit the functionality you need, and you find yourself writing the same code over and over again for different projects? Well, in that case you should consider making your own gem.

It’s considered a good practice to extract a gem out of an existing application, since that way you will have a better understanding of all the requirements as well as how the gem will be used. This blog post will illustrate just that on a real life example, and will take you through the process of creating a slug_converter gem.

Slug converter gem

Source code for slug_converter gem was developed while working on a link shortener application, in order to generate a string consisting of predefined characters, based on a given id number of a link. As it will be described in this blog post, this code was easily extracted from the application into an independent gem that was released on RubyGems.

Although it may seem like a complex task at first, creating a gem is not that difficult, if you have RubyGems and Bundler installed you are good to go. We already know what RubyGems is, and Bundler is a package manager that determines a full set of direct dependencies needed by your application.

Now let’s build a gem!

Creating a gem

First step is to make sure that bundler gem is installed.

1
$ gem install bundler

once bundler is installed creating a structure for your new gem is easy,

1
$ bundle gem slug_converter

The first time you use bundler to create a gem you will be prompted to answer a couple of questions:

1
2
3
4
Do you want to include code of conduct in your gems you generate?
Do you want to licence your code permissively under the MIT licence?
Do you want to generate tests with your gem?
Type rspec or minitest to generate those tests files now and in the future:

Answering these questions will help bundler configure and setup files that are being generated now and in the future. Here we answered yes to first 4 qestions and choose rspec for testing.

Running $ bundle gem slug_converter command resulted with “slug_converter” directory with essential gem file structure being created, and git repository initialized, assuming that you are using git for version management (as you should).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Creating gem 'slug_converter'...
 create  slug_converter/Gemfile
 create  slug_converter/.gitignore
 create  slug_converter/lib/slug_converter.rb
 create  slug_converter/lib/slug_converter/version.rb
 create  slug_converter/slug_converter.gemspec
 create  slug_converter/Rakefile
 create  slug_converter/README.md
 create  slug_converter/bin/console
 create  slug_converter/bin/setup
 create  slug_converter/LICENSE.txt
 create  slug_converter/.travis.yml
 create  slug_converter/.rspec
 create  slug_converter/spec/spec_helper.rb
 create  slug_converter/spec/slug_converter_spec.rb

Let’s go through files that bundler generated for us, .gemspec file is the “heart” of your gem so lets start with slug_converter.gemspec

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
lib = File.expand_path('../lib', __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require 'slug_converter/version'

Gem::Specification.new do |spec|
  spec.name         = "slug_converter"
  spec.version      = SlugConverter::VERSION
  spec.authors      = ["Your Name"]
  spec.email        = ["youremail@example.com"]

  # if spec.respond_to?(:metadata)
  #   spec.metadata['allowed_push_host'] = "TODO: Set to 'http://mygemserver.com' to prevent pushes to rubygems.org, or delete to allow pushes to any server."
  # end

  spec.summary      = %q{Number <-> Slug converter}
  spec.description  = %q{Generates a slug based on the given number and the other way around}
  spec.homepage     = "https://github.com/orangeiceberg/slug_converter"
  spec.license      = "MIT"

  spec.files        = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
  spec.bindir       = "exe"
  spec.executables  = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
  spec.require_paths= ["lib"]

  spec.add_development_dependency "bundler", "~> 1.8"
  spec.add_development_dependency "rake", "~> 10.0"
end

This file contains metadata about your gem and it can be populated directly, so here you can enter all the data such as name, description, licence… This file also contains information about what files should be packaged in your gem, as well as the load path to include the gem directory when the gem is first loaded. Most of these default settings will work for the majority of gems but you can always edit them if you want different behavior. At the bottom of the file add any gem dependencies that are required.

The version number of the gem is set in SlugConverter::VERSION constant which is kept in a separate version.rb file, and you can change it there for every new version of your gem.

lib
 |--slug_converter
         |--version.rb

A very important part of every gem is the README file, where you can describe how to install and use the gem, and the LICENCE file where you can define the terms and conditions under which the gem can be used.

In the lib directory there is a file which has the same name as your gem (recommended), and that file will be loaded when someone requires your gem. If the gem you are writing is simple all the code can be placed in this single file, or in case of more complex gems all the other files from the lib directory are required in this file.

There is also a Gemfile generated, but this file doesn’t have to be managed directly since all it does is look in .gemspec for required dependencies and then loads them through bundler. All the dependencies required by the gem should be specified in the .gemspec file.

Another file that is generated by the bundler is Rakefile which just adds some gem tasks from bundler, and we can see those tasks with explanation by running

1
2
3
4
rake -T
rake build    # Build slug_converter-0.0.1.gem into the pkg directory
rake install  # Build and install slug_converter-0.0.1.gem into system gems
rake release  # Create tag v0.0.1 and build and push slug_converter-0.1.0.gem to Rubygems

Writing tests

If you are following the principles of Test Driven Development you will probably like to start by writing tests for you gem, for that purpose I would suggest using RSpec. To do that you will need to add rspec as a development dependency to you gemspec file:

spec.add_development_dependency 'rspec'

As mentioned in the beginning, when running bundle gem for the first time, bundler will asks if you would like to generate test files for your gem and to choose if you want to use rspec or minitest. If you answer with yes, and choose rspec, bundler will generate a spec directory with two files:

|-- spec
      |-- slug_converter_spec.rb
      |-- spec_helper.rb

In the spec_helper.rb file you can reference any test globals or configuration.

Since we are extracting code from an existing application we already have all the tests written so we just need to copy them into the generated spec/slug_converter_spec.rb file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
require 'spec_helper'
describe SlugConverter do
  it 'has a version number' do
    expect(SlugConverter::VERSION).not_to be nil
  end
  describe ".number" do

    it "returns number when number is set" do
      converted_slug= SlugConverter.new(111)
      expect(converted_slug.number).to eq(111)
    end

    it "returns decoded number for existing slug" do
      converted_slug = SlugConverter.new("vg")
      expect(converted_slug.number).to eq(363)
    end

  end

  describe ".number" do

    it "sets number to given value" do
      converted_slug = SlugConverter.new(211)
      expect(converted_slug.number=210).to eq(210)
    end

    it "sets slug to encoded value of number" do
      converted_slug = SlugConverter.new(211)
      converted_slug.number=210
      expect(converted_slug.slug).to eq("pb")
    end

    it "sets number to integer value of given number passed as string" do
      converted_slug = SlugConverter.new("210")
      expect(converted_slug.number).to eq(210)
    end

    it "sets slug to encoded value of given number passed as string" do
      converted_slug = SlugConverter.new("210")
      expect(converted_slug.slug).to eq("pb")
    end

    it "sets number to integer value of argument that starts with a number but also contains letters" do
      converted_slug = SlugConverter.new("210jj")
      expect(converted_slug.number).to eq(210)
    end

    it "sets slug to encoded value of argument that starts with a number but also contains letters" do
      converted_slug = SlugConverter.new("210jj")
      expect(converted_slug.slug).to eq("pb")
    end

  end

  describe ".slug" do

     it "returns slug when slug is set" do
        converted_slug = SlugConverter.new("hy")
        expect(converted_slug.slug).to eq("hy")
     end

     it "returns encoded slug when link id is set" do
        converted_id = SlugConverter.new(113)
        expect(converted_id.slug).to eq("hy")
     end

  end

  describe ".slug" do

    it "sets slug to given value" do
      converted_slug = SlugConverter.new("ezk")
      expect(converted_slug.slug=("ebk")).to eq("ebk")
    end

    it "sets number to decoded value of slug" do
      converted_slug = SlugConverter.new("pb")
      converted_slug.slug=("ezk")
      expect(converted_slug.number).to eq(1483)
    end

    it "raises Arrgument Error exception if given value is an empty string" do
      expect { SlugConverter.new("") }.to raise_error(ArgumentError)
    end

    it "raises Arrgument Error exception if given value is nil" do
      expect { SlugConverter.new(nil) }.to raise_error(ArgumentError)
    end

    it "raises Arrgument Error exception if given value contains unpermitted letters" do
      expect { SlugConverter.new("iiii") }.to raise_error(ArgumentError)
    end

    it "raises Arrgument Error exception if given value starts with letter but contains numbers" do
      expect { SlugConverter.new("bb12") }.to raise_error(ArgumentError)
    end

  end
end

To make rspec rake task available we will setup tasks folder where we’ll place our rspec.rake file containing only 2 lines:

1
2
require 'rspec/core/rake_task'
RSpec::Core::RakeTask.new(:spec)

and then we will import this file in our Rakefile that bundler provided automatically:

1
Dir.glob('tasks/**/*.rake').each(&method(:import))

Now run:

1
bundle exec rake spec

And watch your tests fail. :)

Add gem functionality

Now we need to make those test go green. To do that we will again copy the existing code from our application in the main gem file lib/slug_converter.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
require "slug_converter/version"
require 'set'
require 'gem_config'

class SlugConverter
  include GemConfig::Base
  with_configuration do
    has :alphabet, default: "qjeghxtrpnfmdzwvsybkuoca"
  end

  def initialize(number_or_slug)
     @alphabet = SlugConverter.configuration.alphabet.split(//)
    if number_or_slug.to_i != 0
      @number = number_or_slug.to_i
    elsif validate_string(number_or_slug)
      @slug = number_or_slug.downcase
    else
      raise ArgumentError, 'Argument must be integer value or non-empty string consisting of predefined letters'
    end
  end

  def number
    if @number.nil?
      @number = bijective_decode
    else
      @number
    end
  end

  def number=(new_number)
    @number = new_number
    @slug = bijective_encode
    @number
  end

  def slug
    if @slug.nil?
      @slug = bijective_encode
    else
      @slug
    end
  end

  def slug=(new_slug)
    @slug = new_slug
    @number = bijective_decode
    @slug
  end

  private

    def bijective_encode
      id = @number
      return @alphabet[0] if id == 0
      s = ''
      base = @alphabet.length
      while id > 0
        s << @alphabet[id.modulo(base)]
        id /= base
      end
      s.reverse
    end

    def bijective_decode
      i = 0
      base = @alphabet.length
      @slug.each_char { |c| i = i * base + @alphabet.index(c) }
      i
    end

    def validate_string(slug)
      unless slug.nil?
        alphabet = Set.new @alphabet
        slug_letters = Set.new slug.downcase().split(//)
        slug != "" && (slug_letters.subset? alphabet)
      end
    end
end

Now when we run the tests again, they should all pass.

Making your gem configurabile

In order to allow users to set their own alphabet that will be used by the SlugConverter, we needed to make our gem configurabile. To do this we used https://github.com/krautcomputing/gem_config gem.

You will notice this code at the begining of the SlugConverter class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class SlugConverter
  include GemConfig::Base

  with_configuration do
    has :alphabet, default: "qjeghxtrpnfmdzwvsybkuoca"
  end

  def initialize(number_or_slug)
     @alphabet = SlugConverter.configuration.alphabet.split(//)
     # ...
  end  

  # rest of the code omitted    
end

this code along with spec.add_runtime_dependency 'gem_config' added as a dependency in slug_converter.gemspec file, alows us to make the gem configureabile.

Custom aphabet can than be defined by adding config/initializers/slug_converter.rb to your application, and defining the alphabet like this:

1
SlugConverter.configuration.alphabet = "your_custom_alphabet_here"

Releasing your gem

Now that we have the test passing and all the code in place it’s time to make the gem available for everyone to use by releasing it on RubyGems, to do that you will need to have a RubyGems account. If this is the first time you release a gem you will be prompted to enter your RubyGems username and password. You will also need to have your repository setup on Github.

Then with just one comand:

1
$ bundle exec rake release
  • your code will be pushed to your Github repository,
  • your git repository will be tagged with the version number using a name like “v1.0.0”.
  • your gem released on RubyGems.

The ruby gem described in this blog post can be found here https://rubygems.org/gems/slug_converter, and all the code is in this GitHub repository https://github.com/orangeiceberg/slug_converter .

For our new project it was necessary to modify the starting id of our database. This can be handled through migration for creating table but we decided to create a rake task that handled this for us.

The rake task that we created detects what database is being used and executes appropriate changes according to that.

You can create a rake task using rails generate command for rake task:

1
rails g task namespace task_name

This will create your task in lib/tasks with chosen namespace and task name.

Here is our task and an explanation that follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
namespace :database do
  desc "Detect database that's being used and then increment its id"
  task autoincrement: :environment do

    db_name_downcase = ActiveRecord::Base.connection.adapter_name.downcase

    if Link.maximum(:id).to_i < 1000
      if db_name_downcase.start_with? "mysql"
        ActiveRecord::Base.connection.execute("ALTER TABLE links AUTO_INCREMENT = 1000")
      end
      if db_name_downcase.start_with? "postgres"
        ActiveRecord::Base.connection.execute("ALTER SEQUENCE links_id_seq START with 1000 RESTART;")
      end
      if db_name_downcase.start_with? "sqlite"
        ActiveRecord::Base.connection.execute("insert into sqlite_sequence(name,seq) values('links', 1000)")
      end
    else
      puts "To perform this task your database shouldn't have records with id number higher than 1000"
    end

  end
end

We need to change the starting id of our database to 1000 so we check that we don’t have a record with id higher than 1000. Link is our Active Record model and links is the name of our table.

ActiveRecord::Base.connection returns the connection currently associated with the class. We use it to detect the name of database and execute appropriate changes.

MySQL

For MySQL we need to set AUTO_INCREMENT value to 1000, Auto-increment allows a unique number to be generated when a new record is inserted into a table. When first record is created it sets its primary key to 1 by default and it will auto increment by 1 for each new record.

PostgreSQL

For Postgres we have to explain what a sequence is. A sequence is a special kind of a database object designed for generating unique numeric identifiers. It is typically used to generate artificial primary keys. Sequences are similar to the Auto-increment concept in MySQL.

SQLite

For SQlite we altered sqlite_sequence table, which is an internal table used to implement AUTOINCREMENT. It is created automatically whenever any ordinary table with an AUTOINCREMENT integer primary key is created.

You can check this Stack Overflow discussion that was very helpful to me.

If you are using Devise gem for authentication and you have been adding custom fields to your model you’ll get in trouble when you try to create a new instance or update an existing one. All your added fields will be treated as unpermitted. The solution for this problem is to customise Devise’s configure_permited_parameters action. All you need to do is to add this action to your Application controller and push parameters that need to be permitted to devise_paremeter_sanitizer array. So let’s say you have a User Model and you have added company_name and website fields to your user’s table, to permit this parameters on sign_up you need to add this to your Application controller:

1
2
3
def configure_permitted_parameters
  devise_parameter_sanitizer.for(:sign_up).push(:company_name, :website)
end

It is the same principle for the :sign_in and :edit_account. You can see what are default permitted parameters here.

Devise has a very useful Trackable module used to track user’s sign in count, timestamps and IP address. There are some occasions when you need to disable tracking. For example for API requests where user signs in on every request; for instances where admin might sign in as an user; and similar.

To disable Devise Trackable module you need to set request.env["devise.skip_trackable"] = true. You have to do that before trying to authenticate user, so you’ll want to put it in a before_filter, or even better prepend_before_filter to make sure it’s before authentication.

Add this to your controller in which you want to disable tracking:

1
2
3
4
5
6
prepend_before_filter :disable_devise_trackable

protected
  def disable_devise_trackable
    request.env["devise.skip_trackable"] = true
  end

Note to self: here is how to upgrade Ubuntu 8.04 LTS (or any other release that is no longer supported) to newer Ubuntu release.

When you are upgrading unsupported release of Ubuntu if you try to do the usual sudo apt-get update it will most likely fail because… well, it’s unsupported. The simple fix for this is to change your /etc/apt/sources.list and replace repository URLs from something like us.archive.ubuntu.com to old-releases.ubuntu.com.

After that you should be able follow normal upgrade procedure (use sudo if you are not root):

1
2
3
apt-get update
apt-get install update-manager-core
do-release-upgrade

References:

Here is a quick way to setup VirtualBox using Vagrant with Heroku-like box on Mac.

  1. Install VirtualBox from https://www.virtualbox.org/wiki/Downloads

  2. Install Vagrant from http://downloads.vagrantup.com/

  3. Create Vagrantfile for Heroku-like box (based on https://github.com/ejholmes/vagrant-heroku) that looks something like:

1
2
3
4
5
6
    Vagrant.configure("2") do |config|
        config.vm.box = "heroku"
        config.vm.box_url = "https://dl.dropboxusercontent.com/s/rnc0p8zl91borei/heroku.box"
      config.vm.synced_folder ".", "/vagrant", :nfs => true
        config.vm.network :private_network, ip: "192.168.1.42"  # required for NFS
    end

Beside telling Vagrant to use Heroku-like box from https://github.com/ejholmes/vagrant-heroku it also sets up shared dir between host and VM machine. It will mount Vagrantfile dir (.) to /vagrant in VM.

vagrant up will setup the VM and start it up.

Now you can use vagrant ssh to login to VM.

Vagrant Heroku-like box comes with Postgresql, but if you want you can easily setup sqlite:

1
sudo apt-get install libsqlite3-dev

Bonus tip: when you are working on multiple projects sometimes you can forget which VMs are running. You can list all running VMs using:

1
VBoxManage list runningvms

Further reading:

RailsDiff is a very useful site when upgrading Rails versions (for example, from Rails 3.2 to Rails 4). It will generate default Rails app using two different Rails versions and it will compare them. The result is that you can see all the configuration changes (like in application.rb) and all other changes – which is really useful when upgrading to new Rails version.

Assume that you have the usual setup with model (MyFile) using simple Carrierwave uploader (MyFileUploader):

# app/models/my_file.rb
class MyFile < ActiveRecord::Base
  mount_uploader :file, MyFileUploader
end

To be able to test Carrierwave uploaders with RSpec using FactoryGirl factories you need:

  • define factory with uploaded file
  • modify test environment storage so test file uploads are separated from other uploads
  • turn off image processing to speed up tests
  • perform cleanup after each test

Define factory

# spec/factories/my_files.rb
FactoryGirl.define do
 factory :my_file do
   photo Rack::Test::UploadedFile.new(File.open(File.join(Rails.root, '/spec/fixtures/myfiles/myfile.jpg')))
 end
end

Setup Carrierwave

First we need to make sure Carrierwave is using local file system for storage and to disable file processing for testing environments. Disabling file processing will speed up tests considerably. We can do that by adding following to Carrierwave initializer:

if Rails.env.test? || Rails.env.cucumber?
  CarrierWave.configure do |config|
    config.storage = :file
    config.enable_processing = false
  end
end

Next we should separate test uploads from any other uploads. We can do that by modifying cache_dir and store_dir methods for all Carrierwave models (i.e. all models that are descendants of CarrierWave::Uploader::Base). So the whole Carrierwave initializer looks something like:

# config/initializers/carrierwave.rb
if Rails.env.test? || Rails.env.cucumber?
  CarrierWave.configure do |config|
    config.storage = :file
    config.enable_processing = false
  end

  # make sure our uploader is auto-loaded
  MyFileUploader

  # use different dirs when testing
  CarrierWave::Uploader::Base.descendants.each do |klass|
    next if klass.anonymous?
    klass.class_eval do
      def cache_dir
        "#{Rails.root}/spec/support/uploads/tmp"
      end

      def store_dir
        "#{Rails.root}/spec/support/uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
      end
    end
  end
end

Clean up uploaded files

Using factory defined above will create uploaded files in cache_dir and store_dir. These are just temporary files and should be removed after each test, so each of them has a clean slate. By adding after :each hook in RSpec configuration block we can remove these files simply by deleting spec/support/uploads dir.

# spec_helper.rb
RSpec.configure do |config|
  config.after(:each) do
    if Rails.env.test? || Rails.env.cucumber?
      FileUtils.rm_rf(Dir["#{Rails.root}/spec/support/uploads"])
    end 
  end
end

Here are some random Rails tips I’ve found useful:

  • rails console sandbox – if you open console like this it will rollback all database changes once you exit. Pretty useful for playing around without making any changes to database.
  • rake db:migrate:status – useful when you want to see the status of current database. It will show the status of all migrations.
  • User.pluck(:email) – since Rails 3.2.1 you can use pluck method to get an array of values in one particular column. It’s the equvivalent of doing User.select(:email).map(&:email)

Setting up Paperclip to use Amazon’s S3 is as simple as setting :storage => :s3 and providing right credentials to Paperclip by setting :s3_credentials option. Best way to provide S3 credentials is to use an YML file (usually config/s3.yml) which allows you to set different credentials for each environment. For example:

# config/s3.yml
development:
  access_key_id: XYZXYZXYZ
  secret_access_key: XYZXYZXYZ
  bucket: mygreatapp-development
production:
  access_key_id: XYZXYZXYZ
  secret_access_key: XYZXYZXYZ
  bucket: mygreatapp-production

Of course you want to treat s3.yml same as database.yml – i.e. you don’t want to track it with git and you want for each person/server to have it’s own.

However, consider this: you are working on Open Source app in a public git repository and you are deploying it on Heroku. Heroku doesn’t allow you to create files (unless they are in git repository) and you can’t commit s3.yml with your credentials to public repository.

One solution is to define different :s3_credentials hash in one of the environment files or to load different YML file for each environment and generate hash from it. Downside is that you need to have a separate YML file for each environment and/or you need to convert YML to hash. Other solution could be to have separate local branch from which you will push to Heroku. Problem with this is that you have to have a local branch for deploying. This means if there are multiple developers who deploy to production each should have separate local branch.

Much simpler way to deploy Paperclip with different S3 credentials for each environment (with one of the environment being deployed on Heroku; and repository being public) is to create s3.yml file as usual (and don’t commit it to git), but define values only for local environment.

For production deployment on Heroku you can write initializer which will set :s3_credentials from ENV variables.

# initializers/s3.rb
if Rails.env == "production"
  # set credentials from ENV hash
  S3_CREDENTIALS = { :access_key_id => ENV['S3_KEY'], :secret_access_key => ENV['S3_SECRET'], :bucket => "sharedearth-production"}
else
  # get credentials from YML file
  S3_CREDENTIALS = Rails.root.join("config/s3.yml")
end

# in your model
has_attached_file :photo, :storage => :s3, :s3_credentials => S3_CREDENTIALS

and you can easily set persistant ENV vars on Heroku with:

$ heroku config:add S3_KEY=XYZXYZ S3_SECRET=XYZXYZ

(according to Heroku docs)