You are currently browsing the category archive for the ‘Gems’ category.

Just finished presenting Tap at this year’s RubyConf.  Here are the slides.

Tap –[Not] a Talk About Replacing Rake

I ran into trouble when I tried turning rake into something it is not. Rake is a build program designed for build-like tasks; rake is not a platform for general-purpose task libraries. Given it’s design goals, rake very sensibly does not facilitate extensive documentation (who needs it to compile something),  inputs (although this has changed somewhat), configuration, testing, or distribution.

It’s also a dependency-based system; workflows constructed by rake are synthesized in reverse — it’ll be you, not your program, that gets forked when you try to make an imperative workflow. It’s simply the nature of rake! Rake is an excellent build program, but these types of things are in a different domain.

Tap (Task Application)

Tap was originally designed as a simple workflow engine, but it’s evolved into a general-purpose framework for creating configurable, distributable task libraries. Tap tasks can be defined in much the same way as a Rake task:

  # Goodnight::manifest your basic goodnight moon task
  # Prints the input with a configurable message.
  Tap.task 'goodnight', {:message=> 'goodnight'} do |task, input|
    task.log task.message, input
    "#{task.message} #{input}"

Tap pulls documentation out of task declarations to generate manifests:

  % tap run -T
    goodnight # your basic goodnight moon task
    dump # the default dump task
    rake # run rake tasks

And help:

  % tap run -- goodnight --help
  Goodnight -- your basic goodnight moon task
    Says goodnight with a configurable message.
  usage: tap run -- goodnight NAME

          --message MESSAGE a goodnight message

      -h, --help Print this help
          --name NAME Specify a name
          --use FILE Loads inputs from file

Tasks are immediately available to run with inputs and configurations:

  % tap run -- goodnight moon
    I[00:09:55] goodnight moon

  % tap run -- goodnight moon --message hello
    I[00:10:01] hello moon

Task declarations define classes which naturally support namespaces, subclassing and testing.   When the shorthand declaration is not enough, task classes can be defined in the standard way:

  # Hello::manifest a hello world task
  # A more complicated hello world task illustrating
  # config blocks and a full task class definition.
  class Hello < Tap::Task

    config :greeting, 'hello', &c.string    # a greeting string
    config :reverse, false, &c.flag         # maps to a flag

    def process(name)
      message = reverse ? greeting.reverse : greeting

      log message, name
      "#{message} #{name} result"

  task =
  task.process('world')     # => "hello world result"
  task.reverse = true
  task.process('world')     # => "olleh world result"
  task.greeting = :symbol   # !> ValidationError

Configurations map to methods and can utilize a validation/transformation block.  Tap defines a number of common blocks (ex c.integer, c.regexp, etc.) that may also imply metadata for the command line (ex c.flag):

  % tap run -- hello world --reverse
  I[20:04:33]              olleh world


Tap supports distribution of tasks as gems. To illustrate, say we installed the sample_tasks gem. Now our manifest looks like this:

  % tap run -T
    goodnight   # your basic goodnight moon task
    hello       # a hello world task
    concat      # concatenate files with formatting
    copy        # copies files
    grep        # search for lines matching a pattern
    print_tree  # print a directory tree
    dump        # the default dump task
    rake        # run rake tasks

Tap checks the installed gems for a ‘tap.yml’ configuration file or a ‘tapfile.rb’ task file; any gems with one (or both) of these files gets pulled into the execution environment. Now tasks can be specified either by a short name when there isn’t a name conflict (ex goodnight, print_tree), or by a full name that includes the environment (ex sample:goodnight, sample_tasks:print_tree).

  % tap run -- sample_tasks:print_tree .
  |- Rakefile
  |- lib
  |- sample.gemspec
  |- tapfile.rb
  `- test
      |- tap_test_helper.rb
      |- tap_test_suite.rb
      `- tapfile_test.rb

Not bad, eh?

Workflows and the Roadmap

Tap support simple workflows in the imperative style. Tasks can be assigned an on_complete block that executes when the task completes, allowing results to be examined and new tasks to be enqued as needed.

  app = Tap::App.instance
  t1 = Tap.task('t1') {|t| 'hellO'}
  t2 = Tap.task('t2') {|t, input| input + ' woRld' }
  t3 = Tap.task('t3') {|t, input| input.downcase }
  t4 = Tap.task('t4') {|t, input| input.upcase }
  t5 = Tap.task('t5') {|t, input| input + "!" }

  # sequence t1, t2
  app.sequence(t1, t2)

  # fork t2 results to t3 and t4
  app.fork(t2, t3, t4)

  # unsynchronized merge of t3 and t4 into t5
  app.merge(t5, t3, t4)


  app.results(t5)       # => ["hello world!", "HELLO WORLD!"]

True support for workflows from the command line is lacking right now, but it will be coming soon. Here’s a short list of what else is planned:

  • Global Rake Tasks (hopefully!). I should be able to find and load rakefiles using the Tap execution environment. Tap already allows you to incorporate local rake tasks into a workflow using the ‘rake’ task.
  • Tap server. In a small-group environment where some people are computer savvy and others aren’t, it would be really useful to serve up your tasks using a web interface.
  • More test support. Tap provides several modules for testing tasks and supporting common types of tasks (ex file transformation tasks). Currently the modules are a bit incomplete, and they’re only geared towards Test::Unit. I’d like to add support for RSpec in the future.

This last summer I finished the joinfix gem providing a solution to the fixture join problem — mainly that it’s a pain to create fixtures by specifying entry ids across multiple fixture files. When Mike Clark and Chad Fowler opened up submissions for Advanced Rails Recipies, I sent JoinFix to them and it was accepted.

I took a look at the new Foxy Fixtures in Rails 2.0 and was shocked/pleased to find something similar has gotten incorporated directly into the core. Goes to show how problematic fixtures really were. JoinFix still is interesting, however, mainly because JoinFix lets you define entries inline.

Consider the following data model:

  class User < ActiveRecord::Base  has_many :user_groups
    has_many :groups, :through => :user_groups

  class Group < ActiveRecord::Base
    has_many :user_groups
    has_many :users, :through => :user_groups

  class UserGroup < ActiveRecord::Base
    belongs_to :user
    belongs_to :group

You can write your fixtures using the naming scheme you lay out in your models, referencing entries across multiple fixture files (similar to Foxy) or you can define them inline:

     full_name: John Doe
     groups: admin_group   # => reference to the 'admin_group' entry

     full_name: Jane Doe
     groups:               # => you can specify an array of entries if needed
       - admin_group
       - worker_group:     # => inline definition of the 'worker_group' entry
           name: Workers

   admin_group:            # => the referenced 'admin_group' entry
     id: 3                 # => you can (but don't have to) specify ids
     name: Administrators

Join entries implied in your definition, as in a has_and_belongs_to_many association, will be created and named by joining together the names of the parent and child, ordered by the ’<’ operator. For example, the users.yml and groups.yml fixtures produce these entries:

    id: 1                  # => primary keys are assigned to all entries
    full_name: John Doe
    id: 2
    full_name: Jane Doe

    id: 3
    name: Administrators
    id: 1
    name: Workers

    id: 1
    user_id: 1             # => references are resolved to their foreign keys
    group_id: 3            # => explicitly set primary keys are respected
    id: 2
    user_id: 2
    group_id: 3
  jane_doe_worker_group    # => Notice the '<' operator in action
    id: 3
    user_id: 2
    group_id: 1

Nesting is allowed. This will make the same entries as above:

    full_name: John Doe
        id: 3
        name: Administrators
            full_name: Jane Doe
                name: Workers

In this final form, JoinFix defines a highly-involved fixture in one chunk, in one file. This can be a BIG advantage when you try to test some cross-table, complicated lookup. The full fixture is centralized and easy to manage.