Create a Ruby gem with Zeitwerk as a development-only dependency (tutorial)

Forget require statements and make your gem lightweight at the same time

Image by Alexa from Pixabay

For the impatient

If you know what Zeitwerk is, understand the problem, and just want to do what the title of this post promises, jump straight into the tutorial.

Otherwise, keep on reading.

The problem

If you work on a non-Rails Ruby project, managing code loading may be challenging. Following the Single Responsibility Principle means having many specialized classes instead of one doing all the work. And if you keep each class in a separate file you quickly end up with a bunch of files that need to be loaded. That’s a lot of sad require statements hiding in your files. You can almost hear their mischievous giggles when you rename a class or move it into a different namespace. It may sound similar to:

cannot load such file -- my_perfect_class (LoadError)

Those exceptions make me angry. And Ruby was meant to be optimized¹ for the programmer’s happiness, right?

Inspired by The Nightmare (1791) by Henry Fuseli, oil on canvas 101.6 × 126.7 cm, Detroit Institute of Arts

Enter Zeitwerk

Well, there are solutions. One of the best, in my opinion, is Zeitwerk, described as an “Efficient and thread-safe code loader for Ruby”.

You probably heard about it, as it is used by Rails and Hanami. Zeitwerk is the reason you don’t need to worry about require statements when working with Rails apps. Just follow the intuitive convention of naming files the same as the classes they contain.

Using Zeitwerk in your own gem

The good news is: Zeitwerk is not limited to Rails. You can use it in any Ruby project, including any gem you create.

Just put it into Gemspec, and add a few lines of code. That’s it, you are now free to focus on features instead of code loading.

The bad news is: your gem now depends on Zeitwerk. This means each project using your gem depends on it too. Some people may not like it and decide to not use your awesome gem!

That’s because it’s generally a good practice to minimize dependencies. The benefits include security, maintainability, and stability.

Does it mean you have to choose between ease of development and dependency minimalization?

Inspired by The Choice of Heracles (1596) by Annibale Carracci, oil on canvas 239 cm × 165 cm, National Museum of Capodimonte

Have your cake and eat it too

Can you use Zeitwerk for development, but not include it as a runtime gem dependency?

I asked myself this question while working on cryptreboot, a gem that allows a machine with an encrypted disk to reboot by requesting a passphrase beforehand, rather than after the reboot.

Rebooting Linux with encrypted disk

I was not able to find an answer on Google. ChatGPT also failed. Therefore I decided to do it by myself.

In this tutorial, I will show you how to create a gem from the ground up. It will use Zeitwerk in development, while not depending on it in runtime. And it will still work.

Conventions

The first character of a line in code blocks has a special meaning:

• $ means the remaining part should be executed in the shell,

• > means the remaining part should be executed in IRB.

In other cases, the block contains the actual code or result of an action you performed.

I assume you use a Unix-based OS such as Linux distribution (may be run in WSL), *BSD, or macOS. If you run Windows, you may need to adjust some shell commands.

Start with an empty gem

We will call our gem zeitgeist. Execute in the console:

$ bundle gem zeitgeist
$ cd zeitgeist

Now adjust the Gemspec to make the gem buildable. Open zeitgeist.gemspec file and:

1. Fill summary, homepage, source_code_uri, and changelog_uri.

2. Delete the line containing the description.

The results should look similar to this (I stripped comments):

require_relative "lib/zeitgeist/version"

Gem::Specification.new do |spec|
  spec.name = "zeitgeist"
  spec.version = Zeitgeist::VERSION
  spec.authors = ["Pawel"]
  spec.email = ["pepawel@users.noreply.github.com"]

  spec.summary = "Awesome gem using Zeitwerk only for development."
  spec.homepage = "https://github.com/pepawel/zeitgeist"
  spec.license = "MIT"
  spec.required_ruby_version = ">= 2.6.0"

  spec.metadata["allowed_push_host"] = "TODO: Set to your gem server 'https://example.com'"

  spec.metadata["homepage_uri"] = spec.homepage
  spec.metadata["source_code_uri"] = spec.homepage
  spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/master/CHANGELOG.md"

  spec.files = Dir.chdir(__dir__) do
    `git ls-files -z`.split("\x0").reject do |f|
      (f == __FILE__) || f.match(%r{\A(?:(?:bin|test|spec|features)/|\.(?:git|travis|circleci)|appveyor)})
    end
  end
  spec.bindir = "exe"
  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
  spec.require_paths = ["lib"]
end

You should spend some more time on Gemspec if you work on a real gem. The above example minimizes changes needed for building, but it’s not production-ready.

Install dependencies:

$ bundle config set --local path .bundle/gems # for easy cleanup
$ bundle install

Let’s commit the changes and check if you can build the gem:

$ git add .
$ git commit -m 'Initial commit'
$ rake build

It should succeed. However, if there is an error it should be self-explanatory. Most probably you will need to adjust Gemspec.

Add some logic

Place the following code in lib/zeitgeist/programmer.rb:

module Zeitgeist
  class Programmer
    def happy?
      # Ruby was created in 1993
      Time.now.year >= 1993 ? 'yes' : 'no'
    end
  end
end

To check if it works, run IRB in the context of the gem by executing in the terminal:

$ bin/console

When you try to access Zeitgeist::Programmer, you will see it’s not loaded:

> Zeitgeist::Programmer
(irb):1:in `<main>':
  uninitialized constant Zeitgeist::Programmer (NameError)
  from bin/console:15:in `<main>'

To make it work, you will need to manually require the correct file:

> require 'zeitgeist/programmer'
=> true
> Zeitgeist::Programmer
=> Zeitgeist::Programmer

But we want it to be required automatically by Zeitwerk. In our example above, it is enough to simply require the correct file. But with your codebase becoming larger, the benefits of using auto-loader will become more and more visible.

Enable Zeitwerk

Add the Zeitwerk to your Gemfile and install it:

$ bundle add zeitwerk

Gems added to the gem’s Gemfile are available in development only, so Zeitwerk won’t become a runtime dependency.

The next step is to make sure your code uses Zeitwerk. Adjust lib/zeitgeist.rb to include the following lines before the Zeitgeist module definition:

require "zeitwerk"
loader = Zeitwerk::Loader.for_gem
loader.setup

Now run the console again and check if your code is being auto-loaded:

$ bin/console
> Zeitgeist::Programmer.new.happy?
=> "yes"

It works, so we can commit our code changes:

$ git add lib Gemfile*
$ git commit -m 'Add business logic and setup autoloader'

However, it won’t work when installed on a system without Zeitwerk. The gem will install normally but will raise an exception on first use because it silently depends on Zeitwerk.

Basic loader

We need a lightweight loader for runtime. Its task would be to load all the Ruby files in gem’s lib/ directory. Why not loop over the files and simply require each one? It’s because the order of loading files matters. In the loop approach, we may end up loading file B depending on file A before file A is loaded.

To make sure we use the correct loading order, let’s create the loader file manually in lib/basic_loader.rb:

# Load every project file in one place
require 'zeitgeist/programmer'

Runtime vs development

We want to use a basic loader for runtime and Zeitwerk for development. Therefore we need to distinguish between those two.

As Gemfile is used only in development, we could use it for this purpose. Add the following code to the beginning of that file:

module ::Zeitgeist # :: is used to escape from Gemfile's scope
  AUTOLOADERS = []
end

It simply sets a constant in the main module of our gem. It will be defined only in development.

You may notice it duplicates the module definition from the zeitgeist.rb. It’s perfectly fine as the modules in Ruby can be reopened as many times as you like.

Now, let’s adjust lib/zeitgeist.rb to decide which loader to use based on the constant we defined:

require 'zeitgeist/version'

if defined? Zeitgeist::AUTOLOADERS
  require 'zeitwerk'
  Zeitgeist::AUTOLOADERS << Zeitwerk::Loader.for_gem.tap do |loader|
    loader.ignore("#{__dir__}/basic_loader.rb")
    loader.setup
  end
else
  require 'basic_loader'
end

module Zeitgeist
  class Error < StandardError; end
  # Your code goes here...
end

The basic_loaded.rb file doesn’t match the Zeitwerk convention - it doesn’t define the BasicLoader constant. Therefore, to avoid warnings we add it to the ignored files as you see above.

The other change was to save the Zeitwerk instance into AUTOLOADERS constant. We will use it later.

Let’s commit our changes, build the gem, and install it locally:

$ git add lib Gemfile
$ git commit -m 'Fix code loading'
$ rake build
$ gem install --user pkg/zeitgeist-0.1.0.gem

As our gem doesn’t contain any executables, you can safely ignore the following warning if it appears:

WARNING:  You don't have /home/user/.gem/ruby/3.0.0/bin in your PATH,
          gem executables will not run.

Now we can test if the gem works:

$ irb
> require 'zeitgeist'
> Zeitgeist::Programmer.new.happy?
=> "yes"
> defined? Zeitwerk
=> nil

It works, and the last line tells us Zeitwerk is not used.

Now let’s verify if development works too:

$ bin/console
> Zeitgeist::Programmer.new.happy?
=> "yes"
> defined? Zeitwerk
=> "constant"

It works too, but uses Zeitwerk just as we like.

However, we need to manually update lib/basic_loader.rb each time we add, remove, rename, or move the files. Why not automate it?

Autogenerate basic loader

Zeitwerk does its magic using Module#autoload. It guarantees the loading order is valid. We can use Zeitwerk to produce lib/basic_loader.rb file containing all the require statements in the correct order.

To generate that file, we will use Zeitwerk’s on_load callback which gets called every time file is loaded. We need to get a list of all files, therefore we will force Zeitwerk to load the entire codebase by using #eager_load method.

Put the following into bin/loader:

#!/usr/bin/env ruby
# frozen_string_literal: true

require 'pathname'
require 'stringio'

class LoaderGenerator
  def call
    writer do |out|
      out.puts <<~HEADER
        # frozen_string_literal: true

        # File generated automatically, do not edit

      HEADER
      yield.each do |auto_loader|
        auto_loader.on_load do |_cpath, _value, abspath|
          next if abspath !~ /.rb$/i # skip directories

          out.puts "require '#{path_to_requirement(abspath)}'"
        end
        auto_loader.eager_load
      end
    end
    true
  end

  private

  def writer(&block)
    output ? block.call(output) : File.open(loader_path, 'w', &block)
  end

  def path_to_requirement(abspath)
    relative_path_from(loader_dir, abspath).sub(/.rb$/i, '')
  end

  def relative_path_from(base_dir, target_path)
    target = Pathname.new(target_path)
    base = Pathname.new(base_dir)
    target.relative_path_from(base).to_s
  end

  attr_reader :loader_path, :loader_dir, :output

  def initialize(loader_path, output = nil)
    @loader_path = loader_path
    @loader_dir = File.dirname(loader_path)
    @output = output
  end
end

class LoaderValidator
  def call(&block)
    StringIO.new.tap do |buffer|
      LoaderGenerator.new(loader_path, buffer).call(&block)
      buffer.rewind
    end.read == current
  end

  private

  def current
    File.read(loader_path)
  end

  attr_reader :loader_path

  def initialize(loader_path)
    @loader_path = loader_path
  end
end

require 'bundler/setup'

loader_path = File.join(__dir__, '..', 'lib', 'basic_loader.rb')

klass = ARGV[0] == 'generate' ? LoaderGenerator : LoaderValidator
action = klass.new(loader_path)
result = action.call do
  require 'zeitgeist'
  Zeitgeist::AUTOLOADERS
end
exit 1 unless result

Now make the file executable and run it:

$ chmod +x bin/loader
$ bin/loader generate

It will:

1. Initialize Zeitwerk and the main module of our gem.

2. Configure Zeitwerk to add require line to the basic loader each time the Ruby file is loaded.

3. Tell Zeitwerk to eagerly load all the code. As a result, lib/basic_loader.rb becomes populated.

Inspired by The Creation of Adam (~1511) by Michelangelo, fresco painting, 230.1 cm × 480.1 cm, Sistine Chapel

The file lib/basic_loader.rb should look like this:

# frozen_string_literal: true

# File generated automatically, do not edit

require 'zeitgeist/programmer'

Ensure the built gem contains current files

If you read the bin/loader code above carefully, you probably noticed LoaderValidator class. It gets activated if generate argument was not specified. It generates a basic loader in memory and compares it with the current. If they don’t match, the script exits with an error.

We will use it to protect us from forgetting to regenerate the basic loader before building the gem.

Let’s add those lines to the end of Rakefile:

Rake::Task.define_task :validate_loader do
  abort "Basic loader is stale, run `bin/loader generate` to fix" unless system("bin/loader validate")
end

Rake::Task[:build].enhance [:validate_loader]

We create a new task called validate_loader which raises an exception if the basic loader is stale. Afterward, we add it as a dependency to the build task, so it gets called before.

From now on running rake build with stale basic loader will fail with the message:

Basic loader is stale, run `bin/loader generate` to fix

Tests: which loader to use?

It makes more sense to use the basic loader instead of Zeitwerk for tests. What if we generate the basic loader incorrectly due to some error? If tests use Zeitwerk, the basic loader is bypassed and we won’t see any problems.

As we use RSpec for our gem, let’s add this line to the beginning of spec/spec_helper.rb:

raise "Failed to regenerate basic loader" unless system "bin/loader generate"

This way before each test, the basic loader will be regenerated automatically.

If you prefer to decide when the basic loader file changes, you can use this instead:

raise "Basic loader is stale, run `bin/loader generate` to fix" unless system "bin/loader validate"

You will be notified when the basic loader needs to be updated. This approach allows you to have full control over updating this file.

Run the tests

We should have a working infrastructure, let’s run the tests:

$ rake
... 
Zeitgeist
  has a version number
  does something useful (FAILED - 1)
...

The second test failed. It’s an example test that is meant to fail. Let’s change it to something related to our logic by replacing spec/zeitgeist_spec.rb with:

RSpec.describe Zeitgeist do
  it "has a version number" do
    expect(Zeitgeist::VERSION).not_to be nil
  end

  it "checks if programmer is happy" do
    expect(Zeitgeist::Programmer.new.happy?).to eq("yes")
  end
end

Run the tests again:

$ rake
...
Zeitgeist
  has a version number
  checks if programmer is happy
...

All green! Let’s commit our changes, build the gem and install it:

$ git add lib spec bin/loader Rakefile
$ git commit -m 'Make sure basic loader is autogenerated'
$ rake build
$ gem install --user pkg/zeitgeist-0.1.0.gem

Final run

We can check if the gem works after installation:

$ irb
> require 'zeitgeist'
> Zeitgeist::Programmer.new.happy?
=> "yes"
> defined? Zeitwerk
=> nil

It works, and it still doesn’t require Zeitwerk.

Now let’s test if Zeitwerk works in development. First, let’s create a random constant:

$ echo "Zeitgeist::Pi = 3.1337" > lib/zeitgeist/pi.rb

This file is not required anywhere. Let’s check if it autoloads:

$ bin/console
> Zeitgeist::Pi
=> 3.1337

It works. From now on, you can easily add new files to your code base and do not worry about require statements. Congratulations :)

Cleanup

To restore your environment to its previous state, you will need to uninstall the gem:

$ gem uninstall zeitgeist

You can also remove the directory containing the repository you were working on. Apart from the code, all the gems you installed including Zeitwerk are located there.

Limitations

Compared to using Zeitwerk, this approach has some limitations. I believe in most cases, minor code adjustments would do. As with every refactoring, good test coverage will help. However, if your project is large and/or you use advanced code-loading features, implementation of this approach may be challenging.

Those limitations were identified by the author of Zeitwerk, Xavier Noria. He gave me awesome feedback which resulted in a major update to the article (the old version could be found here). Thank you, Xavier!

Here are the limitations I’m currently aware of:

Conditional code loading

An example would be to use one module on Linux, and another on macOS:

module Foo
  include Mac if mac?
  include Linux if linux?
end

If OS-specific modules are defined in separate files, the basic loader will be generated differently depending on the developer’s OS.

Effectively gem developed on Linux will crash on macOS and vice-versa.

Delayed code loading

The developer may choose to delay the loading of some code parts. As the method presented in this post depends on eager loading, it won't include those parts. This will lead to crashes in runtime.

Circular references

Let’s assume we have foo.rb:

module Foo
  include Bar
end

And foo/bar.rb:

module Foo::Bar
end

Zeitwerk can handle this, while the generated basic loader will crash.

Implicit namespace definitions

Zeitwerk allows you to define Zeitgeist::Foo::Bar class without defining Zeitgeist::Foo module first. If you want to use the approach presented in this post, you need to define it explicitly.

Final notes

You learned how to use Zeitwerk in gem development while not adding it as a runtime dependency. Here you will find the repository with the gem we just built together.

I encourage you to apply this knowledge to your own project. The code I published here uses a permissive MIT license, so you can use it even in your commercial work.

Also, there are nearly 450 gems depending on Zeitwerk. I think many of them would accept pull requests minimizing runtime dependencies. If you feel brave, go contribute to your favorite gem to make the Open Source world a better place!

Please don’t get me wrong. Zeitwerk is a rock-solid, actively developed gem. I highly respect the developers behind it and I am grateful to them. But I still believe limiting your dependencies is beneficial. Source: xkcd

The code used in this tutorial was extracted from the cryptreboot gem. This little project is very close to my heart, so forgive me I mentioned it for a second time in this post. If you use a Linux system with an encrypted disk, give cryptreboot a try :)

And thank you for reading this post :) You are the best!

If you have any feedback, please leave it in a comment below.

1. At this Google Tech Talk in 2008, Matz said, “I hope to see Ruby help every programmer in the world to be productive, and to enjoy programming, and to be happy. That is the primary purpose of Ruby language.”