Redis and Phusion Passenger: reconnect on fork

We're nearly ready for a new Coconut production release. Of course, this is the moment when bugs start coming in from the beta stage that are difficult to reproduce. One bug report stated: "sometimes, my widgets do not load". I couldn't reproduce this bug, but it was in the back of my mind the whole week. Suddenly, this afternoon, when I was clicking through a review build, the widgets didn't load. So I immediately pulled the log files and found this Redis error:

Got '1' as initial reply byte. If you're running in a multi-threaded environment, make sure you pass the :thread_safe option when initializing the connection. If you're in a forking environment, such as Unicorn, you need to connect to Redis after forking.

After some research, I've found out that the error was caused by the combination of using Redis and Phusion Passenger. We use Redis as a chained backend store for i18n (see Railscast 256 for  our inspiration), so we can have custom translations for each Coconut instance. Which is a very cool feature, because every customer has his own domain language and we can tweak the translation messages accordingly.

As the error states, Phusion Passenger is a "forking environment" like Unicorn. Phusion Passenger spawns new worker processes when it thinks more processes are needed, it uses what they call a "smart spawning method". Basically, it forks a new thread to increase capacity to handle requests. Normally, the newly created thread will try to use the same Redis connection, which causes problems. What you need to do, is create a new connection when the current process is forked.This is done by creating a new Rails initializer and adding some code for handling Phusion Passenger fork events.

Adding a new Rails initializer is simple: just add an .rb file to config/initializers. Our initializer looks like this:

if defined?(PhusionPassenger)
  PhusionPassenger.on_event(:starting_worker_process) do |forked|
    if forked
      # We're in smart spawning mode. If we're not, we do not need to do anything
      Rails.cache.reset
      I18nRedisBackend.connect
    end
  end
end

You might recognize the Rails.cache.reset from Dalli, which has the same issue if used with Phusion Passenger. The I18nRedisBackend.connect creates a new connection with Redis, like this (note: this code was simplified, to make it more readable):

module I18nRedisBackend
  @@default_i18n_backend = I18n.backend

  def self.connect
    redis = Redis.new
    I18n.backend = I18n::Backend::Chain.new(I18n::Backend::KeyValue.new(redis), @@default_i18n_backend)
  end 
end

To summarize, when Phusion Passenger forks a process to create a new worker thread, it automatically creates a new Redis connection. Problem solved!

thanx to the Phusion Passenger users guide, Appendix C which provided me with the correct code example