RubyGems and Memory, again...

Posted by Steve Longdo Tue, 21 Nov 2006 22:47:00 GMT

I’ve been toying with switching over to Mephisto and spending some extra time studying memory consumption, which is quite a bit higher in Mephisto than Typo. Using the latest svn trunk of RubyGems (r1100), the bug with multiple Gem::GemPathSearcher instances being created has been fixed, but now there is a different problem. Multiple Gem::SourceIndex instances are created. I can see where this might be needed with the recent “incremental updates to the gem list” change that went into RubyGems, it doesn’t make sense to have these hanging around in memory for the lifetime of a Rails application.

I took the same approach to a fix as last time, adding the require and include for Singleton into the SourceIndex.rb class. This worked fine for loading up Mephisto and indeed reduced the memory use substantially.

r1100 without Singleton – Gem::SourceIndex x2(1062363)
r1100 with Singleton – Gem::SourceIndex x1(97621)


I haven’t checked to see how this change impairs RubyGems for getting new gems, update, etc. but it seems like there needs to be a different amount of information loaded for bootstrapping the gem command, versus a running Ruby or Ruby on Rails application.

  • Does the running application really have a need to know the complete dependency hierarchy for all of the installed gems on a system?
  • Even if your application did, would you want that information stored into memory from startup to shutdown?

UPDATE: Applying this change will indeed hamper RubyGems ability to install/update gems. I am now wondering how hard it would be to break Rails’s dependency on RubyGems altogether…

Read more...

Installing a patched RubyGems on TextDrive

Posted by Steve Longdo Thu, 31 Aug 2006 03:35:00 GMT

Tying up the loose end called "Installing a patched RubyGems on TextDrive your shared host". Sorry this took me so long to write up it should have been posted on August 13th. Anyway the full and gory details follow.

Let's get the tarball for RubyGems 0.9.0 and explode it in the home directory on our shared host: 
curl -o rubygems-0.9.0.tgz http://rubyforge.rubyuser.de/rubygems/rubygems-0.9.0.tgz
tar xzvf rubygems-0.9.0.tgz
cd rubygems-0.9.0
The tgz distribution can also be found at RubyForge if you don't have curl. Now we need to patch the file ~/rubygems-0.9.0/lib/rubygems/custom_require.rb inside of the RubyGems distribution to make Gem::GemPathSearcher a Singleton. This is chronicled both on this blog and in the still unloved RubyGems patch 5394, reproduced here for ease of cut and paste, minus all comment lines: 
require 'rubygems/source_index'
require 'singleton'

module Kernel
  alias gem_original_require require
  def require(path)
    gem_original_require path
  rescue LoadError => load_error
    begin
      @gempath_searcher = Gem::GemPathSearcher.instance
      if spec = @gempath_searcher.find(path)
        Gem.activate(spec.name, false, "= #{spec.version}")
        gem_original_require path
      else
        raise load_error
      end
    end
  end
end

module Gem
  class GemPathSearcher
    include Singleton
...

After making this change it is time to install, since we are installing as non-root we have to make a couple of changes to the standard procedure. Let's go ahead and plan on installing to a directory in the home directory named "myrubygems". Also we will need to specify a GEM_HOME for our local repository (using "/gems" where <HOME> is the full path to your home folder) and we will specify the GEM_PATH to point to the shared hosts gem repository. To find where your shared host keeps its gem repository run the command gem environment. TextDrive has it installed in the same place as Ruby which will create a little wrinkle for us later. 
export GEM_HOME=<HOME>/gems
export GEM_PATH=/usr/local/lib/ruby/gems/1.8/

ruby setup.rb config --prefix=<HOME>/myrubygems
ruby setup.rb setup
ruby setup.rb install
You'll need to modify your $PATH variable to include the patched version of RubyGems first in the list. Here is the line from my .bash_profile file: 
export PATH=$HOME/myrubygems/bin:$PATH
 Make sure you add this to your $PATH permanently or when your app restarts it will not use the patched version. At this point we can see how we are doing by typing the command which gem it should list the path to the patched version of RubyGems. If not recheck the previous steps.

Two files left to edit, ~/myrubygems/bin/gem and the RAILS_ROOT/public/dispatch.fcgi. RAILS_ROOT is the path to the rails app you want to pick up the patched RubyGems. Both need the shebang line at the top changed to: 
#!/usr/local/bin/ruby -I<HOME>/myrubygems/lib/ruby/site_ruby/1.8/
#Again <HOME> needs to be the fully expanded version here for this to work
 This is the wrinkle I mentioned earlier. The shebang change is necessary due to the shared host gem repository being installed in the same place as Ruby. The -I option specifies that our patched version of RubyGems be included in the $LOAD_PATH before even Ruby is loaded. I will have more to say about $LOAD_PATH in an upcoming post.
Read more...

RubyGems memory reduction... 3

Posted by Steve Longdo Sat, 12 Aug 2006 07:00:00 GMT

I did gain some insight into the number of Gem::GemPathSearcher instances through my efforts. They correspond nicely for each piece of Rails that is loaded; ActionMailer, ActiveSupport, ActiveRecord, ActionPack, ActionWebService and Railties. Each of these has the require 'rubygems' statement in it. I believe this is why I was seeing multiple instances of Gem::GemPathSearcher.

As explained before instances of these take up an amount of memory proportional to the number of gems/gem versions in a repository. Making Gem::GemPathSearcher a Singleton of course, resolves more than one instance being loaded into memory. This has been submitted as a patch to RubyGems [ticket #5394].

Installing a patched version of RubyGems 0.9.0 on TextDrive to demonstrate the fix with Typo proved to be the most frustrating part. I will write a follow up posting on exactly how to do this in the near future. Hopefully they will release an updated RubyGems soon!

Shared hosts will definitely want to patch their RubyGems installs to reduce their memory consumption across all of their Rails users. TextDrive in particular should buy me a present or something :-)

I've learned a great deal about the Ruby memory ecosystem by digging into this issue with RubyGems. I got help from some folks on the Rails mailing list but far and away I want to thank Stef Telford for the help. Mad props to you for putting up with all of my questions and ramblings!

Finally some numbers:
Gem::Dependency x330(57415)
Gem::Version::Requirement x693(61939)
Gem::SourceIndex x1(1632488)
Gem::GemPathSearcher2 x1(1647806) <-- Gem::GemPathSearcher x5(7901012)
Hash x1219(1789928)
Gem::Specification x362(1795089)
String x113992(3876455)
Array x7415(5080185)
TOTAL == 16967273                          <-- TOTAL == 28496875

Update: These memory numbers are lower than actual process size. They vary by number of Modules loaded as well as Objects counts. Not all Objects and Modules can be evaluated for memory size. The new routing code in the latest version of Rails may also account for some of the difference as well. So grain of salt...

Older posts: 1 2