Musings of a Trained Monkey

I’ve been toying with switching over to Mephisto and spending some extra time studying memory consumption, which is quite a bit higher in Mephisto than Typo. Using the latest svn trunk of RubyGems (r1100), the bug with multiple Gem::GemPathSearcher instances being created has been fixed, but now there is a different problem. Multiple Gem::SourceIndex instances are created. I can see where this might be needed with the recent “incremental updates to the gem list” change that went into RubyGems, it doesn’t make sense to have these hanging around in memory for the lifetime of a Rails application.

I took the same approach to a fix as last time, adding the require and include for Singleton into the SourceIndex.rb class. This worked fine for loading up Mephisto and indeed reduced the memory use substantially.

r1100 without Singleton	– Gem::SourceIndex x2(1062363)
r1100 with Singleton	– Gem::SourceIndex x1(97621)

I haven’t checked to see how this change impairs RubyGems for getting new gems, update, etc. but it seems like there needs to be a different amount of information loaded for bootstrapping the gem command, versus a running Ruby or Ruby on Rails application.

• Does the running application really have a need to know the complete dependency hierarchy for all of the installed gems on a system?

• Even if your application did, would you want that information stored into memory from startup to shutdown?

Eric Hodel has announced the release of mem_inspect and png gems. Ryan Davis has an excellent movie assembled showing output from these gems. The beauty of the code behind mem_inspect, like looking at the sun, will make your eyes water if you look at it too long. Seriously though everyone should check it out and gain some Ruby I.Q. points. The excellent RubyInline is used internally and there is a rake task that builds a patched version of Ruby from source, wicked cool stuff guys!

I think this should start people thinking about better ways to gain insight into the Ruby memory heap, at least until YARV arrives and changes the playing field altogether.

I did gain some insight into the number of Gem::GemPathSearcher instances through my efforts. They correspond nicely for each piece of Rails that is loaded; ActionMailer, ActiveSupport, ActiveRecord, ActionPack, ActionWebService and Railties. Each of these has the require 'rubygems' statement in it. I believe this is why I was seeing multiple instances of Gem::GemPathSearcher.

As explained before instances of these take up an amount of memory proportional to the number of gems/gem versions in a repository. Making Gem::GemPathSearcher a Singleton of course, resolves more than one instance being loaded into memory. This has been submitted as a patch to RubyGems [ticket #5394].

Installing a patched version of RubyGems 0.9.0 on TextDrive to demonstrate the fix with Typo proved to be the most frustrating part. I will write a follow up posting on exactly how to do this in the near future. Hopefully they will release an updated RubyGems soon!

Shared hosts will definitely want to patch their RubyGems installs to reduce their memory consumption across all of their Rails users. TextDrive in particular should buy me a present or something :-)

I've learned a great deal about the Ruby memory ecosystem by digging into this issue with RubyGems. I got help from some folks on the Rails mailing list but far and away I want to thank Stef Telford for the help. Mad props to you for putting up with all of my questions and ramblings!

Finally some numbers:
Gem::Dependency x330(57415)
Gem::Version::Requirement x693(61939)
Gem::SourceIndex x1(1632488)
Gem::GemPathSearcher2 x1(1647806) <-- Gem::GemPathSearcher x5(7901012)
Hash x1219(1789928)
Gem::Specification x362(1795089)
String x113992(3876455)
Array x7415(5080185)
TOTAL == 16967273                          <-- TOTAL == 28496875

Update: These memory numbers are lower than actual process size. They vary by number of Modules loaded as well as Objects counts. Not all Objects and Modules can be evaluated for memory size. The new routing code in the latest version of Rails may also account for some of the difference as well. So grain of salt...