RubyGems memory reduction...

Steve Longdo — Sat, 12 Aug 2006 07:00:00 +0000

I did gain some insight into the number of Gem::GemPathSearcher instances through my efforts. They correspond nicely for each piece of Rails that is loaded; ActionMailer, ActiveSupport, ActiveRecord, ActionPack, ActionWebService and Railties. Each of these has the require 'rubygems' statement in it. I believe this is why I was seeing multiple instances of Gem::GemPathSearcher.

As explained before instances of these take up an amount of memory proportional to the number of gems/gem versions in a repository. Making Gem::GemPathSearcher a Singleton of course, resolves more than one instance being loaded into memory. This has been submitted as a patch to RubyGems [ticket #5394].

Installing a patched version of RubyGems 0.9.0 on TextDrive to demonstrate the fix with Typo proved to be the most frustrating part. I will write a follow up posting on exactly how to do this in the near future. Hopefully they will release an updated RubyGems soon!

Shared hosts will definitely want to patch their RubyGems installs to reduce their memory consumption across all of their Rails users. TextDrive in particular should buy me a present or something :-)

I've learned a great deal about the Ruby memory ecosystem by digging into this issue with RubyGems. I got help from some folks on the Rails mailing list but far and away I want to thank Stef Telford for the help. Mad props to you for putting up with all of my questions and ramblings!

Finally some numbers:
Gem::Dependency x330(57415)
Gem::Version::Requirement x693(61939)
Gem::SourceIndex x1(1632488)
Gem::GemPathSearcher2 x1(1647806) <-- Gem::GemPathSearcher x5(7901012)
Hash x1219(1789928)
Gem::Specification x362(1795089)
String x113992(3876455)
Array x7415(5080185)
TOTAL == 16967273 <-- TOTAL == 28496875

Update: These memory numbers are lower than actual process size. They vary by number of Modules loaded as well as Objects counts. Not all Objects and Modules can be evaluated for memory size. The new routing code in the latest version of Rails may also account for some of the difference as well. So grain of salt...

"RubyGems memory reduction..." by Steve Longdo

Wed, 20 Sep 2006 02:48:03 +0000

I updated RubyGems from SVN (r1073) and it resolves the issue. Sent you an email as well :-)

"RubyGems memory reduction..." by Jim Weirich

Mon, 18 Sep 2006 18:28:26 +0000

Steve, I've updated RubyGems to use a mutex while creating the Gem::GemPathSearcher. Could you see if it solves your problem? You can get a beta version with the fix from:

   gem update --system --source http://onestepback.org/betagems

Please let me know the results. Thanks for tracking this down. -- Jim Weirich

"RubyGems memory reduction..." by Ben Sinclair

Sat, 12 Aug 2006 14:26:19 +0000

Nice work, Steve!

Musings of a Trained Monkey: RubyGems memory reduction...

RubyGems memory reduction...

"RubyGems memory reduction..." by Steve Longdo

"RubyGems memory reduction..." by Jim Weirich

"RubyGems memory reduction..." by Ben Sinclair