Musings of a Trained Monkey: Tag rails

ActiveMessaging is still alive...

Steve Longdo — Thu, 04 Jan 2007 20:49:00 +0000

I am happy to see that ActiveMessaging (a13g<-corny) is returning to life. It was announced with a bit of fanfare by Obie in early 2006 and then disappeared.

Jon Tirsen sent out some invites for a new Google Group and introduced Andrew Kuklewicz as the new maintainer of the project. I think this will become the premier way for RoR apps to interact with Java backends in 2007. Take a look at the code and see how you can contribute!

RubyGems and Memory, again...

Steve Longdo — Tue, 21 Nov 2006 22:47:00 +0000

I’ve been toying with switching over to Mephisto and spending some extra time studying memory consumption, which is quite a bit higher in Mephisto than Typo. Using the latest svn trunk of RubyGems (r1100), the bug with multiple Gem::GemPathSearcher instances being created has been fixed, but now there is a different problem. Multiple Gem::SourceIndex instances are created. I can see where this might be needed with the recent “incremental updates to the gem list” change that went into RubyGems, it doesn’t make sense to have these hanging around in memory for the lifetime of a Rails application.

I took the same approach to a fix as last time, adding the require and include for Singleton into the SourceIndex.rb class. This worked fine for loading up Mephisto and indeed reduced the memory use substantially.

r1100 without Singleton	– Gem::SourceIndex x2(1062363)
r1100 with Singleton	– Gem::SourceIndex x1(97621)

I haven’t checked to see how this change impairs RubyGems for getting new gems, update, etc. but it seems like there needs to be a different amount of information loaded for bootstrapping the gem command, versus a running Ruby or Ruby on Rails application.

Does the running application really have a need to know the complete dependency hierarchy for all of the installed gems on a system?

Even if your application did, would you want that information stored into memory from startup to shutdown?

Java, Spring and Why I Haven't Posted Lately...

Steve Longdo — Fri, 06 Oct 2006 03:37:00 +0000

Things have been busy on the work side of my activities lately. Work is currently revolving around Java and using Spring MVC. Since I have primarily focused on Ruby and Rails information on my blog, I've been reluctant to blog about where my head is at in the Java space. If someone is interested, leave a comment and I'll see what I can come up with.

Which isn't to say I am not tracking recent developments in the Rails community, just that I haven't had time to analyze their impact and vomit forth a blog entry or two :-)

Okay, one little morsel of my experience with Spring relates to one of the core tenets of Rails: "Convention over configuration".
It finally found a way into a Java web development framework!

Unspace.ca: The Agony, The Ecstasy

Steve Longdo — Thu, 14 Sep 2006 04:05:00 +0000

Unspace.ca has released some interesting code to the open source community this year. September 13th, 2006 has them unleashing the first Rails plugin code that will make you want to gouge out your own eyes upon viewing the example code.

"The Agony" in the title of this post relates to the release of the HAML plugin. (Somewhat irresponsibly linked to by me, seriously please don't gouge out your eyes) Why, oh why, did they decide to combine white space sensitivity with HTML? Is there a disgruntled postal worker/Python programmer working there or what? Perhaps it is used in some sort of hazing ritual for their new employees.

On the other hand we have "The Ecstasy", the very useful Datagrid. This is an excellent and worthy open source release. It combines functionality with a great example of use. Granted it would be nice if it was packaged as a Rails plugin, but back in March adding routes in a plugin would have been difficult.

Anyway, I am looking forward to see what Unspace.ca will do next, hopefully they will serve up more of "The Ecstasy"

Loose Ends

Steve Longdo — Fri, 25 Aug 2006 04:07:00 +0000

Some loose ends will be tied up here soon. Here is a quick list of things that I intend to post more about soon. Suggestions/requests will be taken from comments as well :-)

AJAX-ed OpenRico Accordion widget
$LOAD_PATH and Revealing some Rails Magic Tricks
Edge Typo's New Pluggable Sidebars
~~Installing RubyGems on TextDrive~~ Done!
Further Exploring the Limits on TextDrive:

[rhesus@davie ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) 40960
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 32768
max memory size (kbytes, -m) 49152
open files (-n) 240
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 16384
cpu time (seconds, -t) 1200
max user processes (-u) 20
virtual memory (kbytes, -v) 81920

RubyGems memory reduction...

Steve Longdo — Sat, 12 Aug 2006 07:00:00 +0000

I did gain some insight into the number of Gem::GemPathSearcher instances through my efforts. They correspond nicely for each piece of Rails that is loaded; ActionMailer, ActiveSupport, ActiveRecord, ActionPack, ActionWebService and Railties. Each of these has the require 'rubygems' statement in it. I believe this is why I was seeing multiple instances of Gem::GemPathSearcher.

As explained before instances of these take up an amount of memory proportional to the number of gems/gem versions in a repository. Making Gem::GemPathSearcher a Singleton of course, resolves more than one instance being loaded into memory. This has been submitted as a patch to RubyGems [ticket #5394].

Installing a patched version of RubyGems 0.9.0 on TextDrive to demonstrate the fix with Typo proved to be the most frustrating part. I will write a follow up posting on exactly how to do this in the near future. Hopefully they will release an updated RubyGems soon!

Shared hosts will definitely want to patch their RubyGems installs to reduce their memory consumption across all of their Rails users. TextDrive in particular should buy me a present or something :-)

I've learned a great deal about the Ruby memory ecosystem by digging into this issue with RubyGems. I got help from some folks on the Rails mailing list but far and away I want to thank Stef Telford for the help. Mad props to you for putting up with all of my questions and ramblings!

Finally some numbers:
Gem::Dependency x330(57415)
Gem::Version::Requirement x693(61939)
Gem::SourceIndex x1(1632488)
Gem::GemPathSearcher2 x1(1647806) <-- Gem::GemPathSearcher x5(7901012)
Hash x1219(1789928)
Gem::Specification x362(1795089)
String x113992(3876455)
Array x7415(5080185)
TOTAL == 16967273 <-- TOTAL == 28496875

Update: These memory numbers are lower than actual process size. They vary by number of Modules loaded as well as Objects counts. Not all Objects and Modules can be evaluated for memory size. The new routing code in the latest version of Rails may also account for some of the difference as well. So grain of salt...

RubyGems: Single/Multi-tons and memorysize

Steve Longdo — Wed, 09 Aug 2006 03:41:00 +0000

There is a line in the RubyGems source code, @gempath_searcher ||= Gem::GemPathSearcher.new, from custom_require.rb that seeks to make Gem::GemPathSearcher a Singleton class. For some reason this isn't sufficient as I have seen as many as five instances of it existing in the same thread during profiling of Typo. I am not sure why more than one gets generated. It may relate to Rails aliasing the Kernel.require method after RubyGems has in custom_require.rb.

Review of the source makes me understand why a Gem::GemPathSearcher instance can be so large, it holds the gem specifications for every gem and version on the load path. Running locally this is not a big deal, as you probably don't have a lot of gems and their versions installed. On a shared host though there are tons of gems and versions of gems installed (350+ on my TextDrive server!).

A permanent solution to fix Gem::GemPathSearcher to truly be a Singleton class would probably be something as simple as checking Object.const_defined?(:GemPathSearcher) to see if the class is already in memory.

A quick alternative would be restricting the gem library path to a gem repository other than the shared host's repository. This can be done by exporting the GEM_HOME variable in your shell. There is also a GEM_PATH variable that can be used to support multiple repositories. Examples of how to work with these variables can be found on the Rails wiki. It should be possible to make a repository containing just the gems your app needs and nothing else. This will reduce the memory footprint of Gem::GemPathSearcher quite a bit, no matter how many instances of them there might be!

Obviously I am still learning the implications of RubyGems working together with Rails apps. I apologize for you the reader being dragged along on my quest to learn. Anyone wishing to bring this quest to a quick end just tell me how it ends.

Update: I will continue my research until we arrive at the truth. I still have some other RubyGems optimizing techniques to apply. My hope is using something like the gemconfigure file described in the documentation will be the difference between the Whole Milk Memory and Skim Milk Memory. Also I will reformat the code in the post to be pretty tomorrow. need to add some stuff to scribbish to accomplish this and I want to go to bed now...

RubyGems dependency management and memory use...

Steve Longdo — Tue, 08 Aug 2006 06:30:00 +0000

I've been looking into the nebulous matter of Rails memory management lately, using Typo4 as my model app. In my research thus far I have noticed that several of the highest memory using items are from the RubyGems library. Particularly the Gem::GemPathSearcher object. After running a few days I have picked up five instances of it taking up nearly 8MB of RAM which is odd because the first time I profiled it there were only four of them. I am not sure why RubyGems needs to be in memory after the app initially loads, especially the ~1.5MB SourceIndex and the 352 Gem::Specification objects for another ~1.6MB. Seems like wiring of any dependencies could be done once at start up. I am not sure if this relates specifically to how Typo uses RubyGems or if all Rails apps incur the memory overhead penalty for RubyGems. If any one with knoweldge of the internal functioning of RubyGems would take the time to educate me on this it would be much appreciated. Also if someone could explain why RubyGems pops up at all since all Typo dependencies are deployed into the vendor folder.

Another one to watch is the slow accumulation of ActiveRecord::ConnectionAdpaters::MysqlColumn, up to 217 instances in memory. Not a particularly large memory hit, but they have steadily grown in number over the last few days. i may swith over to my Postgres database and see if the same thing happens with its connecton adapter.

Finally the allocation of Array objects is interesting. The number of instances in use has gone down by roughly -2000 instances but the memory use of the remaining Arrays has gone up by about 2MB. I need to explode these Arrays and see what they hold and why they stick around so long. Partial output of profiling included below:

Class x852(21194)
Gem::Version x709(24981)
ActiveRecord::ConnectionAdapters::MysqlColumn x217(35410)
Regexp x1343(52312)
Gem::Dependency x309(53743)
Gem::Version::Requirement x662(59168)
Gem::SourceIndex x1(1563056)
Gem::Specification x352(1714117)
Hash x1257(1858255)
String x117204(3869923)
Gem::GemPathSearcher x5(7901012)
Array x6093(11102839)
TOTAL == 28496875

Please do share your experiences with Rails or RubyGems memory use in the comments.

Typo 4.0 and memory reduction...

Steve Longdo — Fri, 04 Aug 2006 22:56:00 +0000

Have you upgraded to the latest Typo release only to find that it hits the per process memory limit on your host? It happened to me.

Here is how I went about reducing the memory footprint of Typo:

Removed sidbars that aren't used from the installation.

Amazon through XBox.

Removed unused requires from environment.rb

In particlular aggregation classes for the Sidebars that were removed.

Make sure to not load any unnecessary parts of Rails (i.e. ActiveWebService) That's right I use the admin screen to post!
Most importantly deleted the sparklines_textfilter.

That last one makes a huge difference in the memory use of Typo. It removes the need to load RMagick saving around ~8MB from your process size. In fact these changes have brought Typo down to about 33MB in memory. Since the average Rails app uses 30MB this isn't too bad, however memory consumption slowly creeps up over a period of days and hits the limit. Seemingly a memory leak. I will be digging into this issue as time allows, but work is keeping me busy.

I will eventually be using a diferent theme but I switched to this one while checking out memory use and haven't spent enough time on my new one yet.

RailsConf2006 DHH keynote...

Steve Longdo — Mon, 26 Jun 2006 18:02:00 +0000

David Heinemeier Hansson's keynote was rather, um CRUD-dy :-)

It mainly focused on the importance of modelling the application domain as well as the relationships between the models, at the correct level of abstraction/encapsulation. This of course is nothing new to anyone who has been doing software development for a while.

The new part comes in with using all of the HTTP "verbs" (POST,GET,PUT,DESTROY) in Rails as a default convention. So for CRUD; Create=POST, Read=GET, Update=PUT, and Delete=DESTROY. Using this convention it is possible to build a higher level abstraction on top, tentatively called ActiveResource. He also discussed the use of respond_to, the HTTP Accept Headers, and even extensions to handle multiple input and output types from a single controller. So the same code that handles a HTML page could be reused as a webservice for example, or for returning a JavaScript representation. Definitely an area to keep an eye on.

All in all he is an excellent presenter. He definitely has a charisma that holds your attention in person. Outside of the material in his presentation he explained some of the reasons why things are the way they are in Rails. In particular the intentionally lackluster support for composite primary keys. Hearing this in person is much more effective than reading about it on a mailing list.