[Catalyst] Testcase for DBIx::Class::Loader - demonstrates failure under load

[Toby Corkindale]

On Mon, Feb 13, 2006 at 08:31:48AM -0600, Brandon Black wrote:
> On 2/13/06, Toby Corkindale <tjc at wintrmute.net> wrote:
> > Hi,
> > I think DBIx::Class::Loader is demonstrating a similar bug to the one I
> > hit in Class::DBI::Loader some time ago, that was subsequently fixed by
> > Daisuke Maki.
> >
> > I've attached a test case that demonstrates this in DBIx::Class:Loader,
> > against PostgreSQL.
> >
> > The testcase is standalone, but replicates the way ::Loader is used by
> > Catalyst, which is where I originally hit both these errors, when testing
> > under load.
> >
> > Curiously, I tried to use the workaround that worked for Catalyst with
> > Class::DBI::Loader, but it didn't seem to help here.  (ie. forcing a
> > disconnect with ->storage->disconnect immediately after the Cat setup()
> > call.)
> >
> Loader 0.18 actually does its own ->storage->disconnect at the end of
> setting itself up, to avoid people needing to do things like that. 

I noticed that when I was looking at the code last night; however the
connection appears to be maintained anyway?

> Are you sure your testcase isn't causing a new Loader object to be
> initialized every time a new mod_perl child spawns?  That would
> probably cause problems eventually if one kicked off before the other
> was done.  (Whereas in the standard Cat usage with C::M::DBIC where
> the whole Cat app is loaded up pre-forking, Loader should only ever
> get initialized once).

In answer to part a) No, that's not the cause of the problem. It's caused by
several forked apache processes sharing a single socket to the database as
result of the connection being opened prior to forking. For a simple proof,
example the number of child handlers of apache vs postgres after a run..
Instead of a one-to-one relationship, it's more like 1 to 10.

In answer to part b) I originally hit this problem in standard Catalyst usage
- the attached test case is merely a reduction/example that allows the problem
to be demonstrated on its own.

> I'll run the testcase and start figuring this out a bit later today

Thanks!

It's worth noting that if you disable Apache::DBI then the problem goes away..
but without persistent database connections, the webserver performance
suffers - and one can use Apache::DBI safely with cDBI::Loader.

It might be worth chatting to the new maintainer of cDBI::Loader, as they were
able to fix this after I provided a similar testcase.
Although, looking at the diff between .22 and .26, it seems like the only
change was to add a $dbh->disconnect.. which you do anyway.

I'm a bit lost by now, but hopefully the testcase can help.

-Toby

-- 
Turning and turning in the widening gyre/The falcon cannot hear the falconer;
Things fall apart, the centre cannot hold/Mere anarchy is loosed upon the world
(gpg --keyserver www.co.uk.pgp.net --recv-key B1CCF88E)