In message <199409140400.VAA18402@osiris.ac.hmc.edu>,
Jared_Rhine@hmc.edu writes:
>AdL == Arnold de Leon <arnold@Synopsys.COM>
>
>AdL> I don't believe ordering of near simultaneous commands to majordomo is
>AdL> important.
>
>But resource starvation is. On a heavily-loaded machine, it's not
>out of the question that a command might fail because every time it tries to
>get the lock, some newer process has it, instead. The problem is compounded
>by an exponential backoff because the older the process, the less frequently
>it will attempt the lock, meaning that young, whipper-snapper processes are
>more likely to obtain the lock.
>
>Majordomo design needs to think more about scalability. I'm currently
>having a bitch of a time because a single 'lists' command on an unloaded
>machine is starting to push 10 minutes to complete. We've got 190 lists,
>and a 'lists' command has to run a 'get_config' for each, which is an
>enormously expensive process. I'm thinking that I may implement the config
>variables in a dbm file, allowing me to go straight to the variable I want
>without any extraneous parsing. I don't know how this can be modularized
>into the majordomo design - it probably won't be too bad since I can just
>write a drop-in replacement for &get_config. Any comments on this, or other
>ideas about how to speed up &get_config by at least an order of magnitude?
Try turning off the validity checks in the interior loop unless you
are in a newconfig operation. It uses the following code segment:
if (defined($main'config_opts{$list,$key})) {
$main'config_opts{$list,$key} .= "\001" .
&$parse($value, $list, $key);
} else { # we are starting an array
$main'config_opts{$list,$key} =
&$parse($value, $list, $key);
this is unecessary iff all of the config files are installed using
newconfig. You just need to add a parameter that says yes or no to the
config code, and then create some generic (non checking) parse
function that spits back the arguments as required.
This is the first place I would look. I think dbm files are the wrong
approach since there is no way for perl to tell if the dbm file is of
the wrong byte order, and perl just crashes stupidly. I know of a
number of sites whose backup mail hub is of different endidness from
their primary, I have two sites that run that way myself. If the dbm
files could be discovered to be of the wrong endidness before perl
coredumps, and the regular config files were used instead, then the
dbm file idea is great. Don't forget to compare the touch dates of the
config files against the touch dates of the dbm files.
As an alternate idea, how about a cache file (written in perl) that
holds all of the contents of the config file(s). That way you just
require the file, and the parsing overhead that goes on isn't
necessary anymore. It would be pretty easy to do, actually you could
have one cache file for all lists since the config files don't change
that often. To write the cache file just do a loop like:
foreach i (keys %config_opts) {
print CACHE "\$config_opts{$i} =\"" . $config_opts{$key} . '"';
}
Or something like that.
-- John
John Rouillard
Senior Systems Administrator IDD Information Services
rouilj@dstar.iddis.com Waltham, MA (617) 890-1576 x225
Senior Systems Consultant (SERL Project) University of Massachusetts at Boston
rouilj@cs.umb.edu (preferred) Boston, MA, (617) 287-6480
===============================================================================
My employers don't acknowledge my existence much less my opinions.
References:
|
|
