Great Circle Associates Majordomo-Workers
(September 1996)
 

Indexed By Date: [Previous] [Next] Indexed By Thread: [Previous] [Next]

Subject: Re: Majordomo Bombing
From: Brent Chapman <Brent @ GreatCircle . COM>
Date: Wed, 25 Sep 1996 16:33:38 -0600
To: Jason L Tibbitts III <tibbs @ hpc . uh . edu>, Evan Champion <evanc @ synapse . net>
Cc: John Orthoefer <jco @ BBNPlanet . com>, majordomo-workers @ GreatCircle . COM
In-reply-to: <ufan2yea11a.fsf@sina.hpc.uh.edu>
References: Evan Champion's message of Wed, 25 Sep 1996 11:58:44 -0400<199609251348.JAA10344@smuggler.bbnplanet.com> <324956B4.49B8@synapse.net>

At 1:43 PM -0500 9/25/96, Jason L Tibbitts III wrote:
>>>>>> "EC" == Evan Champion <evanc@synapse.net> writes:
>
>EC> I don't see why the load needs to get so high as to force sendmail to
>EC> start queuing.
>
>It's configurable.  I don't see the problem.
>
>EC> Perhaps Majordomo should queue outgoing messages by default, however...
>
>Majordomo shouldn't queue anything.  Majordomo is not a queuing system,
>it's designed to process requests as they come in.  Since Sendmail just
>won't deliver new messages once the load gets high, there shouldn't be a
>problem.

The problem is, if all those messages hit at once, a bunch of processes get
forked off and the load zooms before Sendmail notices and hits the brakes;
by then it's too late.

>EC> As soon as the machine restarted, sendmail started to run its queue
>EC> again and within moments the load went from its normal .1-.5 to > 50
>EC> and growing every second.  The only way the tech could stop it was to
>EC> blitz the mail queue.
>
>Something is seriously misconfigured, then.  I sent 1000 subscribe requests
>to a test list while TLB was doing two deliveries on a large list of mine.
>The load hit 17, sendmail stopped delivering, TLB switched to using a
>backup host and everything was fine.  The mail that was queued emptied on
>the next queue run.
>
> - J<

"The load hit 17" is past the recovery point for many machines...  Once it
gets that bad, they start thrashing, and it never gets any better.

We had lots of problems like this on our main machine
(miles.greatcircle.com) a few months ago.  I did 3 things that seem to have
taken care of the problems:

1) Added a "-odq" flag to all invocations of sendmail from majordomo (via
the /etc/majordomo.cf $mailer setting) and resend (via a "-m -odq" argument
to resend in the /etc/aliases entry).

2) Added the load average test I described earlier to /etc/majordomo.cf,
causing  anything that executes /etc/majordomo.cf to report a temporary
failure to its invoker (the mailer, i.e. sendmail) if the load is over 8.0.

3) Set up an alternative sendmail queue for outgoing list postings that's
run with "delivery mode" set to "queue" rather than immediate, and that
sends all outgoing list mail to my Internet service provider; that way, my
machine and my relatively slow link sees one outgoing message with 5000
recipients, instead of ~5000 outgoing messages with ~1 recipient each.  To
accomplish this, I needed an alternative sendmail.cf file.  To use an
alternative sendmail.cf file without tripping Sendmail's safety checks
(which disable all sorts of things if you use a "-C" argument to Sendmail
to specify an alternative sendmail.cf file), I compiled an alternative
sendmail binary with the path to the alternative sendmail.cf compiled in.


-Brent

----------------------+----------------------------+------------------------
Brent Chapman         | Great Circle Associates    | 1057 West Dana Street
Brent@GreatCircle.COM | http://www.greatcircle.com | Mountain View, CA 94041
----------------------+----------------------------+------------------------
                   Internet Tutorials from the Experts!




Follow-Ups:
References:
Indexed By Date Previous: Re: Beta3 torture tactics
From: Brock Rozen <brozen@webdreams.com>
Next: Re: Majordomo Bombing
From: Jason L Tibbitts III <tibbs@hpc.uh.edu>
Indexed By Thread Previous: Re: Majordomo Bombing
From: Evan Champion <evanc@synapse.net>
Next: Re: Majordomo Bombing
From: Jason L Tibbitts III <tibbs@hpc.uh.edu>

Google
 
Search Internet Search www.greatcircle.com