Re: load problems with Majordomo and bulk_mailer

[Daniel Liston <dliston @ sonny . org>]

Are you using subscription confirmation messages?  This is
your first defense in keeping a clean list.

Do you clean your lists of bad addresses after experiencing
the load and delay problems?  We have to realize that people
do move/change addresses without bothering to update their
subscription information too.  This should be a daily process
on a busy list, and monthly on less active lists.

Do you sort your lists by domain periodically?  This won't
necessarily help load, but will improve efficiency.

Keep in mind, majordomo is NOT delivering mail.  Majordomo
does ALL it's mail communication with the local MTA.  Tuning
majordomo really means tuning your MTA and items your MTA
has dependencies on. (like how clean you keep your lists)

The MTA can be short-circuited for additional processing by
bulk_mailer, but ultimately the MTA still does the delivery.

We need to look to the MTA to stop accepting mail (or even
delivering) when the load average is above "X".  Remember,
majordomo only applies access control and other rules before
passing the message and the distribution addresses to the MTA.

Dan Liston

Stewart Dean wrote:
> I have a number of moderately large lists (300-3000) whose processing is 
> causing CPU loading problems on my majordomo server.  I've had some 
> success with a few things I've tried, but I'd appreciate your 
> suggestions for a better way to throttle the load.
> 
> Here's what I've done so far:
> 
> With the original vanilla majordomo install:
> 1) these large lists were hanging and being delayed by bad addresses 
> (over which I have no control)
> 2) I was seeing extended 100%CPU utilizations
> 3) I was seeing miserable throughput, which I was attributing to 
> n-squared proportionality
> 4) I was concerned with sending envelopes with some 30-50 messages to 
> ISPs like AOL.
> 
> 
> Then I installed bulk_mailer and set maxrcpts=10 and cured some of these 
> problems:
> 1) a bad address just hangs the group of 10 mailings it's in
> 2) I ended up with even heavier....but much briefer...loads
> 3) throughput has turned into an avalanche...which was a problem
> I was now seeing uptime load averages as high as 200 or more and I was 
> getting errors from delayed or aborted operations (Draining Input, 
> Operating System error, etc), some duplicate mailings.
> 
> So I uncommented the section in majordomo.cf that defers incoming mail 
> to majordomo when the uptime load average gets above a defined value and 
> set that to 10.
> 
> That helped a bit...but not a lot...here's why:
> a single job gets submitted into majordomo....which processes it and (so 
> far no great load, so no throttling) hands it off to bulkmailer which 
> splits it into multiple jobs of 10 recipients each.../this/ puts a 
> substantial load on the majordomo host, but the load is already past the 
> gatekeeper.
> 
> What's happening is that numerous jobs get accepted by majordomo 
> /BEFORE/ the load average gets high enough to defer further mailings. 
> The numerous jobs that did get accepted by majordomo then get expanded 
> by bulk-mailer and the system gets clobbered.
> So majordomo throttling will not keep the CPU from going to 100%, but it 
> will defer new jobs once it has reached the load average 
> trigger...somewhat too late.  Though the uptime load average trigger is 
> set at 10, once bulk_mailer expands the jobs and things get rolling, the 
> load average can go as high as 30....even though majordomo deferred 
> things after it hit 10!
> 
> Does anyone know of a better way to throttle majordomo and/or bulk_mailer?