I have a number of moderately large lists (300-3000) whose processing is
causing CPU loading problems on my majordomo server. I've had some
success with a few things I've tried, but I'd appreciate your
suggestions for a better way to throttle the load.
Here's what I've done so far:
With the original vanilla majordomo install:
1) these large lists were hanging and being delayed by bad addresses
(over which I have no control)
2) I was seeing extended 100%CPU utilizations
3) I was seeing miserable throughput, which I was attributing to
n-squared proportionality
4) I was concerned with sending envelopes with some 30-50 messages to
ISPs like AOL.
Then I installed bulk_mailer and set maxrcpts=10 and cured some of these
problems:
1) a bad address just hangs the group of 10 mailings it's in
2) I ended up with even heavier....but much briefer...loads
3) throughput has turned into an avalanche...which was a problem
I was now seeing uptime load averages as high as 200 or more and I was
getting errors from delayed or aborted operations (Draining Input,
Operating System error, etc), some duplicate mailings.
So I uncommented the section in majordomo.cf that defers incoming mail
to majordomo when the uptime load average gets above a defined value and
set that to 10.
That helped a bit...but not a lot...here's why:
a single job gets submitted into majordomo....which processes it and (so
far no great load, so no throttling) hands it off to bulkmailer which
splits it into multiple jobs of 10 recipients each.../this/ puts a
substantial load on the majordomo host, but the load is already past the
gatekeeper.
What's happening is that numerous jobs get accepted by majordomo
/BEFORE/ the load average gets high enough to defer further mailings.
The numerous jobs that did get accepted by majordomo then get expanded
by bulk-mailer and the system gets clobbered.
So majordomo throttling will not keep the CPU from going to 100%, but it
will defer new jobs once it has reached the load average
trigger...somewhat too late. Though the uptime load average trigger is
set at 10, once bulk_mailer expands the jobs and things get rolling, the
load average can go as high as 30....even though majordomo deferred
things after it hit 10!
Does anyone know of a better way to throttle majordomo and/or bulk_mailer?
--
====
Stewart Dean, Unix System Admin, Henderson Computer Resources
Center of Bard College, Annandale-on-Hudson, New York 12504
sdean@bard.edu voice: 845-758-7475, fax: 845-758-7035
|
|
