Great Circle Associates Majordomo-Users
(March 1995)
 

Indexed By Date: [Previous] [Next] Indexed By Thread: [Previous] [Next]

Subject: Re: Truncated subscriber lists!
From: debbie @ qsun . ho . att . com (Deborah A Hamilton)
Date: Wed, 15 Mar 95 08:47:18 EST
To: Thomas Leavitt <leavitt @ webcom . com>, majordomo-users @ greatcircle . com

Thomas Leavitt writes:
> One of our users, who maintains a relatively large mailing list, has 
> had his subscriber lists truncated by Majordomo at least twice. Needless to 
> say, this is quite distressing.
> 
> I have been told by another list-owner at Netcom that this has happened
> to her a number of times as well... from this, I suspect this phenomenon,
> is likely to occur when the load on the server the list is running
> on peaks, since Netcom machines are cronically overloaded, and the
> truncations on our site have occured when we've be dealing with rogue processes
> spamming the system.
> 
> We are working on a custom solution to this problem, but I thought
> I'd raise the issue here, and see if anyone else had experienced it,
> or has identified the cause (we added a number of error checks after
> the first occurance, but these seem to have been ignored when the
> phenomenon repeated itself).

This problem also happened to us after transferring two large and
one enormous mailing list from InterNIC Information Services to
our host on Information Directory and Database Services.  The 
enormous mailing list (scout-report) got spammed at least 4 times
in one week.  We also took a tremendous performance hit since the
other two large lists are very active.

I've been working on solutions to this problem and have developed
the following facilities that appear to working somewhat okay:

1.)  Queue all mail coming in to majordomo and the mailing lists
     and process the queue in order to single thread all requests.
     This was a hack on the queuer we have for our email server
     applications.  Instead of sending the mail directly to majordomo
     or to resend, it's saved in a queue file with a flag telling
     which service it should be forwarded to.  Every half-hour, cron
     kicks off the job to process the queue in order, waiting until
     requested service completes before sending off the next request.

2.)  For large mailing lists (> 2500 subscribers), batch the 
     subscription requests and process them as a group once/day.
     This involved a new parameter in the list config files
     (batch_subs = yes|no).  If turned on, the subscribe/unsubscribe
     requests are saved in <listname>.sub.  Notification is sent
     to the requester that the request has been queued.  Once a day,
     cron kicks off a job to process a list's .sub file.  The
     requesters are notified when the subscribe/unsubscribe completes,
     and the list-owner gets a list of the subscribed/unsubscribed
     ids.

Since I've implemented these two facilities, the lists have not
been spammed.  Yes, it slows down getting mail posted to the lists,
but it has also pretty much eliminated the battles for the lock files
causing failed majordomo requests and posting email to the lists.

I'm still working out some minor glitches in the batch subscription
processor (changing from a grep to a hash seach to verify if an
id is in the list).  If you're interested, I'll share what I've got.

Debbie Hamilton
InterNIC Directory and Database Services Support
AT&T Bell Laboratories
debbie@qsun.att.com
(908) 949-9459

Indexed By Date Previous: Global noadvertise for a list
From: Herve DEMARTHE (CEA France) <demarthe@alpha.cad.cea.fr>
Next: Linux 1.1.59 / Files written by majordom
From: "Nielsen, Morten N." <mnn@fsk.dk>
Indexed By Thread Previous: Re: Truncated subscriber lists!
From: rs0thp@rohmhaas.com (Dr. Tom Pierce)
Next: Re: Truncated subscriber lists!
From: Anne Bennett <anne@alcor.concordia.ca>

Google
 
Search Internet Search www.greatcircle.com