From majordomo-docs-owner@greatcircle.com Wed Nov 27 16:13:50 2002 Received: from daniel.liston.nu (58-25.siocpe.cableone.net [24.116.58.25]) by mycroft.greatcircle.com (Postfix) with ESMTP id E009A1959F8; Wed, 27 Nov 2002 16:13:47 -0800 (PST) Received: from maxima.liston.nu (maxima.liston.nu [192.168.1.80]) by daniel.liston.nu (8.12.5/8.12.5) with ESMTP id gAS0DLMC010788; Wed, 27 Nov 2002 18:13:26 -0600 Received: from sonny.org ([192.168.1.60]) by maxima.liston.nu (iPlanet Messaging Server 5.2 HotFix 0.9 (built Jul 29 2002)) with ESMTP id <0H6900K06F5S2G@maxima.liston.nu>; Wed, 27 Nov 2002 18:10:41 -0600 (CST) Date: Wed, 27 Nov 2002 18:13:44 -0600 From: Daniel Liston Subject: Archive/Digest HOW-TO (revision 1.2) repost To: majordomo-users@greatcircle.com, majordomo-docs@greatcircle.com, steveo@syslang.net Message-id: <3DE55FB8.8060407@sonny.org> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii; format=flowed Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 References: <3BB1B813.6080202@pandasoftware.es> X-Archive-Number: 200211/1 X-Sequence-Number: 5 Disclaimer: I am not a professional writer or programmer. Please report any/all discrepencies in this document to me via the majordomo-users or majordomo-docs mailing lists at greatcircle.com. I will be happy to make corrections and repost. Majordomo list owners often have trouble setting up a digest version of their list. I am writing this in hopes of clearing up some of this confusion. Successfully archiving your lists could be a side effect of implementing the knowledge gained from the examples included. Assumptions: Majordomo version 1.94.4 Sendmail version 8.8 (or higher) You have root permission to create users, modify the aliases file, and rebuild the aliases database. You have created a majordomo user and majordomo group. The majordomo $HOME directory is /usr/local/majordomo and is owned by user majordomo and group majordomo. The smrsh directory (if used) is /etc/smrsh and holds a link owned by root to /usr/local/majordomo/wrapper. There are no group or world "writable" directories in the path to /usr/local/majordomo, including / (root). Sendmail "trusts" majordomo. (a Tmajordomo line in sendmail.cf or a majordomo line in sendmail.ct). Sendmail.cf only recognizes one aliases file. (/etc/aliases) Sendmail.mc uses the "always_add_domain" feature. Majordomo's config file is /etc/majordomo.cf (can be a root owned link to /usr/local/majordomo/majordomo.cf) The majordomo.cf variables are defined statically. (no `/bin/dnsdomainname` in the $whereami variable). You already have a working list that utilizes resend. (wrapper resend -l -outgoing). Your aliases file contains "nobody: /dev/null". YOU DO NOT ALLOW HTML/MIME MESSAGES INTO YOUR LIST! This WILL uglify your digest! Preface: The directories above may be different, but in that case, you will modify anything stated further to match your installation. To set up a digest, you must also have a handle on archiving. Digest has some dependencies, or I should say makes some assumptions. One is that you have an archive directory configured to hold the digestified message before delivery, and another is that you will call your delivery address -digest-outgoing. Your digest and archive directories are defined in the majordomo.cf file as $digest_work_dir and $filedir. $filedir_suffix is also (or can be) used for defining the location of your archives. For the purpose of this recipe, I will keep these in the same directory tree as my other majordomo data files and directories. The sample.cf or default majordomo.cf gives an example of /usr/local/mail/digest when lists are in /usr/local/lists and archive files are in /usr/local/mail/files. Here are my suggested settings based on the assumptions above; $homedir = "/usr/local/majordomo"; $digest_work_dir = "$homedir/digests"; $listdir = "$homedir/lists"; $filedir = "$homedir/archives"; $filedir_suffix = ""; What is not explained very well, is that you will need to create these directories (if you haven't already) as the majordomo user, or change ownership to the majordomo user if you created them as root. Make sure these directories are not group or world writable, to prevent sendmail from complaining. (These directories may be group and/or world readable and executable.) You will also have to reference these directories in your /etc/aliases so mail is passed to the scripts that will be doing the actual work. In some (maybe all) cases, there will also be subdirectories involved that will need the same ownerships and permissions as their parents. A "test" mailing list that does archives AND digests will have aliases that look like this; owner-test: test-owner, test: "|/usr/local/majordomo/wrapper resend -l test test-outgoing,nobody" test-owner: real.user@some.domain test-request: "|/usr/local/majordomo/wrapper majordomo -l test" test-approval: moderator@your.domain ### could also be pointed to test-owner test-outgoing: :include:/usr/local/majordomo/test,test-archiver,test-digestify test-archiver: "|/usr/local/majordomo/wrapper archive2.pl -f /usr/local/majordomo/archives/test/test -a -M" test-digestify: "|/usr/local/majordomo/wrapper digest -r -C -l test-digest test-digest-outgoing" owner-test-digest: owner-test test-digest: test, test-digest-owner: test-owner, test-digest-request: "|/usr/local/majordomo/wrapper majordomo -l test-digest" test-digest-approval: test-approval, test-digest-outgoing: :include:/usr/local/lists/test-digest,nobody" The -approval addresses are not required, and will default to owner-(-digest), and the -request addresses could just be pointed to the owner/moderator alias depending on your specific needs or requirements. Some of the "owner" aliases are redundant, but safe. Let me also explain that you do not need to enable archiving to use a digest version of your list, but you WILL need the majordomo.cf and archives directory properly configured for digestification to happen. The digest script also "assumes" your outgoing address for the digest is called -digest-outgoing. In the case above, test-digest-outgoing. Continuinge with our example, we now have a dependency on subdirectories for the -archiver and -digestify aliases. Create these directories, then run "newaliases" or "sendmail -bi". To save time, you can `su majordomo` before you create the directories. mkdir /usr/local/majordomo/archives/test mkdir /usr/local/majordomo/archives/test-digest mkdir /usr/local/majordomo/digests/test-digest The above need to be owner:group majordomo:majordomo and chmod 755. Loose security could be tightened up, but that is a different HOW-TO. :) The test-archiver: alias will stamp the archive with a 4 digit year and 2 digit month. If you notice your archive showing up as 1901xx instead of 2001xx, please apply the folllowing patch to archive2.pl =========================================== --- archive2.pl.orig 2000/01/07 11:00:49 +++ archive2.pl 2000/05/15 00:55:00 @@ -152,7 +152,11 @@ local($junk, $addr, $dow, $moy, $dom, $time, $year, @rest); ($junk, $addr, $dow, $moy, $dom, $time, $year, @rest) = split(/\s+/,$from); - &open_archive($FH, $year % 100, $MoY{$moy}, $dom); + if ($year =~ /\d{4}/) { + &open_archive($FH, $year -1900, $MoY{$moy}, $dom); + } else { + &open_archive($FH, $year % 100, $MoY{$moy}, $dom); + } } sub open_archive { =========================================== NOTE: To do digesting without the standard archive, there are two changes to the aliases. test-outgoing: :include:/usr/local/majordomo/test,test-digestify and you do not "need" the test-archiver: alias. If this is the case, you also do not need to create the /usr/local/majordomo/archives/test directory. Leaving them in, even though they are never used does not really hurt either. It is all a matter of calling or not calling the test-archiver: alias via the test-outgoing: alias. The final steps in making your digest happen daily/nightly, is to set your test-digest.config file variables high enough that digestification does not try to happen automatically, and to create a crontab entry that will make majordomo execute the digest script. In your test-digest.config file, depending on how busy you expect your list to be, adjust these variables appropriately high. I have not tried it, but maybe they can all be left blank. digest_maxdays = 366 digest_maxlines = 1000 maxlength = 100000 The maxlength is in bytes, not Kbytes. The other two are self explanatory. You only have to set one value, but you will have to experiment to get it right. A good starting point, might be to set digest_maxdays to 7, and leave the other two blank. Otherwise, you will probably have to tinker with the maxlength in hundreds of thousands or millions of bytes. As long as we are in the test-digest.config file, don't forget about the other digest_* variables either. digest_archive = digest_issue = 1 digest_name = test-digest digest_rm_footer = test digest_rm_fronter = test digest_volume = 1 digest_work_dir = To keep your digests clean, it is also a good idea to use purge_received = yes on your regular list (test.config) as well as the digest. However, this does make tracking problems more difficult. Then you can get a little fancy and set your fronters and footers, message_fronter << END - Included in this issue: _SUBJECTS_ END message_footer << END unsubscribe instructions, copyright message etc. END If your security uses cron.allow or cron.deny, install the appropriate setting for . Your crontab entry should not be owned by root or majordomo to avoid "Majordomo ABORT" messages. I typically fill in with the ID of the list-owner or majordomo-owner. Now to access the crontab... crontab -u -e SHELL=/bin/sh MAILTO="" 30 4 * * 5 echo mkdigest test-digest | mail majordomo #| | | | | ^here to % or EOL is field 6 for commands #| | | | +-- 3 letter day of week or 0-7 0=Sun #| | | +-- 3 letter month or 1-12 #| | +-- day 1-31 #| +-- hours 0-23 #+-- minutes 0-59 The 1st line above sends majordomo an email forcing it to make a digest and send it to the default test-digest-outgoing alias every Friday at 4:30AM. If your delivery address is not test-digest-outgoing, that line can be changed to something like 30 4 * * 5 echo mkdigest test-digest test-digest-whatever | mail majordomo If you want the list to go out daily, change the "5" above, to a "*". Having a password stored in the crontab is probably not a good idea either, but it does get the job done. The rest of the lines are just an explanation of the crontab fields. Dan Liston PS. Please feel free to add comments/criticisms about this document. It is after all my first attempt at explaining the steps necessary to archive/digest a mailing list. It is possible that I may have omitted something useful.