From list-managers-owner@greatcircle.com Thu Mar 17 01:58:25 2005 X-Original-To: list-managers@greatcircle.com Received: from panther.mmj.dk (panther.mmj.dk [62.79.83.143]) by mycroft.greatcircle.com (Postfix) with ESMTP id 0817D32C5E6 for ; Thu, 17 Mar 2005 01:56:32 -0800 (PST) Received: by panther.mmj.dk (Postfix, from userid 500) id 054881A3D33; Thu, 17 Mar 2005 10:59:19 +0100 (CET) Date: Thu, 17 Mar 2005 10:59:19 +0100 From: Mads Martin Joergensen To: list-managers@greatcircle.com Subject: sendmail and VERP Message-ID: <20050317095919.GE74249@mmj.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.8i X-Archive-Number: 200503/1 X-Sequence-Number: 1840 Hey all, So if any of you are running lists on sendmail and want to use VERP as well, you might want to have a look at the work Andreas Barisan did. He's using mlmmj with verp enabled sendmail: http://dev.gentoo.org/~lcars/misc/sendmail-hacks.txt I love such sendmail rulesets--they make write only perl look like the work of an artist ;-) -- Mads Martin Joergensen, http://mmj.dk "Why make things difficult, when it is possible to make them cryptic and totally illogical, with just a little bit more effort?" -- A. P. J. From list-managers-owner@greatcircle.com Thu Mar 17 02:06:12 2005 X-Original-To: list-managers@greatcircle.com Received: from panther.mmj.dk (panther.mmj.dk [62.79.83.143]) by mycroft.greatcircle.com (Postfix) with ESMTP id DEA8E32C5C0 for ; Thu, 17 Mar 2005 02:05:46 -0800 (PST) Received: by panther.mmj.dk (Postfix, from userid 500) id 5A6311A3D33; Thu, 17 Mar 2005 11:08:54 +0100 (CET) Date: Thu, 17 Mar 2005 11:08:54 +0100 From: Mads Martin Joergensen To: list-managers@greatcircle.com Subject: Re: sendmail and VERP Message-ID: <20050317100854.GG74249@mmj.dk> References: <20050317095919.GE74249@mmj.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20050317095919.GE74249@mmj.dk> User-Agent: Mutt/1.5.8i X-Archive-Number: 200503/2 X-Sequence-Number: 1841 * Mads Martin Joergensen [Mar 17. 2005 11:02]: > So if any of you are running lists on sendmail and want to use VERP as > well, you might want to have a look at the work Andreas Barisan did. I should learn to spell. Andrea Barisani is his name. -- Mads Martin Joergensen, http://mmj.dk "Why make things difficult, when it is possible to make them cryptic and totally illogical, with just a little bit more effort?" -- A. P. J. From list-managers-owner@greatcircle.com Thu Mar 17 04:02:57 2005 X-Original-To: list-managers@greatcircle.com Received: from nina.cs.keele.ac.uk (nnina.cs.keele.ac.uk [160.5.89.71]) by mycroft.greatcircle.com (Postfix) with ESMTP id 5CE4132C640 for ; Thu, 17 Mar 2005 03:58:23 -0800 (PST) Received: from jonathan by nina.cs.keele.ac.uk with local (Exim 4.43) id 1DBtmg-00030A-HL for list-managers@greatcircle.com; Thu, 17 Mar 2005 12:07:26 +0000 Date: Thu, 17 Mar 2005 12:07:26 +0000 From: Jonathan Knight To: list-managers@greatcircle.com Subject: Re: sendmail and VERP Message-ID: <20050317120726.GA11383@cs.keele.ac.uk> References: <20050317095919.GE74249@mmj.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20050317095919.GE74249@mmj.dk> User-Agent: Mutt/1.4.2.1i X-Anti-Virus: ClamAV at Keele University X-Archive-Number: 200503/3 X-Sequence-Number: 1842 On Thu, Mar 17, 2005 at 10:59:19AM +0100, Mads Martin Joergensen wrote: > So if any of you are running lists on sendmail and want to use VERP as > well, you might want to have a look at the work Andreas Barisan did. > He's using mlmmj with verp enabled sendmail: Seeing as I'm just running into a lot of VERP problems I thought I'd see what experience other list managers are having. I'm implementing greylisting for users at Keele as an anti-spam measure. http://projects.puremagic.com/greylisting/ A quick description of greylisting is that a mail server takes a look at the sender address and recipient address and the IP number of the sender and takes a look to see if that combination has been seen before. If it hasn't then a temporary failure code is returned to force the sending mail system to retry at a later date. This block remains in force for an hour. After an hour the mail system will then accept the mail and whitelist the sender, recipient and IP triplet for about a month. If that triplet is seen again then the whitelist is extended each time. The idea being that if there's normal traffic between two mail addresses then they quickly become whitelisted and no further delays occur. It has a high success rate as it neatly stops all the current spam software running on PC's as they (at present) do not have queue enabled MTA's and so never retry a temporary failure. If they do develop queue enbaled software then we still win because we have roughly 1 hour to detect a spamming PC and get it on an RBL before they call back to our mail server. If spammers start using ISP's to spam from to get a queuing mechanism then they will end up on spamcop and we block them there. The problem with greylisting is that VERP enabled mailing lists always fall foul of the greylist and never get themselves whitelisted. Normal mailing lists like mailman do not have this problem and very quickly become whitelisted. As I've implemented greylisting it has become obvious that there are quite a few VERP enabled mailing lists out there that are now getting 1 hour delays from Keele. What's worse is that some seem to use a unique sender for every attempt to send the mail and therefore never get a mail to us because we never see the same sender twice. Even worse than that, there are sites, such as yahoo groups, which seem to take a temporary failure as a permanant failure and throw the message away. As the number of sites that are greylisting increases I have some doubts as to whether VERP has a future in its current form. -- ______ jonathan@cs.keele.ac.uk Jonathan Knight, / Department of Computer Science / _ __ Telephone: +44 1782 583437 University of Keele, Keele, (_/ (_) / / Fax : +44 1782 713082 Staffordshire. ST5 5BG. U.K. From list-managers-owner@greatcircle.com Thu Mar 17 13:35:12 2005 X-Original-To: list-managers@greatcircle.com Received: from xuxa.iecc.com (xuxa.iecc.com [208.31.42.42]) by mycroft.greatcircle.com (Postfix) with SMTP id D106A32C1D3 for ; Thu, 17 Mar 2005 13:35:10 -0800 (PST) Received: (qmail 13848 invoked by uid 100); 17 Mar 2005 21:35:08 -0000 Date: 17 Mar 2005 21:35:08 -0000 Message-ID: <20050317213508.13847.qmail@xuxa.iecc.com> From: John Levine To: list-managers@greatcircle.com Subject: Re: sendmail and VERP In-Reply-To: <20050317120726.GA11383@cs.keele.ac.uk> Organization: I.E.C.C., Trumansburg NY USA Cc: jonathan@cs.keele.ac.uk Mime-Version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-transfer-encoding: 7bit X-Archive-Number: 200503/4 X-Sequence-Number: 1843 >Seeing as I'm just running into a lot of VERP problems I thought I'd see >what experience other list managers are having. I'm implementing >greylisting for users at Keele as an anti-spam measure. ... >A quick description of greylisting is that a mail server takes a look >at the sender address and recipient address and the IP number of the >sender and takes a look to see if that combination has been seen >before. This problem is easy to solve: Don't Do That. The point of greylisting is to lose mail from misimplemented hosts that don't retry after a soft failure. As soon as you see a retry from a given host, you know that, no matter what other flaws it may have, it's able to retry so there's no point in soft failing its mail any more. That means that when you see a retry, you whitelist the IP, not the specific bounce/recipient addresses. To keep the size of your whitelist under control you might want to age out the IPs if you don't hear from them for a month, but other than that there's no reason ever to greylist mail from a known-to-retry IP again. My greylist system does this and it works very well, rejects lots of spam, delays very little real mail. Anyone's welcome to a copy, although it's written for qmail, so you'd have to rewrite the client that's part of the SMTP daemon from scratch. By the way, it's increasingly unwise to expect that you can predict the bounce address on any incoming mail, not just mailing list mail. All my outgoing mail all uses BATV to put a signature in the bounce address, so I can tell real bounces from spam blowback. Since this lets me reliably detect and reject several hundred thousand blowback attempts per day, it's not going away. My version of BATV has a time stamp that changes once a day, so I have a new bounce address every day. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, Mayor "I shook hands with Senators Dole and Inouye," said Tom, disarmingly. From list-managers-owner@greatcircle.com Fri Mar 18 03:17:51 2005 X-Original-To: list-managers@greatcircle.com Received: from nina.cs.keele.ac.uk (nnina.cs.keele.ac.uk [160.5.89.41]) by mycroft.greatcircle.com (Postfix) with ESMTP id AC8A232C44E for ; Fri, 18 Mar 2005 03:17:23 -0800 (PST) Received: from jonathan by nina.cs.keele.ac.uk with local (Exim 4.43) id 1DCFae-0007BT-5r for list-managers@greatcircle.com; Fri, 18 Mar 2005 11:24:28 +0000 Date: Fri, 18 Mar 2005 11:24:28 +0000 From: Jonathan Knight To: list-managers@greatcircle.com Subject: Re: sendmail and VERP Message-ID: <20050318112428.GB27258@cs.keele.ac.uk> References: <20050317120726.GA11383@cs.keele.ac.uk> <20050317213508.13847.qmail@xuxa.iecc.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20050317213508.13847.qmail@xuxa.iecc.com> User-Agent: Mutt/1.4.2.1i X-Anti-Virus: ClamAV at Keele University X-Archive-Number: 200503/5 X-Sequence-Number: 1844 On Thu, Mar 17, 2005 at 09:35:08PM -0000, John Levine wrote: > The point of greylisting is to lose mail from misimplemented hosts > that don't retry after a soft failure. As soon as you see a retry > from a given host, you know that, no matter what other flaws it may > have, it's able to retry so there's no point in soft failing its > mail any more. I agree with you regarding whitelisting the IP number - thanks for that - I'll be altering my implementation this morning. However I think that saying that as soon as you see a retry from a host you know that it is safe is not absolutely true. We currently insist that the host retries with the same sender and recipient between 1 hour and 5 hours from the initial attempt. The problem will be that currently spammers don't retry, but you can be sure that they are already looking at their spamming code and working out how to adapt it to do retries. The paper I read from Evan Harris pointed out that if you refuse for the first hour then in theory there's a good chance that the spammer will hit a spam trap and end up on an RBL list. So even if they do implement a queuing mechanism they shouldn't be too successful at getting spam to you. -- ______ jonathan@cs.keele.ac.uk Jonathan Knight, / Department of Computer Science / _ __ Telephone: +44 1782 583437 University of Keele, Keele, (_/ (_) / / Fax : +44 1782 713082 Staffordshire. ST5 5BG. U.K. From list-managers-owner@greatcircle.com Fri Mar 18 11:05:24 2005 X-Original-To: list-managers@greatcircle.com Received: from xuxa.iecc.com (xuxa.iecc.com [208.31.42.42]) by mycroft.greatcircle.com (Postfix) with SMTP id CB88432C22B for ; Fri, 18 Mar 2005 11:05:22 -0800 (PST) Received: (qmail 22943 invoked by uid 100); 18 Mar 2005 19:05:20 -0000 Date: 18 Mar 2005 19:05:20 -0000 Message-ID: <20050318190520.22940.qmail@xuxa.iecc.com> From: John Levine To: list-managers@greatcircle.com Subject: Re: sendmail and VERP In-Reply-To: <20050318112428.GB27258@cs.keele.ac.uk> Organization: I.E.C.C., Trumansburg NY USA Cc: jonathan@cs.keele.ac.uk Mime-Version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-transfer-encoding: 7bit X-Archive-Number: 200503/6 X-Sequence-Number: 1845 >I agree with you regarding whitelisting the IP number - thanks for that - >I'll be altering my implementation this morning. > >However I think that saying that as soon as you see a retry from a host you >know that it is safe is not absolutely true. We currently insist that the >host retries with the same sender and recipient between 1 hour and 5 hours >from the initial attempt. I agree that it makes sense to match the triple on the retry, but once you have a satisfactory retry, you whitelist the IP. Even that has its problems since big mailers with server farms can retry from a different IP, but in practice they'll retry from the same IP often enough to get on the whitelist at which point it doesn't matter. Having looked through my greylist logs, I find that accepting after 5 minutes is just as effective as accepting after an hour and greatly decreases the amount of legit mail that's delayed. I also find that I need to accept retries as late as 12 hours later since some hosts retry that infrequently. > The problem will be that currently spammers don't retry, but you > can be sure that they are already looking at their spamming code and > working out how to adapt it to do retries. Sure. If they fix it to retry, greylisting won't work any more. The question is which will happen first, the bad guys will fix the ratware to retry, or they'll change it to route the spam through the zombies' own ISP's mail server. My money is on the latter. If your goal is merely to stall mail until the sending IP has a chance to hit spam traps and perhaps be added to a blacklist, you can use a much simpler scheme. When you see mail from a hitherto unseen IP, you soft fail and remember the IP and time. When it's been an hour since the IP was first seen, you add it to the whitelist. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, Mayor "A book is a sneeze." - E.B. White, on the writing of Charlotte's Web