qmail/RFCLOOPS

Tools in the war on mail loops
D. J. Bernstein, djb@pobox.com
19970201


1. Introduction

   An automailer means any program that receives a mail message and
   automatically sends one or more mail messages. This term is meant to
   include not only a mail-based server, such as a mailing list exploder
   or a vacation program, but also an SMTP server, which receives a
   message from the network and relays it to a local or remote user.

   In a network full of automailers, any mistake can cause a mail loop.
   Since some automailers generate several outputs in response to a
   single input, a loop can produce an exponential explosion of mail.

   All the automailers in the qmail package follow a general philosophy
   designed to prevent mail loops and limit the damage from any loops
   that do occur. These automailers have been repeatedly observed to
   fail safe: they stop loops in the face of typical failures by other
   hosts. This document explains the philosophy and describes the
   automailers.

   To some extent the philosophy here simply repeats and amplifies
   standard practice as codified in RFC 974 and RFC 1123. Unfortunately,
   the standards do not adequately control bounce loops, since they do
   not recognize that postmasters want to see double bounces; they do
   not adequately control relaying loops; and they do not prevent
   cross-host forwarding loops.

   Terminology: The mail message received by an automailer is called
   input. The mail messages sent by an automailer are called outputs.
   For simplicity, this document focuses on the case that the input has
   just one envelope recipient.

   REMINDER: This document describes the automailers in the qmail
   package. Other packages include automailers that do not fit the
   descriptions given here.

   Beware that the war on mail loops can never be won: any method of
   preventing mail loops can be subverted by other hosts. I welcome
   further development of techniques that work well in practice.


2. Basics

   The output from an automailer is always further down the following
   list than the input.

      0 hops, <sender> is neither <> nor <#@[]>  normal messages
      1 hop, <sender> is neither <> nor <#@[]>
      2 hops, <sender> is neither <> nor <#@[]>
      etc.
      0 hops, <sender> is <>                     bounces
      1 hop, <sender> is <>
      2 hops, <sender> is <>
      etc.
      0 hops, <sender> is <#@[]>                 double bounces
      1 hop, <sender> is <#@[]>
      2 hops, <sender> is <#@[]>
      etc.

   Here sender means the envelope sender address. Hops means the number
   of Received and Delivered-To fields in the header. See sections 3.3
   and 3.4 for an explanation of <> and <#@[]>.

   Consequently, no automailer ever generates an entirely new normal
   message in response to a normal message. If the output is a normal
   message, it always has more hops than the input.

   When input and output are both normal messages, both bounces, or both
   double bounces, the output header is essentially the same as the
   input header. However, when an automailer moves from a normal message
   to a bounce, or from a bounce to a double bounce, it generates an
   entirely new header.

   An automailer may refuse to operate if the input has too many hops.
   The definition of too many hops depends on the automailer. This
   practice is called hop counting. Note that some existing messages
   legitimately take as many as 20 hops. One automailer uses a limit of
   100 hops; this will be adequate for all messages in the foreseeable
   future.

   Hop counting is a weapon of last resort. It will, if correctly
   implemented, prevent all infinite loops; however, even a finite loop
   can do practically infinite damage, as illustrated in section 4.3.


3. Pre-delivery automailers

   Conceptually: The input is a message that has not yet reached its
   envelope recipient address. It is fed to a relay, which attempts to
   deliver the message directly to, or at least closer to, that address;
   if the relay fails permanently, the message is fed to a bouncer or a
   double-bouncer. Relays, bouncers, and double-bouncers are examples of
   pre-delivery automailers.

   A pre-delivery automailer produces at most one output.

   The basic weapon against pre-delivery mail loops is gravity. A normal
   message always moves closer to its envelope recipient, according to a
   notion of distance defined in section 3.1. If it bounces before
   reaching the recipient, it turns into a bounce message, which always
   moves closer to the original envelope sender. If that in turn
   bounces, it turns into a double bounce, which always moves closer to
   a local postmaster. (Triple bounces do not exist.)


3.1. Distance

   The distance from a DNS domain D to a recipient U@R is defined as
   follows, when R has an MX list: the minimum preference of D in the
   MX list, or 100000 if D does not appear in the list.

   When R has no MX records, the distance from R to U@R is defined as 0,
   and the distance from any other domain to U@R is defined as 100000.

   Exception: If R is an alias, i.e., if R has a CNAME record, the
   distance from any domain to U@R is defined as 500000.

   The distance from a host H to U@R is defined as the minimum distance
   to U@R from any domain that touches H. (``D touches H'' means ``D has
   an A record listing one of H's IP addresses.'')

   Exception: If H does not accept mail from the network, its distance
   to any recipient is defined as 999999.


3.2. Relays

   A relay is a pre-delivery automailer that sends the output towards
   the envelope recipient. What this means for intra-host relays is not
   discussed here. What this means for cross-host relays is the
   following: if the relay is at host H, and it sends its output to host
   T, then the distance from T to the output envelope recipient is
   always smaller than the distance from H to the input envelope
   recipient.

   The following facts guarantee that certain cross-host relay behavior
   is safe. For proofs of these facts, see Appendix A.

      Fact 1: If R is an alias for X, X is not an alias, D touches T,
      and T accepts mail from the network, then the distance from T to
      U@X is smaller than the distance from H to U@R.

      Fact 2: If R is not an alias, R has no MX records, H is not
      touched by R, T is touched by R, and T accepts mail from the
      network, then T is closer to U@R than H is.

      Fact 3: If R is not an alias, R has an MX record with domain X and
      preference p, H is not touched by any of the domains in the MX
      list for R with preference <= p, T is touched by X, and T accepts
      mail from the network, then T is closer to U@R than H is.

   Also, a host that does not accept mail from the network can relay
   messages to a nearby hub.

   A relay adds a new Received header field to the top of the output.
   Other than this, the output header, body, and envelope are exactly
   the same as the input header, body, and envelope. Exception: If the
   input envelope recipient is U@R, R is an alias for X, and X is not
   an alias, the output envelope recipient is U@X.


3.3. Bouncers

   A bouncer is a pre-delivery automailer that lets the envelope sender
   know what happened to a message. Most bouncers send failure notices.
   Some bouncers, such as vacation servers and echo servers, send
   success notices.

   In a bouncer's output, the envelope sender is <>, and the envelope
   recipient is the input envelope sender. A bouncer refuses to operate
   if the input envelope sender is <> or <#@[]>.

   Some mailers on the Internet do not understand the <> convention. In
   fact, some mailers will rewrite <> as <@host>. So any message with an
   envelope recipient of <> or <@host> is discarded upon local delivery.

   Unlike a relay, a bouncer produces output with a new header, not
   simply a copy of the input header. For example:

      (envelope) from <> to <djb@silverton.berkeley.edu>
      Date: 2 Jan 1996 03:38:25 GMT
      From: DELIVERY NOTICE SYSTEM <MAILER-DAEMON@heaven.af.mil>
      To: djb@silverton.berkeley.edu
      Subject: failure notice

   However, the body of the bounce indicates the relevant input envelope
   recipient, as well as the Message-ID of the input, if the input had a
   Message-ID. The body of a failure notice includes a copy of the
   entire input message.


3.4. Double-bouncers

   A double-bouncer is a pre-delivery automailer that informs a local
   postmaster of permanent failures to deliver bounce messages. Such
   failures are generally caused by poorly configured hosts that produce
   normal messages with faulty envelope sender addresses.

   A double-bouncer refuses to operate unless the input envelope sender
   is <>. The output envelope sender from a double-bouncer is <#@[]>;
   note that <#@[]> cannot be used as an SMTP envelope sender under
   RFC 821. The output envelope recipient is predetermined.

   Note that double bounces are not suggested by RFC 1123. However,
   faulty envelope sender addresses are usually configuration errors
   that can and should be fixed. Some postmasters, faced with mail
   software that throws away double bounces, resort to keeping copies of
   all bounces; but single bounces are rarely the postmaster's problem.


4. Post-delivery automailers

   Conceptually: The input is a message that has reached its envelope
   recipient address. It is fed to a post-delivery automailer at that
   address.

   The basic weapon against post-delivery loops is a new header field,
   Delivered-To, tracing all the forwarders and mailing lists that a
   message has been through. This field has the side benefit of making
   it much easier for a user (or for a postmaster seeing a bounce) to
   figure out the path that the message took. Delivered-To is similar to
   RFC 1327's DL-Expansion-History, but (1) it omits the time stamp,
   removing any need for parsing, and (2) it has a much better name.


4.1. Exploders and repliers

   There are two basic types of post-delivery automailers: exploders,
   where the output envelope recipients are predetermined; and repliers,
   where there is just one output, with envelope recipient determined
   from the input.

   Repliers normally determine the output envelope recipient as either
   the input Reply-To header field, if it exists; or else the input
   From header field, if it exists; or else the envelope sender. A
   replier never produces an output to <> or <#@[]>.

   Exploders are classified into mailing lists, where the output
   envelope senders are predetermined, and forwarders, where every
   output has envelope sender equal to the original envelope sender.

   Exception: if the input envelope sender is <> or <#@[]>, then the
   output envelope senders are equal to the input envelope sender, even
   for a mailing list.

   Note that, if the envelope sender of a mailing list with M bad
   addresses is another exploder with E bad addresses, the local
   postmaster will receive EM double bounces for each message to the
   mailing list.


4.2. Delivered-To

   Every post-delivery automailer adds a new Delivered-To header field
   to the top of each output.

   The contents of the Delivered-To field are typically the address of
   the automailer, i.e., the input envelope recipient, conventionally
   without any quoting. The contents of the Delivered-To field are in
   any case entirely predetermined. The automailer checks if exactly the
   same Delivered-To field already appears in the header; if so, it
   refuses to operate. 

   A post-delivery automailer preserves existing Delivered-To and
   Received fields. In fact, a post-delivery automailer generally
   preserves all header fields. The exceptions are limited to known
   fields that are not used for loop detection and that must be removed
   for correct operation. For example, a replier generally changes the
   body of a message and thus should not preserve the SVR4
   Content-Length field.


4.3. An example

   Aliases and mailing lists are highly dangerous, because they can
   generate several outputs for each input.

   Here is an extreme example. A user has three accounts, and wants any
   message to any of the accounts to be delivered to all three. So he
   forwards luser@host1 to luser@host2 and luser@host3, forwards
   luser@host2 to luser@host1 and luser@host3, and forwards luser@host3
   to luser@host1 and luser@host2.

   Without Delivered-To, someone who sends a message to luser@host1 will
   receive a practically infinite series of bounces. For example, with a
   hop count limit of 50, the sender will receive 1125899906842624
   bounces.

   If all the hosts, or two out of the three, support Delivered-To, the
   message will bounce just a few times. If just one of the hosts
   supports Delivered-To, it will be the unfortunate victim of a loop
   between the other two hosts---although the total number of bounces
   will drop from practically infinite down to a few hundred, with
   typical hop count limits.


Appendix A. Proofs of correctness for MX handling

   Section 3.2 states three facts about the notion of distance defined
   in section 3.1. Here are mathematical proofs of those facts.

   Symbols: D, E, R, and X are domains; H and T are hosts; p and q are
   nonnegative integers. {} is the empty set.

   Hypotheses: M(R), the ``MX list for R,'' is a set of pairs (p,D)
   where p <= 65535. There is a set A of domains, called ``aliases.''
   There is a relation D->H, called ``D touches H.'' There is a set N of
   hosts, called ``hosts that accept mail from the network.''

   Definitions: m(D,R) = min { p: p = 100000 or (p,D) in M(R) } when
   M(R) is nonempty. When M(R) is empty, m(D,R) is 0 if D = R, 100000
   otherwise. f(D,R) is defined as 500000 if R is in A, m(D,R)
   otherwise; this is the ``distance from D to U@R,'' for any U. g(H,R)
   is defined as min { f(D,R): D->H } if H is in N, 999999 otherwise;
   this is the ``distance from H to U@R,'' for any U.

   Fact 1 (generalized): If R is in A, X is not in A, D->T, and T is in
   N, then g(T,X) < g(H,R). Proof: R is in A, so f(E,R) = 500000 for any
   E; thus g(H,R) >= 500000. X is not in A, so f(D,X) = m(D,X) <=
   100000; hence g(T,X) <= f(D,X) <= 100000 < g(H,R).

   Fact 2: If R is not in A, M(R) = {}, R->T, T is in N, and not R->H,
   then g(T,R) < g(H,R). Proof: f(R,R) = m(R,R) = 0 since R is not in A
   and M(R) = {}. T is in N so g(T,R) <= f(R,R) = 0 so g(T,R) = 0.
   Suppose that g(H,R) <= g(T,R). Then g(H,R) = 0, so f(D,R) = 0 for
   some D with D->H, so m(D,R) = 0. But then D = R by definition of m,
   so R->H. Contradiction. Thus g(T,R) < g(H,R).

   Fact 3: If R is not in A, (p,X) is in M(R), X->T, T is in N, and
   (q,D) is not in M(R) whenever D->H and q <= p, then g(T,R) < g(H,R).
   Proof: First m(X,R) <= p. R is not in A, so f(X,R) = m(X,R). T is in
   N, so g(T,R) <= f(X,R). Thus g(T,R) <= p. Suppose that g(H,R) <= p.
   Then f(D,R) <= p for some D with D->H, so m(D,R) <= p. But then
   (m(D,R),D) is in M(R). Contradiction. Thus g(T,R) <= p < g(H,R).