Qmail bugs and wishlist.

This document should be linked as:
URL: http://home.pages.de/~mandree/qmail-bugs.html

Preface

If any part of the information below is incorrect or incomplete, feel free to send corrections and amendments.

Given DJB's claims about qmail security, observing the behavior of some of his followers, their denying qmail bugs and calling people who name qmail bugs "trolls", I feel obliged to document a collection of warnings about qmail bugs, problems, with solutions where known.

This document contains pointers to other people's work, I am not claiming that work as my own, but referring you to that work.

All subsequent contents of this document refer to qmail-1.03.

This document exists collect qmail problems and let YOU decide if you want to install this software or rather go for some other mail software. My personal preference is Postfix, but other people are happy with Exim, and Courier might also be worth a look.

If you read "Fix: none" below, that means: go patch qmail yourself if you want the issue fixed. Dan has not done it. Neither am I going to.

Part I: Bugs

1. Security

1.1. qmail-smtpd memory exhaustion attacks

While qmail is claimed to be the most secure MTA, it is not in its default install.

Wietse Venema found two ways to instrument qmail-smtpd to eat up all available memory and reported them in 1997: 1. Against RCPT. 2. With long lines, I am however not sure if this bug persists in qmail-1.03. This could lead to denial of service. Dan J. Bernstein disclaimed this bug and pointed to system resource limits. The INSTALL document however has never been updated.

The actual bug here is in the documentation: No part of the documentation that ships with qmail tells you which resource limits must be set (and how) to avoid this vulnerability.

Fix/Workaround: Dave Sill's "Life With Qmail" shows start scripts that prevent this attack by means of softlimit. (See: "The supervise scripts")

Dave suggests something similar to this (replace 1234 with your numeric qmaild user id and 4321 with your numeric nofiles gid:

      /usr/local/bin/softlimit -m 2000000 \
	  /usr/local/bin/tcpserver -v -R -l 0 -x /etc/tcp.smtp.cdb \
	  -u 1234 -g 4321 0 smtp /var/qmail/bin/qmail-smtpd
     

1.2. qmail-smtpd bounce flood

Qmail's Delayed-Bounce behavior is a security violation. It can be used to exhaust local resources and abuse the qmail site to annoy a third party with bounces.

While forging a bounce mail is not too difficult in general, the prepended bounce header can be exploited to increase the bandwidth of an attack. The qmail-smtpd server can be used to contact a third host on behalf of the first, which is generally viewed as security problem. The "delayed bounces" section has more information on this.

Fix/Workaround: no canonical one yet. Some third party workarounds are the goodrcptto patch by Eben Pratt, the qmail-realrcptto patch by Paul Jarc, the qmail-smtpd extensions by Tim Clarke, Erwin Hoffmann's RECIPIENTS extension for qmail-smtpd and the SMTP front end in Bruce Guenter's mailfront package.

1.3. qmail-qmtpd buffer overflow

Georgi Guninski reports a buffer overflow in qmail-qmtpd if the RELAYCLIENT variable contains between 4 and 1003 bytes and malformed input lengths are passed.

Workaround: disable and remove qmail-qmtpd, and use SMTP instead. QMTP is not an Internet standard protocol anyways.

Fix: none. Guninski suggests a patch, but it appears untested.

1.4. Remote root access on machines with large memory

Georgi Guninski reports several exploitable overflows that give root on 64-bit machines with lots of memory.

Guninski's report is backed by James Craig Burley's research, and Burley states that the problem is not limited to 64-bit platforms but may also affect 32-bit platforms with segmented memory models.

Personal note: The bugs found by Guninski are "classic" coding bugs, integer overflow, signed index variables, which refute any claims that DJB were a better coder than other people, and which also refute DJB's claim to write bug-free code. DJB still claims no security bugs had been found in qmail to date and denies Guninski's claim, arguing that nobody would give Gigabytes of memory to each qmail-smtpd process. Yet, DJB's qmail-1.03 tarball makes no mention of memory limiting requirements. Still, the information in DJB's qmail-1.03 tarball does not mention that the administrator installing qmail must work around qmail's dangerous assumptions.

Workaround: Disable and remove qmail-popup and qmail-pop3d and install a POP3 server that does not have such bugs.

Fix: none.

2. Reliability

2.1. Bounce message contents are not crash-proof.

Qmail does not value the contents of a bounce message. Dan documents this in a subordinate clause of his qmail reliability FAQ. That means: if your qmail is bouncing mail and at the same time, your system crashes, the bounce mail contents may be corrupt or incomplete.

Fix/Workaround: none

2.2. Maildir file name creation is not collision proof. Mail loss possible depending on mail user agent.

Qmail's maildir delivery code assumes that a process ID is unique for one second. It is not, on fast and heavily loaded systems, or systems that randomize the process ID, the same process ID can be reused in the same second, even multiple times. If mail delivery and mail reading with a user agent overlap in time, this can lead to mail loss depending on how cautious the mail user agent is when moving mail from new/ to cur/. DJB claims qmail-pop3d is safe, other applications, for example mutt, rely on the uniqueness of the file names and can cause mail loss. It's qmail's responsibility to use unique file names though.

Fix: Toby Betts has a patch to make maildir file names unique

Workaround: Use a recent version of maildrop for local delivery to maildir folders, it has a bugfixed maildir creation code that guarantees unique file names even on the fastest machines (it includes the file serial or inode number into the file name, which is guaranteed to be unique per file system).

2.3. IP aliases

Qmail does not reliably detect IP aliases on Linux.

Qmail assumes that only systems that have a "sa_len" member in "struct sockaddr" can have multiple IPv4 addresses (aka. IP aliases/virtual interfaces) for the same interface name. This is not true for Linux (ip addr add 1.2.3.4 dev eth0 ; ip addr add 4.3.2.1 dev eth0).

Impact: qmail may not detect the own IP address properly when sending mail, or when receiving mail for a "domain literal" (RCPT TO: <postmaster@[1.2.3.4]>).

Workaround: On Linux, always configure IP-aliases in Solaris style, ifconfig eth0 1.2.3.4 ; ifconfig eth0:1 4.3.2.1 or ip addr add 1.2.3.4 dev eth0 ; ip addr add 4.3.2.1 dev eth0:1.

Fix: apply qmail/patch-qmail-1.03-4.3BSD-ipalias.diff (signature).

3. RFC violations

3.1. Deliberate RFC-1652 "8BITMIME" violation

RFC-1652 describes a SMTP protocol extension to negotiate for the transport of characters outside the US-ASCII octet range.

A SMTP server offering this extension replies with 8BITMIME in the greeting, and when a client concludingly offers a mail with the tag BODY=8BITMIME, the server guarantees that it will either bounce the 8BITMIME mail or convert it to 7bit (for example, using the base64 MIME encoding) when it relays the mail to a destination which does not announce this extension.

qmail claims RFC-1652 compliance, and indeed offers the 8BITMIME extension, but it does not bounce or convert an 8BITMIME mail that is sent to a 7bit destination. While it's a good thing that qmail is 8bit clean, for the lack of the conversion or bounce, qmail is not allowed to offer the 8BITMIME extension. Qmail's "just-send-eight" practice can corrupt mail.

This is dangerous. Read Dan's view on 8BITMIME: he describes RFC-1652 as "fantasy" network. qmail announces 8BITMIME without conversion, to prevent Q-P conversion of a SMTP client that connects to qmail. This works out as long as qmail does not relay, but is only a leaf site. If however mail is being forwarded to a 7bit host, bad things can happen: characters get corrupted, the mail bounces because it has 8bit characters, or similar.

As Dan is aware of the implications, he should fix qmail and remove the 8BITMIME announcement. My patch below does just that. It will however not protect against dumb clients that still feed unencoded 8bit data.

Workaround: none

Fix: apply qmail/patch-qmail-1.03-rfc1652.diff

3.2. RFC-2821 (SMTP) violation (Mail Routing)

Formerly documented in RFC-974.

In 2002, after a [German] problem report in de.comm.software.mailserver, I found out that qmail would never try any other MX than the first it connects to, even if that MX does not reply with a 220 greeting. When the server replies with e. g. a 554 greeting, qmail disconnects and later retries the same server again, ignoring the fact that 554 is a permanent error.

qmail is supposed to retry all listed MX servers in order of preference until the mail delivery succeeds:

RFC-2821, markup not present in original version: "5. Address Resolution and Mail Handling
[...]
When the lookup succeeds, the mapping can result in a list of alternative delivery addresses rather than a single address, because of multiple MX records, multi-homing, or both. To provide reliable mail transmission, the SMTP client MUST be able to try (and retry) each of the relevant addresses in this list in order, until a delivery attempt succeeds. [...]"

Fix: use qmail/patch-qmail-1.03-rfc2821.diff.

Workaround: use /var/qmail/control/smtproutes and explicitly route mail to a backup MX.

3.3. Potential for Violation of RFC-2045 (MIME) and Lack of RFC-1894 (Delivery Status Notifications) support

qmail deliberately skips RFC-1894 and invents its own specific bounce format, the "qmail-send bounce message format (QSBMF)". Qmail claims QSBMF is easy to parse, but it has serious shortcomings:

Fix: none

Workaround: Frederik P. Lindberg has a patch to preserve MIME-ness in qmail-send bounces. This patch does not provide RFC-1894 compliance, but it attaches the original mail as message/rfc822 part, so "no trailing newline" problems with bouncing are fixed and the bounced mail can be properly decoded by MIME-aware mailers when non-ASCII character sets or difficultly readable encodings (such as base64) or formats (HTML, rich text) had to be bounced.

3.4. RFC-1939 (POP3) violation (Mail size)

qmail-pop3d does not calculate mail sizes properly as documented in RFC-1939, section 11, but gives figures that are too low.

Part II: Shortcomings

4. Other technical issues

4.1. Local resource hogging

In 2001, I benchmarked some mail server software packages. While parts of the benchmark were later invalidated, the most important part still stands: qmail takes much more synchronous writes per mail than any other big Unix MTA and is outperformed even by sendmail, now that CPU power is so cheap compared with fast disk I/O. qmail has been the mail transfer agent with the lowest throughput (pieces of mail per second) in my benchmark.

4.2. Qmail is not softupdates aware.

Qmail does not support softupdated FFS. A softupdated FFS strives to reduce the amount of synchronous writes necessary, while guaranteeing file system consistency. While Dan is right that qmail is currently unsafe on softupdated FFS, it could be made safe. (Postfix, for example, can safely be used with its /var/spool/postfix on a softupdated FFS.)

4.3. Bandwidth hogging (violates SHOULD clause in RFC-2821)

qmail unbundles all mail. Common other mail software transfers a mail for a@same.example.com and b@same.example.com in the same transaction. qmail makes two separate mail transactions of this, one for a@same.example.com, one for b@same.example.com. This consumes your bandwidth, you pay twice with qmail.

RFC-2821, section "4.5.4.1 Sending Strategy", recommends that multi-RCPT be sent when possible: "When a mail message is to be delivered to multiple recipients, and the SMTP server to which a copy of the message is to be sent is the same for multiple recipients, then only one copy of the message SHOULD be transmitted. That is, the SMTP client SHOULD use the command sequence: MAIL, RCPT, RCPT,... RCPT, DATA instead of the sequence: MAIL, RCPT, DATA, ..., MAIL, RCPT, DATA. However, if there are very many addresses, a limit on the number of RCPT commands per MAIL command MAY be imposed. Implementation of this efficiency feature is strongly encouraged."

Technically, unbundling is only required for VERP mail which is exclusively used by mailing list manager software.

4.4. No client-side ESMTP PIPELINING

While ESMTP PIPELINING can considerably reduce the time it takes to transmit the mail envelope, qmail itself does not implement it when sending mail (it does when receiving it). The serialmail package does implement it, but it is not part of qmail and still suffers from qmail's mail unbundling, so there is little to gain without fixes to qmail-send to address the problems mentioned in the previous section.

Workaround: none

Fix: Rolf Eike Beer has an ESMTP patch for SIZE and PIPELINING to enhance qmail-remote.

Note: instead, Dan J. Bernstein has invented a new protocol, quick mail queuing protocol (QMQP for short) which is outperformed by ESMTP PIPELINING on large volumes. QMQP has the additional disadvantage that you cannot save bandwidth by rejecting mail for unknown local users or from unwanted senders right after the envelope: the envelope is sent after the mail.

4.5. Delayed bounces (also waste of bandwidth and spam-friendly)

Qmail generally accepts all mail except relay attempts (provided /var/qmail/control/rcpthosts exists) and mail from blacklisted senders (/var/qmail/control/badmailfrom). That means: if someone sends a mail to you for a non-existent user (some spammers are notorious for grabbing Message-IDs and considering them as e-mail addresses) at your site, qmail will happily take the mail in, say "250 Ok" to the sender, then figure it has no user to deliver to, generate a bounce and send it back. Now assume someone sends a 5 MB mail: qmail accepts 5 MB in, and tries to send a 5 MB bounce out: 10 MB wasted.

Even worse, if the sender address is forged, qmail may annoy the wrong site with the 5 MB bounce, or it may not be able to deliver the bounce at all, and it may happen that the bounce sticks in your queue until its lifetime is expired. qmail does log the sender's IP, but that usually does not help much, particularly not if the bounce has been sent to an innocent.

Fix/Workaround: see section 1.2.

4.6. Parallel delivery of qmail-remote sets back.

Florian Weimer reported that he has seen excessive parallel delivery of qmail-remote cause unnecessary deferrals of mail. This happens if qmail-remote opens more SMTP connections to the remote MX than the remote site wants to handle (if the other site has a tighter concurrency limit than qmail-remote). Some qmail-remote processes then see "Connection refused" or some temporary error and defer the mail, although the mail could be delivered if qmail opened less connections to the remote site at the same time. Further research needs to be done in this area.

4.7. Mail is sent from "anonymous", particularly by daemons.

This behavior is undocumented, therefore it is listed here.

qmail-inject (thus, qmail's sendmail wrapper as well) only looks at the environment to determine the mail sender. It checks the QMAILUSER, MAILUSER, USER and LOGNAME variables in that order until it finds a match. If qmail-inject finds no match, it puts "anonymous" as the sender. Your daemon program however will not have USER and LOGNAME set in the environment, because these are set by the login(1) program -- that is not used by your daemon. qmail-inject does not look at getuid() and getpwuid(), which it should.

Fix: None.

Workaround: explicitly set the sender address, with sendmail's -f option (but see 4.12), or in the environment.

4.8. qmail is hogged by high rate of incoming mail.

qmail-send becomes a bottleneck as mail is injected at a high rate. I already noted this in my 2nd qmail benchmark. As mail comes in, the "todo" preprocessing of qmail's brings down the qmail-remote concurrency, which delays mail because most qmail-remote slots are idle.
André Oppermann has graphs to show this. Check near his "silly qmail syndrome patch".

Workaround: none.

Fix: André Oppermann has a has a patch to remedy this. Check near his "silly qmail syndrome patch".

4.9. errno misuse

Some qmail source files fail to #include the <errno.h> system header and assumes extern int errno; instead. While this works on many systems, POSIX requires the inclusion of the <errno.h> header to declare errno. qmail relies on undocumented errno behavior, and this breaks on GNU libc since release 2.3.1.

While DJB now recommends using -include errno.h in conf-cc, this workaround is not sufficiently portable.

Fix: Phil Edwards' errno.h patch.

4.10. Integer overflow in qmail-smtpd

Georgi Guninski reports: Qmail-smtpd uses a signed index variable into a reference string, without checking for overflow. Sending a header line of 2^31 bytes without intervening LF character causes qmail-smtpd to crash.

4.11. qmail does not recognize 0.0.0.0 as special address

Scott Gifford found that qmail does not consider 0.0.0.0 a local address (it is a special address for local host in current network), and tries to forward messages there in the belief that it is a remote site. This leads to a local mail loop in many setups. (I'm not sure if the problem is large enough to warrant calling it a denial of service, or if hop counting will finally eject and bounce the message.)

Workaround: none

Fix: Scott Gifford's qmail-0.0.0.0.patch (there's also qmail-0.0.0.0.README).

4.12. qmail's sendmail -f is not compatible with sendmail.

qmail's sendmail wrapper does not implement -f properly, in that it does not set the default From: address for messages lacking one, but only setting the envelope. This differs from original sendmail behavior and breaks some applications.

Workaround: none

Fix: David Phillips's sendmail-flagf.patch.

4.13. qmail-remote does not handle CR properly.

qmail-remote will pass on bare CR and insert a CR before any LF. This can cause messages with bare CR or CRCRLF to appear on the wire.

Workaround: none

Fix: Rolf Eike Beer has a patch, qmail-remote-CRLF.diff, that will turn CR to CRLF and leave CRLF as is.

4.14. qmail-smtpd does not detect CR LF properly on packet boundaries.

Mark Martinec reports that qmail-smtpd will incorrectly refuse to accept messages and refer the sender to http://cr.yp.to/docs/smtplf.html if the CR is the last character in a packet and LF the first character in the next packet.

Workaround: if you have control over the sending side of the connection, make sure that CRLF sequences are always sent in the same packet. Mark Martinec reports that this can be achieved, at a cost, by sending messages line by line.

Fix: none known.

4.15 qmail always generates GMT time zone dates

While this is not a reason for concern, some people prefer dates to be in the local time zone.

Fix: There is a patch to let qmail emit dates in the local time zone. It is mirrored at various sites and has varying author designations, so I cannot attribute this to a particular author.

5. Misrepresentation

5.1. Security guarantee

The security guarantee is a smoke ball. Several people, among them Wietse Venema and Georgi Guninski, have documented exploitable qmail security bugs, yet the USD 500 have never been paid to either of them. DJB denies Guninski's claim by making assumptions about resource limits that people should have established, without documenting them in the qmail installation instructions.

The remainder of this section are personal notes:

DJB's code is flawed, and does not hold up to DJB's claims to write bug-free code. DJB's assumptions for instance that QMTP clients would only send properly formatted netstrings (and omitting validation that characters are actually in the 0-9 range), using signed index variables, improper bounds checking, making assumptions about data type widths, assuming that users know qmail is insecure without memory limits, show that DJB is not the magic programmer some people believe him to be (and document so on their fan sites) after his bold claims.

DJB failed to document crucial assumptions his code made about changes the administrator should have made to the enviroment, and which are non-default. If Guninski's default install of qmail on FreeBSD 5.4 amd64 gives remote root access, Guninski deserves the 500 dollars, period. Other people, for instance, James Craig Burley, think the same.

DJB's denying bug claims makes his security guarantee idle talk. DJB's not fixing the security bugs (either through documentating safe installation procedures inside the tarball, or through changing code) so that the default qmail install becomes secure, is a disservice to the users, leaving many sites vulnerable to dangers that should be avoided. Denying bug claims doesn't make the code more secure, nor does shifting the blame.

DJB has lost his face, not through the bugs found themselves (to err is human, after all), but denying their existence and shifting the blame on the end user whom DJB failed to inform about qmail requirements, and in some cases attacking the messenger, too, or trying to deny people the moral right to report bugs if the reporter's own code has bugs.

Part III: Wishlist

6. Wishlist

6.1. Copy bounces to postmaster.

Qmail has no native feature to forward bounces to the local postmaster.

6.2. Let qmail accept bare LF in the mail body.

Qmail will tell other postmasters to piss off and go fix their mailers unless they strictly adhere to the CRLF standard. This violates the basic interoperability principle "Be liberal in what you accept, and be conservative in what you send.".

Workaround: Wrap the fixcrio program around qmail-smtpd. fixcrio ships with UCSPI-TCP.

Fix: Dean Gaudet has a patch to let qmail-smtpd accept bare LF at http://www.arctic.org/~dean/patches/qmail-0.95-liberal-lf.patch.


Appendix

A. Revision history

2006-07-14
Section 4.15 (local time zone) new, hint from Robert Felber.
General: added URL to this document's home location, to avoid people from linking to it improperly.
2006-04-06
Section 4.14 (qmail misdetects CR|LF on packet boundary): new, after info from Mark Martinec.
2006-01-04
General: Make the document more accessible.
General: Converted to (strict) HTML 4.01, converting a few tags to CSS.
General: Fix punctuation in headers.
Preface: Courier-IMAP recommendation removed, I've switched to Dovecot too long ago to still recommend Courier-IMAP. Rewritten.
Section 1.2: update link to Paul Jarc's patch, avoiding 301-type redirect.
Section 1.3: qmail-qmtpd buffer overflow new (Guninski security advisory #67, 2004).
Section 1.4: remote root access on machines with large memory (Guninski security advisory #74, 2005).
Section 2.2: update link to maildrop, avoiding 301-type redirect.
Section 3.3: Chase changed link to qmail-mime.tgz.
Section 4.4: add Rolf Eike Beer's qmail-remote SIZE/PIPELINING patch.
Section 4.9: Chase changed GMANE URI layout. Mention -include errno.h doesn't work.
Section 4.10 (qmail-smtpd integer overflow, Guninski security advisory #65, 2004) new.
Section 4.11: (0.0.0.0 problem) new
Section 4.12: (sendmail -f compatibility) new.
Section 4.13: (qmail-remote CR handling) new.
Section 5.1: revise section on the security guarantee.
Revision history: rearranged today's entries in document order.
2004-10-28
Section 3.4: (POP3 size bug) new.
2004-03-05
Section 4.5: refer to section 1.2 for workarounds.
Mention Paul Jarc's qmail-realrcptto patch and Erwin Hoffmann's RECIPIENTS extension.
2003-11-03
Section 1.2: Add pointers to Eben Pratt's goodrcptto patch, Tim Clarke's qmail-smtpd patch and Bruce Guenter's mailfront package. Suggested by Tim Clarke, thanks!
Add GnuPG signature for patch-qmail-1.03-4.3BSD-ipalias.
2003-08-14
Section 2.3: Felix von Leitner reported that qmail fails to detect IP aliases properly on Linux. Add a patch.
2003-06-20
Section 3.3: Adam Aube pointed out that Frederik P. Lindberg had a patch to make qmail bounce the original message as MIME attachment.
Mention that non-ASCII or base64 mail that ships in a bounce may not be readable.
2003-06-19
Move 1.3 to 4.9 because it's not security related.
2003-06-16
Add link to Toby Betts' qmail-1.03-maildir-uniq.patch to section 2.2.
2003-04-30
Add section 2.2 on maildir name collisions.
2003-02-20
Run through ispell.
2003-02-19
Add stronger partitioning to point out my opinion on what constitutes a bug, a shortcoming or a wish for improvement. Suggested by Sven Paas.
Add section 1.3: errno compile failure on GNU libc 2.3.1.
2003-02-18
Move qmail patches from postfix/ subdirectory into qmail/ subdirectory.
Refine RFC-2821 compliance patch to incorporate the semantics of Adrian Ho's accept-5xx.patch.
2003-01-29
Mention fixcrio in section 6.2, suggested by Balazs Nagy.
2002-11-12
Clean up preface.
Clean up and META description (invisible).
After comments from Joe Kelsey:
  • 3.2: quote from RFC-2821
  • 4.3: quote from RFC-2821 and document not meeting SHOULD recommendation
2002-09-19
Add section 6.2 on Dean Gaudet's qmail liberal LF patch
2002-09-16
Add section 4.8 on big todo hogging and André Oppermann's EXTTODO patch
2002-09-11
After comments from Paul Jarc:
  • 1.1: reword, list Dave Sill's suggestions as fix.
  • 1.2: refer to section 4.5
  • 4.5: add Paul Jarc's qmail-queue wrapper suggestion.
  • 5.1: drop compensation issue
2002-09-06
After comments from Russell Nelson:
  • 4.7: add note about missing documentation
  • 5.1: drop non-payment of Wietse Venema, it's a moot point.
  • 6.1: renamed to "Wishlist".
  • meta: (invisible) Add details to the description line to prevent that someone infers problems of qmail-2 from this document.
2002-07-15
Add section 1.2, documenting delayed bounces as security sensitive (bandwidth amplification, contacts third host).
2002-06-29
Add section 4.7 on "daemon sends mail as anonymous" issue.
2002-06-13
Add section 6.1 on forwarding bounces to postmaster.
2002-06-01
Add section 4.6 on the downsides of qmail's parallel delivery, based on observations by Florian Weimer.
2002-05-30
Added revision history.
Add section 4.5 on delayed bounces.

B. Copyright and License

This document is © Copyright 2002 - 2007 by Matthias Andree. All rights reserved.


Matthias Andree
Valid HTML 4.01! Valid CSS!