Bug#954192: exim4-config: prdr_enable = true breaks exim4+dkimproxy when using multiple recipients

Wed Mar 18 05:32:05 GMT 2020

Package: exim4-config
Version: 4.89-2+deb9u6
Severity: normal

While I am using oldstable (stretch) I know based on my debugging that the same issue applies to newer versions (buster, see below).
Note that this is also related to package dkimproxy but I am not sure where to best report.

Starting from stretch, the following option is enabled by default:

   prdr_enable = true

This advertises per recipient data responses.
The way to use dkimproxy together with exim4 is to loop the messages through a pseudo SMTP server provided by dkimproxy which is then
passed back to exim.
One example is written here: https://lxadm.com/Setting_up_DKIMproxy_with_Exim_for_DKIM_and_DomainKeys_signing
But it basically involves the following router+transport:

dkimproxy:
  driver = manualroute
  condition = "${if eq {$interface_port}{10029} {0}{1}}"
  transport = dkimproxy_smtp
  route_list = "* localhost byname"
  self = send

dkimproxy_smtp:
  driver = smtp
  port = 10027
  allow_localhost

Every normal email (that is not coming from port 10029) is sent to localhost:10027 where dkimproxy listens. dkimproxy adds the DKIM signatures
and sends back the message to exim via port 10029. Then normal processing then continues.

This worked flawlessly until jessie (for me, from 2008 until now). However, with prdr_enable = true, exim4 hangs when looping back the message when
using multiple recipients. It hangs with message:

  353 PRDR content analysis beginning

The result is that exim puts the message into the queue because the it does not receive a success message (but hangs).
However, the message itself is received and delivered to the recipients.
Now the message is stuck in the queue forever (although devlivered successfully!) and re-delivers after each queue re-run when
retry time is met (24hrs). The messages have to be removed manually via "exim4 -Mrm" after the timeout has occured.

To summarize, the issue appears only under the following conditions:

1. The user upgraded from Debian lenny to Debian stretch (or newer!)
2. The user uses exim4 as MTA
3. The user uses dkimproxy to add DKIM signatures
4. The user sends an email containing MULTIPLE recipients (such that prdr is used)

I verified the issue observing the traffic transmitted to dkimproxy while sending a message to only one recipient:

# ngrep -d lo -W byline -q port 10028
[...]
T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
250 OK id=1jEPuw-0005Cq-IJ.

T 127.0.0.1:48486 -> 127.0.0.1:10028 [AP]
QUIT.

T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
221 mail.nobaq.net closing connection.

All good, just as expected.
Now repeating the whole thing while sending the message to TWO recipients:

# ngrep -d lo -W byline -q port 10028
[...]
DATA.
[...]
T 127.0.0.1:10028 -> 127.0.0.1:48586 [AP]
353 PRDR content analysis beginning.

>From there, it hangs, the connection is never terminated (only after the timeout).

Again, this is not only exim4 specific because if I replay the ngrep traffic manually over telnet the message is received successfully.
The issue only happens if the message is transparently looped through dkimproxy (which receives data from port 10028 and on-the-fly connects
back to exim on port 10029 and instantly loops back the message).

Setting

  prdr_enable = false

fixes the issue. But this is far from optimal.

At the very least, information about prdr (and implications) would be useful to prevent people from debugging for days why suddenly after
12 years there are weird redeliveries and mails stuck in the queue.

Furthermore, a Debian-style control macro would be desirable that allows more flexible control without directly changing the config file
(like MAIN_TLS_ADVERTISE_HOSTS etc).

The next best solution would require exim4 changes directly in order to prevent use of PRDR in the exim<->dkimproxy loop.

And the best solution would be to fix this bug altogether but I am not completely sure why exim4 is hanging there in the first place
and how much it is related to dkimproxy.

-- Package-specific info:
Exim version 4.89 #1 built 03-Sep-2019 18:01:38
Copyright (c) University of Cambridge, 1995 - 2017
(c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007 - 2017
Berkeley DB: Berkeley DB 5.3.28: (September  9, 2013)
Support for: crypteq iconv() IPv6 PAM Perl Expand_dlfunc GnuTLS move_frozen_messages Content_Scanning DKIM DNSSEC Event OCSP PRDR PROXY SOCKS TCP_Fast_Open
Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb dsearch ldap ldapdn ldapm mysql nis nis0 passwd pgsql sqlite
Authenticators: cram_md5 cyrus_sasl dovecot plaintext spa tls
Routers: accept dnslookup ipliteral iplookup manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Fixed never_users: 0
Configure owner: 0:0
Size of off_t: 8
Configuration file is /var/lib/exim4/config.autogenerated
# /etc/exim4/update-exim4.conf.conf
#
# Edit this file and /etc/mailname by hand and execute update-exim4.conf
# yourself or use 'dpkg-reconfigure exim4-config'
#
# Please note that this is _not_ a dpkg-conffile and that automatic changes
# to this file might happen. The code handling this will honor your local
# changes, so this is usually fine, but will break local schemes that mess
# around with multiple versions of the file.
#
# update-exim4.conf uses this file to determine variable values to replace
# the DEBCONFsomethingDEBCONF strings in the configuration template files.
#
# Most settings found in here do have corresponding questions in the
# Debconf configuration, but not all of them.
#
# This is a Debian specific file

dc_eximconfig_configtype='internet'
# together with "localhost" --> local_domains
dc_other_hostnames='/etc/exim4/domains_local.txt:/etc/exim4/domains_hubbed.txt:/etc/exim4/domains_virtual.txt:/etc/exim4/domains_virtual_forward.txt'
dc_local_interfaces=''
# dc_readhost='mail.nobaq.net'

# this sets MAIN_RELAY_TO_DOMAINS
#dc_relay_domains='stiftingtal.net;w-lan.mine.nu'
dc_relay_domains=''
dc_minimaldns='false'
dc_relay_nets='192.168.200.0/24 : 93.83.102.170 : 93.83.102.173 : 188.20.174.102 : 2001::470::770d::3::::173 : 2001::470::770d::200::::121'
dc_smarthost=''
CFILEMODE='644'
dc_use_split_config='true'
dc_hide_mailname='false'
dc_mailname_in_oh='true'
dc_localdelivery='maildir_home'
dc_readhost=''
mailname:mail.nobaq.net
# /etc/default/exim4
EX4DEF_VERSION=''

# 'combined' -	 one daemon running queue and listening on SMTP port
# 'no'       -	 no daemon running the queue
# 'separate' -	 two separate daemons
# 'ppp'      -   only run queue with /etc/ppp/ip-up.d/exim4.
# 'nodaemon' - no daemon is started at all.
# 'queueonly' - only a queue running daemon is started, no SMTP listener.
# setting this to 'no' will also disable queueruns from /etc/ppp/ip-up.d/exim4
QUEUERUNNER='combined'
# how often should we run the queue
QUEUEINTERVAL='15m'
# options common to quez-runner and listening daemon
COMMONOPTIONS=''
# more options for the daemon/process running the queue (applies to the one
# started in /etc/ppp/ip-up.d/exim4, too.
QUEUERUNNEROPTIONS=''
# special flags given to exim directly after the -q. See exim(8)
QFLAGS=''
# options for daemon listening on port 25
# SMTPLISTENEROPTIONS='-oX 465:25 -oP /var/run/exim4/exim.pid'

-- System Information:
Debian Release: 9.12
  APT prefers oldstable
  APT policy: (500, 'oldstable')
Architecture: i386 (x86_64)

Kernel: Linux 4.19.0-6-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_AT.UTF-8, LC_CTYPE=de_AT.UTF-8 (charmap=UTF-8), LANGUAGE=de_AT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)

Versions of packages exim4-config depends on:
ii  adduser                3.115
ii  debconf [debconf-2.0]  1.5.61

exim4-config recommends no packages.

exim4-config suggests no packages.

-- Configuration Files:
/etc/exim4/conf.d/acl/30_exim4-config_check_rcpt changed [not included]
/etc/exim4/conf.d/auth/30_exim4-config_examples changed [not included]
/etc/exim4/conf.d/main/02_exim4-config_options changed [not included]
/etc/exim4/conf.d/main/90_exim4-config_log_selector changed [not included]
/etc/exim4/passwd.client [Errno 13] Keine Berechtigung: '/etc/exim4/passwd.client'

-- debconf information excluded