Bug#954192: exim4-config: prdr_enable = true breaks exim4+dkimproxy when using multiple recipients

Thu Mar 19 18:41:34 GMT 2020

Hi Marc,

On 2020-03-19 08:44, Marc Haber wrote:
> On Wed, Mar 18, 2020 at 06:32:05AM +0100, Niki Hammler wrote:
>> This worked flawlessly until jessie (for me, from 2008 until now). However, with prdr_enable = true, exim4 hangs when looping back the message when
>> using multiple recipients. It hangs with message:
>>
>>   353 PRDR content analysis beginning
> 
> That happens when dkimproxy re-delivers the message back to exim? What's
> the SMTP dialog before? Does exim advertise PRDR? Does the client
> request it?

Yes, it happens when dkimproxy redelivers it.
However, as I understand dkimproxy, don't think of it as a full-fledged
SMTP server. Once I connect to dkimproxy, it transparently opens back a
connection to exim. So the greeting message comes actually from exim:

mail:~# netstat -anp | grep 10028
tcp        0      0 127.0.0.1:10028         0.0.0.0:*
LISTEN      6988/perl
mail:~# ps aux |grep [6]988
dkimpro+  6988  0.0  0.0  22400 16316 ?        S    Mär18   0:00
/usr/bin/perl -I/usr/lib /usr/sbin/dkimproxy.out --domain=nobaq.net
--method=simple --conf_file=/etc/dkimproxy/dkimproxy_out.conf
--keyfile=/var/lib/dkimproxy/private.key --user=dkimproxy
--group=dkimproxy --daemonize --pidfile=/var/run/dkimproxy.out
--signature=dkim --signature=domainkeys --min_servers=5
mail:~# telnet 127.0.0.1 10028
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 mail.nobaq.net ESMTP Exim 4.89 Thu, 19 Mar 2020 19:15:45 +0100

dkimproxy only changes the data it passes back and forth between exim.
Since client and server are technically exim, yes, server advertises and
client requests it.

See below:

>> I verified the issue observing the traffic transmitted to dkimproxy while sending a message to only one recipient:
>>
>> # ngrep -d lo -W byline -q port 10028
>> [...]
>> T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
>> 250 OK id=1jEPuw-0005Cq-IJ.
>>
>> T 127.0.0.1:48486 -> 127.0.0.1:10028 [AP]
>> QUIT.
>>
>> T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
>> 221 mail.nobaq.net closing connection.
>>
>>
>> All good, just as expected.
>> Now repeating the whole thing while sending the message to TWO recipients:
>>
>> # ngrep -d lo -W byline -q port 10028
>> [...]
>> DATA.
>> [...]
>> T 127.0.0.1:10028 -> 127.0.0.1:48586 [AP]
>> 353 PRDR content analysis beginning.
> 
> The things you have left out would have been interesting.

Ok, here is the full trace. First case, only one recipient:

# ngrep -d lo -W byline -q port 10028
interface: lo (127.0.0.0/255.0.0.0)
filter: (ip or ip6) and ( port 10028 )

T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
220 mail.nobaq.net ESMTP Exim 4.89 Wed, 18 Mar 2020 05:02:54 +0100.

T 127.0.0.1:48486 -> 127.0.0.1:10028 [AP]
EHLO mail.nobaq.net.

T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
250-mail.nobaq.net Hello localhost [127.0.0.1].
250-SIZE 52428800.
250-8BITMIME.
250-PIPELINING.
250-PRDR.
250 HELP.

T 127.0.0.1:48486 -> 127.0.0.1:10028 [AP]
MAIL FROM:<niki at hammler.net> SIZE=3934.
RCPT TO:<niki at aveer.io>.
DATA.

T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
250 OK.

T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
250 Accepted.
354 Enter message, ending with "." on a line by itself.

T 127.0.0.1:48486 -> 127.0.0.1:10028 [AP]
Received: from gate.nobaq.net ([93.83.102.170]:51908
helo=[192.168.200.209]).
.by mail.nobaq.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128).
.(Exim 4.89).
.(envelope-from <niki at hammler.net>).
.id 1jEPut-0005Cg-H8.
.for nhammler at stanford.edu; Wed, 18 Mar 2020 05:02:54 +0100.
To: nhammler at stanford.edu.
From: Nikolaus Hammler <niki at hammler.net>.
Autocrypt: addr=niki at hammler.net; prefer-encrypt=mutual; keydata=[SNIP]
Message-ID: <06d9bce6-f730-5b70-dfa1-52e4bc9a34b3 at hammler.net>.
Date: Wed, 18 Mar 2020 00:02:45 -0400.
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.24).
 Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.5.0.
MIME-Version: 1.0.
Content-Type: text/plain; charset=utf-8.
Content-Language: en-US.
Content-Transfer-Encoding: 7bit.
X-SA-Exim-Connect-IP: 93.83.102.170.
X-SA-Exim-Mail-From: niki at hammler.net.
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.nobaq.net.
X-Spam-Level: .
X-Spam-Status: No, score=0.1 required=5.0
tests=ALL_TRUSTED,AWL,FSL_BULK_SIG,.
.PYZOR_CHECK,TVD_SPACE_RATIO autolearn=no autolearn_force=no.
.version=3.4.2.
Subject: test1.
X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000).
X-SA-Exim-Scanned: Yes (on mail.nobaq.net).
.
test1.
.
..

T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
250 OK id=1jEPuw-0005Cq-IJ.

T 127.0.0.1:48486 -> 127.0.0.1:10028 [AP]
QUIT.

T 127.0.0.1:10028 -> 127.0.0.1:48486 [AP]
221 mail.nobaq.net closing connection.

Second case, having two recipients:

interface: lo (127.0.0.0/255.0.0.0)
filter: (ip or ip6) and ( port 10028 )

T 127.0.0.1:10028 -> 127.0.0.1:48586 [AP]
220 mail.nobaq.net ESMTP Exim 4.89 Wed, 18 Mar 2020 05:05:47 +0100

T 127.0.0.1:48586 -> 127.0.0.1:10028 [AP]
EHLO mail.nobaq.net

T 127.0.0.1:10028 -> 127.0.0.1:48586 [AP]
250-mail.nobaq.net Hello localhost [127.0.0.1]
250-SIZE 52428800
250-8BITMIME
250-PIPELINING
250-PRDR
250 HELP

T 127.0.0.1:48586 -> 127.0.0.1:10028 [AP]
MAIL FROM:<niki at hammler.net> SIZE=3961 PRDR
RCPT TO:<niki.hammler at stanford.edu>
RCPT TO:<nhammler at stanford.edu>
DATA

T 127.0.0.1:10028 -> 127.0.0.1:48586 [AP]
250 OK, PRDR Requested

T 127.0.0.1:10028 -> 127.0.0.1:48586 [AP]
250 Accepted
250 Accepted
354 Enter message, ending with "." on a line by itself

T 127.0.0.1:48586 -> 127.0.0.1:10028 [AP]
Received: from gate.nobaq.net ([93.83.102.170]:52099 helo=[192.168.200.209])
	by mail.nobaq.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
	(Exim 4.89)
	(envelope-from <niki at hammler.net>)
	id 1jEPxh-0005EN-6R; Wed, 18 Mar 2020 05:05:47 +0100
To: niki.hammler at stanford.edu, nhammler at stanford.edu
From: Nikolaus Hammler <niki at hammler.net>
Autocrypt: addr=niki at hammler.net; prefer-encrypt=mutual; keydata=[SNIP]
Message-ID: <640a37ad-d435-2f42-c864-38c9cbf0a1ea at hammler.net>
Date: Wed, 18 Mar 2020 00:05:43 -0400
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.24)
 Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.5.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-SA-Exim-Connect-IP: 93.83.102.170
X-SA-Exim-Mail-From: niki at hammler.net
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.nobaq.net
X-Spam-Level:
X-Spam-Status: No, score=0.1 required=5.0
tests=ALL_TRUSTED,AWL,FSL_BULK_SIG,
	PYZOR_CHECK,TVD_SPACE_RATIO autolearn=no autolearn_force=no
	version=3.4.2
Subject: test2
X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000)
X-SA-Exim-Scanned: Yes (on mail.nobaq.net)

test2

.

T 127.0.0.1:10028 -> 127.0.0.1:48586 [AP]
353 PRDR content analysis beginning.

>> Setting
>>
>>   prdr_enable = false
>>
>> fixes the issue. But this is far from optimal.
> 
> I am not sure, but if the value for prdr_enable is expanded at
> connection-time, one could use an expression that expands to "true" in
> the default case and to "false" in the "I am talking to dkimproxy" case.

I don't think so but I am not an exim expert.

I tried

prdr_enable = ${if eq{$received_port}{10029} {false}{true}}

but I get the error

Mär 19 19:32:06 mail exim4[10887]:   "" is not a valid value for the
"prdr_enable" option

> Generally, having messages looped out of exim and in again is seldomly a
> good idea because internal information is lost between the two exim
> runs.
> 
>> At the very least, information about prdr (and implications) would be useful to prevent people from debugging for days why suddenly after
>> 12 years there are weird redeliveries and mails stuck in the queue.
> 
> prdr has a (short) explanation in exim's spec.txt. I don't think that it
> should be the responsibility of the packaging to explain every feature
> of e-mail transport>
>> Furthermore, a Debian-style control macro would be desirable that allows more flexible control without directly changing the config file
>> (like MAIN_TLS_ADVERTISE_HOSTS etc).
> 
> Agreed. Can we have a documented patch please?

I haven't found one yet, unfortunately.
My attempt above with variable expansion did not work.
Unfortunately I found very little information about prdr.

> The dkimproxy package could also dump a configuration snippet. Allowing
> this is one of the reasons we came up with split config.

I agree, this is the reason why I said I am unsure of submitting the bug
to exim4-config or dkimproxy. I couldn't select two packages.

>> The next best solution would require exim4 changes directly in order to prevent use of PRDR in the exim<->dkimproxy loop.
> 
> How would that be done?

The suggestion above, having prdr_enable not set for dkimproxy
connections (10028 and 10029).

Please advise the best way forward. Shall I resubmit to dkimproxy package?

Thanks
NH