[Amavisd-new-debian-devel] amavisd-new 2.5.2-1
Ondřej Surý
ondrej at sury.org
Thu Sep 13 08:49:54 UTC 2007
Hi guys,
attached is patch to update your mercurial repo to 2.5.2-1.
Compiles cleanly.
I have yet to test it on some server :-).
I had to update few patches.
Ondrej
--
Ondřej Surý <ondrej at sury.org> *** http://blog.rfc1925.org/
Kulturnà obÄasnÃk *** http://www.obcasnik.cz/
Nehoupat, prosÃm *** http://nehoupat.blogspot.com/
-------------- next part --------------
# HG changeset patch
# User Ondřej Surý <ondrej at sury.org>
# Date 1189673239 -7200
# Node ID 8a46f5ab6edd3931fc1d4f88e268d5ee5850c82e
# Parent f14c306762a6d899163a5bb69b11d2b432041060
Update HEAD to 2.5.2 upstream source
diff --git a/INSTALL b/INSTALL
--- a/INSTALL
+++ b/INSTALL
@@ -26,19 +26,16 @@ Prerequisites:
file(1) utility is required, the most recent version is heartly recommended!
There are a number of security and robustness problems with earlier versions.
-Use file(1) 4.06 or later to avoid it crashing upon seeing certain files
-and to avoid possible control characters leaking into its output.
-
-Archive::Tar (Archive-Tar-x.xx)
+Use file(1) version 4.21 or later to avoid known security vulnerabilities.
+
Archive::Zip (Archive-Zip-x.xx) (1.14 or later should be used!)
Compress::Zlib (Compress-Zlib-x.xx) (1.35 or later)
Convert::TNEF (Convert-TNEF-x.xx)
-Convert::UUlib (Convert-UUlib-x.xxx) (1.05 or later, stick to new versions!)
+Convert::UUlib (Convert-UUlib-x.xxx) (1.08 or later, stick to new versions!)
MIME::Base64 (MIME-Base64-x.xx)
-MIME::Parser (MIME-Tools-x.xxxx) (latest version from CPAN - currently 5.417)
+MIME::Parser (MIME-Tools-x.xxxx) (latest version from CPAN - currently 5.420)
Mail::Internet (MailTools-1.58 or later have workarounds for Perl 5.8.0 bugs)
Net::Server (Net-Server-x.xx) (version 0.88 finally does setuid right)
-Net::SMTP (libnet-x.xx, ports/net/p5-Net) (>= libnet-1.16 for performance)
Digest::MD5 (Digest-MD5-x.xx) (2.22 or later)
IO::Stringy (IO-stringy-x.xxx)
Time::HiRes (Time-HiRes-x.xx) (use 1.49 or later, older can cause problems)
@@ -76,6 +73,7 @@ The most crucial programs are marked wit
nomarch: http://rus.members.beeb.net/nomarch.html
arc: ftp://ftp.kiarchive.ru/pub/unix/arcers/
lha: http://www2m.biglobe.ne.jp/~dolphin/lha/prog/
+ 7z: http://p7zip.sourceforge.net/, http://www.7-zip.org/
unarj: ftp://ftp.kiarchive.ru/pub/unix/arcers/
arj: http://testcase.newmail.ru/files/ (arj is preferable to unarj)
rar, unrar: http://www.rarsoft.com/, ftp://ftp.kiarchive.ru/pub/unix/arcers/
@@ -207,6 +205,9 @@ Installing the daemon:
chown -R amavis:amavis /var/amavis/home
chmod 750 /var/amavis /var/amavis/home
+ If $TEMPBASE resides on a dedicated file system, it may be prudent to
+ specify mount options: noexec,nosuid,nodev.
+
- install virus scanners (if they are to be used), and Perl module
Mail::SpamAssassin (if desired), and adjust variables in /etc/amavisd.conf.
There are several other Perl modules needed by amavisd daemon
@@ -220,9 +221,10 @@ Installing the daemon:
Some virus scanners may require write permission to the $TEMPBASE directory
to be able to create auxiliary files there.
- If different UID is preferred for an AV scanner, a solution for
- ClamAV is to add user clamav to the amavis group, and then add
- AllowSupplementaryGroups to clamd.conf.
+ If a different UID is preferred for an AV scanner, a solution for ClamAV
+ is to add user clamav to the amavis group (e.g.: vscan:*:110:clamav
+ in a file /etc/group), and then add: AllowSupplementaryGroups yes
+ to clamd.conf.
- start the program 'amavisd', either as root (possibly with option
-u user), or with su(1) as the user chosen above. It should
diff --git a/MANIFEST b/MANIFEST
--- a/MANIFEST
+++ b/MANIFEST
@@ -14,6 +14,7 @@ amavisd.conf its configuration file
amavisd.conf its configuration file (should go into /etc/)
amavisd.conf-default lists all configuration variables with their defaults
amavisd.conf-sample traditional-style commented amavisd.conf with examples
+amavisd-custom.conf example custom hooks, to be invoked from amavisd.conf
amavisd-agent a demo program to access and display SNMP-like counters
being updated and made available as a Berkeley DB by amavisd
@@ -42,9 +43,3 @@ amavisd-new-qmqpqq.patch adds support
amavisd_init.sh sample init shell script
amavisd-new.spec rpm spec file
-
-Macintosh.tar.gz auto-startup and installation instructions for Macintosh OSX;
- This is a contributed set of files and hasn't been checked
- for correctness or usability by the amavisd-new author.
- In particular, use provided procedures and patches at your
- own risk.
diff --git a/Macintosh.tar.gz b/Macintosh.tar.gz
deleted file mode 100644
Binary file Macintosh.tar.gz has changed
diff --git a/README_FILES/README.courier b/README_FILES/README.courier
--- a/README_FILES/README.courier
+++ b/README_FILES/README.courier
@@ -2,6 +2,10 @@ How to use amavisd-new with Courier
***********************************
by Martin Orr <amavis at martinorr.name>
+
+
+ There may be additional or more up-to-date information at:
+ http://www.martinorr.name/amavisd-new
WARNING: This README applies to the current version of the Courier patch,
and requires Net::Server version 0.90 or later. For older versions of
diff --git a/README_FILES/README.customize b/README_FILES/README.customize
--- a/README_FILES/README.customize
+++ b/README_FILES/README.customize
@@ -1,6 +1,6 @@ Customization of notification messages a
Customization of notification messages and log entries
======================================================
- Mark Martinec <Mark.Martinec at ijs.si>, 2002, 2004, 2006
+ Mark Martinec <Mark.Martinec at ijs.si>, 2002, 2004, 2006, 2007
Since March 2002 amavisd-new provides a way to customize e-mail notification
messages that are sent in response to a virus (and spam) detection,
@@ -67,7 +67,8 @@ The substitution text for the following
is still in place);
h dns name of this host, or configurable name (variable $myhostname)
HOSTNAME same as h
- n amavis internal message id (as seen in log entries)
+ n amavis internal task id (also called message id, am_id) as shown
+ in the log and by amavisd-nanny, e.g. 58725-05-2
b message digest of mail body (MD5, hex)
date_unix_utc timestamp of the message reception - Unix time
@@ -87,16 +88,16 @@ The substitution text for the following
a original SMTP session client IP address(empty if unknown,e.g. no XFORWARD)
g original SMTP session client DNS name (empty if unknown, e.g. no XFORWARD)
e best guess of the originator IP address collected from the Received trace
- l (letter ell) true if sender is local;
- - if SMTP client IP is known (via XFORWARD or AM.PDP from mailer), then
- locality is determined by checking client IP address against @mynetworks
- - if SMTP client IP is NOT provided by MTA, locality is determined by
- comparing sender mail address to @local_domains_maps (can be faked);
+ l (letter ell, suggesting 'local') is true if a variable 'originating' is
+ true, and is an empty string otherwise; the boolean variable 'originating'
+ is under policy bank control, and usually corresponds to a sending host
+ (SMTP client's IP address) matching @mynetworks_maps, or the client being
+ an authenticated roaming user;
o best attempt at determining true sender of the virus - normally same as %s
S address that will get sender notification;
this is normally a one-entry list containing sender address (same as %s),
but may be unmangled/reconstructed in an attempt to undo the address
- forging done by some viruses; in case of unkown (e.g. forged) sender
+ forging done by some viruses; in case of unknown (e.g. forged) sender
address, the result is empty.
R a list of original envelope recipients
@@ -113,29 +114,66 @@ The substitution text for the following
header_field field body of the header field specified in the argument;
can only obtain header field stored in $msginfo->orig_header_fields,
and only its first occurrence; currently these are: from, to, cc, sender,
- subject, received, message-id, resent-message-id, precedence, user-agent,
- x-mailer, dkim-signature, domainkey-signature, authentication-results;
- argument is case-insensitive
- useragent 'User-Agent: ...' or 'X-Mailer: ...' header field (whichever
- is present); note that this is an entire field, including a header
- field head, unlike macros header_field and x-mailer;
- x-mailer 'X-Mailer' header field body (just body, without header field head),
+ subject, received, message-id, resent-message-id, in-reply-to, references,
+ precedence, list-id, user-agent, x-mailer, dkim-signature,
+ domainkey-signature, authentication-results; argument is case-insensitive
+ useragent returns 'User-Agent: ...' or 'X-Mailer: ...' header field
+ (whichever is present); an optional argument specifies whether
+ an entire field is to be returned (empty or unrecognized argument),
+ or just a field name (argument: 'name'), e.g. 'X-Mailer';
+ or just a field body (argument 'body'), e.g. 'Thunderbird_1.5.0.9';
+ x-mailer 'X-Mailer' header field body (just body, without header field name),
deprecated, use macros header_field or useragent instead;
Q MTA queue ID of the message if available (Courier, sendmail milter/AM.PDP)
H a list of all header lines (field may be wrapped over more than one line);
- this does not include the 'Return-Path:' or 'Delivered-To:' headers,
+ this does not include the 'Return-Path:' or 'Delivered-To:' header fields,
which would have been added (or will be added later) by the local
delivery agent if mail would have been delivered to a mailbox.
z original mail size (in bytes)
-
- i long-term unique mail id on this system
+ i long-term unique mail_id on this system, possibly used in log and in
+ quarantine names (also in releasing from a quarantine), e.g. jaUETfyBMJHG
q list of quarantine mailbox names, or empty if not quarantined
- ccat_maj minor category number of the main contents category,
- see constants CC_* in file amavisd
- ccat_min minor category number of the main contents category,
- usually 0 unless more specific information is available
- ccat_name display name of the contents category best describing mail contents
+ ccat_maj (deprecated, use [:ccat|major]) a major category number
+ of the blocking or a main contents category, see constants
+ CC_* in file amavisd
+ ccat_min (deprecated, use [:ccat|minor]) a minor category number
+ of the blocking or a main contents category, usually 0
+ unless more specific information is available (e.g.
+ details about bad header or tag3 spam level)
+ ccat_name (deprecated, use [:ccat|name]) a display name of the
+ blocking or main contents category best describing mail contents
+
+ ccat a new general-purpose macro providing access to information about
+ a mail contents category. Macro 'ccat' takes two optional fixed-string
+ arguments, which are interpreted case-insensitively. In their absence
+ it expands to a string "(maj,min)" which shows a major and a minor
+ contents category number of a blocking ccat for a blocked message,
+ and of a main contents category for a passed message.
+
+ The first argument specifies which attribute of a ccat is to be provided,
+ the second argument specifies whether a main or a blocking contents
+ category is to be consulted:
+
+ The first argument may be any of the following strings:
+ name ... provide a human-readable name of a ccat (%ccat_display_names)
+ major ... provide a number: a major contents category,
+ values correspond to CC_* constants in the program
+ minor ... provide a number: a minor contents category, often a 0
+ <empty>... empty argument (also a default) results in a string "(maj,min)"
+ is_blocking ... '1' if blocking_ccat is true (message is being blocked),
+ or an empty string when a message is being passed;
+ is_nonblocking .. the opposite: '1' if blocking_ccat false, '' otherwise
+ is_blocked_by_nonmain .. '1' if blocking_ccat is true
+ _and_ is different from a main contents category;
+
+ The second argument may be any of the following strings:
+ main ... provide information on main contents category
+ when asked for name/major/minor/<empty>
+ blocking.. provide information on blocking contents category if it exists,
+ otherwise it falls back to providing info on main ccat;
+ this is also a default in the absence of this argument;
+
v output of the (last) virus checking program
V a list of virus names found; contains at least one entry (possibly empty)
if a virus was found, otherwise a null list
@@ -181,23 +219,41 @@ The substitution text for the following
Note that to get a percent character it needs to be doubled, to avoid
its special meaning as a macro call introducer,
e.g. [:sprintf|%%s %%.1f|text|[:SCORE]]
- dquote encloses its argument in double quotes and replace existing
- double quotes by \" (suitable to sanitize Subject header field);
- provisional - exact interpretation may change;
+ join (just like a Perl function join): the first argument is a separator
+ string, remaining arguments are strings to be concatenated, with a
+ separator string inserted at every concatenation point;
+ dquote encloses its argument in double quotes and doubles existing double
+ quotes within a string (suitable to sanitize Subject header field,
+ e.g. ab"oh"cd -> "ab""oh""cd"); provisional - exact interpretation
+ may change (and has changed, prior to 2.4.5 double quotes were replaced
+ by \", which made parsing tricky as backquotes themselves were not
+ escaped);
uquote replaces one or more consecutive space or tab characters by '_',
but does not protect existing underlines, which makes it a lossy
transformation (suitable for logging of From or To header fields);
provisional - exact interpretation may change;
- a couple of SpamAssassin lookalike macros with the same names and arguments
+ supplementary_info gives access to some additional information provided by
+ content scanners, such as a provided by SpamAssassin API routine get_tag.
+ The macro takes two arguments, the first is a tag name (a name of some
+ attribute which is expected to provide an associated value), the second
+ argument is a sprintf format string and is optional, if missing a %s
+ is assumed. Currently the only available attributes are AUTOLEARN,
+ AUTOLEARNSCORE, SC, SCRULE, SCTYPE, RELAYCOUNTRY, and LANGUAGES.
+ These are nonempty only when an associated SpamAssassin plugin or
+ function is enabled.
+
+
+ A couple of SpamAssassin look-alike macros with the same names and arguments
as in SA, see 'man Mail::SpamAssassin::Conf' for details:
- AUTOLEARN autolearn status
+ AUTOLEARN autolearn status (deprecated, use supplementary_info instead)
+
DATE same as date_rfc2822_local
SCORE similar to macro 'c', but returns a single number (sum of
SA score and boost), and allows padding as per SA documentation.
In a per-message log ($log_templ) when a message has multiple
- recipients, a minimum value is given;
+ recipients, a minimum value across all recipients is given;
STARS score as in macro SCORE, but represented as a bar of characters
REPORT a SA terse report of tests hit (for header reports)
SUMMARY similar to macro A, but provides a single multiline string,
@@ -209,6 +265,11 @@ The substitution text for the following
YESNO similar to macro '2', but provides Yes/No instead of 1/0
YESNOCAPS similar to macro '2', but provides YES/NO instead of 1/0
REQD minimal tag2 level of all recipients
+
+ and two additional SA-lookalikes, but with no counterparts in SpamAssassin:
+
+ LOGID log id (a.k.a. am_id) e.g. 58725-05-2, synonym for macro n
+ MAILID mail_id as in quarantine e.g. jaUETfyBMJHG, synonym for macro i
- when $log_recip_templ is expanded (by-recipient log entry), certain
macros keep their general semantics but reflect a value for that recipient:
@@ -225,7 +286,7 @@ The substitution text for the following
%2 above tag2 level for this recipient: Y or 0
%k above kill level for this recipient: Y or 0
REQD recipient's tag2_level
- ccat_maj minor category number, takes into account per-recip bypass_*
+ ccat_maj major category number, takes into account per-recip bypass_*
ccat_min minor category number, takes into account per-recip bypass_*
ccat_name display name of the c.cat, takes into account per-recip bypass_*
remote_mta MTA to which a message was forwarded
@@ -312,11 +373,11 @@ to a macro as its first argument (beside
to a macro as its first argument (besides argument 0, which is a macro
name). Commas within (...) are not special, calls like _TESTS(,)_ and
_SPAMMYTOKENS(2,short)_ still provide a single argument: "," or "2,short"
-respectively, to accomodate SA peculiarity. These all-capitals macros can
+respectively, to accommodate SA peculiarity. These all-capitals macros can
still be called by a normal general-purpose form of a macro call for greater
flexibility, as described above.
-A macro is evaluated only in nonquoted context. Enclosing strings between
+A macro is evaluated only in non-quoted context. Enclosing strings between
tokens [" and "] prevents its evaluation. Quoting may be nested, quote tokens
must be balanced. Evaluating a quoted input strips off one level of quotes.
@@ -393,7 +454,7 @@ as a null lexical separator.
] same thing: a newline-separated list of virus names
[
- %V] a list of virus names, each preceeded by NL and spaces
+ %V] a list of virus names, each preceded by NL and spaces
[ %R |%s --> <%R>|, ] a comma-space separated list of sender/recipient
name pairs where recipient is iterated over the list
diff --git a/README_FILES/README.lookups b/README_FILES/README.lookups
--- a/README_FILES/README.lookups
+++ b/README_FILES/README.lookups
@@ -175,8 +175,8 @@ or false (no, deny, drop). Falling throu
or false (no, deny, drop). Falling through without a match
produces false (undef). Search is case-insensitive.
-lookup_acl is not aware of address extensions and they are not
-handled specially.
+NOTE: lookup_acl is not aware of address extensions and they are
+not handled specially!
If a list element contains a '@', the full e-mail address is compared,
otherwise if a list element has a leading dot, the domain name part is
diff --git a/README_FILES/README.milter b/README_FILES/README.milter
--- a/README_FILES/README.milter
+++ b/README_FILES/README.milter
@@ -185,8 +185,8 @@ is mandatory):
INPUT_MAIL_FILTER(`milter-amavis',
`S=local:/var/amavis/amavis-milter.sock, F=T, T=S:10m;R:10m;E:10m')
- define(`confMILTER_MACROS_ENVFROM',
- confMILTER_MACROS_ENVFROM``, {b}'')dnl # supply macro {b} to helper
+ define(`confMILTER_MACROS_ENVFROM',
+ confMILTER_MACROS_ENVFROM`, r, b') # supply macros b,r to helper
Now rebuild your sendmail.cf file and install it (usually
/etc/mail/sendmail.cf).
diff --git a/README_FILES/README.postfix b/README_FILES/README.postfix
--- a/README_FILES/README.postfix
+++ b/README_FILES/README.postfix
@@ -1,665 +1,1128 @@ This file README.postfix is part of the
-This file README.postfix is part of the amavisd-new distribution,
-which can be found at http://www.ijs.si/software/amavisd/
-
-Author: Mark Martinec <Mark.Martinec at ijs.si>
-Last updated: 2006-09-15
-
-
-How to use amavisd-new with Postfix
-***********************************
-
-Sections labeled 'COMMENT' may be skipped on first reading.
-
-Your Postfix must not be ancient, it must support parameter 'content_filter'.
-Check for the purpose of this parameter in ./README_FILES/FILTER_README
-of the Postfix distribution. This file was revised in postfix-1.1.9-20020512,
-and again in postfix 20030120, you may want to read the latest version.
-In the more recent Postfix documentation the setup described here is known
-as 'Postfix After-Queue Content Filter', section 'Advanced content filter'.
-
-For compatibility with previous versions of amavisd the choice of default
-tcp port numbers is 10024 and 10025, in contrast to 10025 and 10026 as used
-in FILTER_README examples. The service name chosen here is 'smtp-amavis'
-instead of 'scan' as in the Postfix documentation.
-
-We are assuming that Postfix is already installed, configured and is
-working as expected. As a safety net during experimenting one might feel
-better by setting 'soft_bounce=yes' in /etc/postfix/main.cf, and doing
-a 'postfix reload'. It will turn hard errors experienced by Postfix into
-temporary failures, causing failed mail operations to be retried later.
-Don't forget to remove it later when things appear to be running well.
-
-
-1. Install and start amavisd (as explained in INSTALL - just the daemon,
-no helper programs amavis(.c) or amavisd-milter(.c) are needed)
-
-For the first time it is best to start amavisd daemon interactively
-and keep it attached to the terminal:
-
- $ /usr/local/sbin/amavisd debug
-
-From another window check that it is listening on a
-local SMTP port 10024 (the default port):
-
---> $ telnet 127.0.0.1 10024
- Trying 127.0.0.1...
- Connected to 127.0.0.1.
- Escape character is '^]'.
-
- 220 [127.0.0.1] ESMTP amavisd-new service ready
-
---> quit
-
- 221 Bye
- Connection closed by foreign host.
-
-
-2. With a text editor add to the Postfix master.cf file
-the following two entries, e.g. near the end of the file:
-
-smtp-amavis unix - - y/n - 2 smtp
- -o smtp_data_done_timeout=1200
- -o smtp_send_xforward_command=yes
- -o disable_dns_lookups=yes
- -o max_use=20
-
-127.0.0.1:10025 inet n - y/n - - smtpd
- -o content_filter=
- -o smtpd_restriction_classes=
- -o smtpd_delay_reject=no
- -o smtpd_client_restrictions=permit_mynetworks,reject
- -o smtpd_helo_restrictions=
- -o smtpd_sender_restrictions=
- -o smtpd_recipient_restrictions=permit_mynetworks,reject
- -o smtpd_data_restrictions=reject_unauth_pipelining
- -o smtpd_end_of_data_restrictions=
- -o mynetworks=127.0.0.0/8
- -o smtpd_error_sleep_time=0
- -o smtpd_soft_error_limit=1001
- -o smtpd_hard_error_limit=1000
- -o smtpd_client_connection_count_limit=0
- -o smtpd_client_connection_rate_limit=0
- -o smtpd_milters=
- -o local_header_rewrite_clients=
- -o local_recipient_maps=
- -o relay_recipient_maps=
- -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks
-
-Change the 'y/n' to either 'y' or 'n', depending on how you prefer
-the smtp and smtpd postfix services to run - either chroot-ed, or not.
-See your other (normal) smtp and smtpd postfix services in master.cf
-and use the same setting here.
-
-COMMENTS:
-- Of all the options specified above in the second entry,
- the one that is essential is the '-o content_filter=' .
-- The '-o smtp_send_xforward_command=yes'
- (or '-o lmtp_send_xforward_command=yes' if using LMTP)
- is optional, but recommended - amavisd-new benefits from it since V2.0.
- It does not hurt if specified even if not yet supported by the currently
- running Postfix or amavisd-new.
-- the '-o max_use=20' is optional, it overrides the default value of 100,
- and is primarily useful with lmtp, as the Postfix lmtp client is more
- aggressive in keeping the connection open than the smtp client;
-- If there is an entry like 'vscan unix - n n - 2 pipe user=vscan ...'
- from an ancient amavisd installation, it is not needed any longer
- and may be removed. Keeping it does no harm.
-- for IPv6 enabled MTA, consider: -o mynetworks=127.0.0.0/8,[::1]/128
-
-
-3. Do a 'postfix reload', check its log file for any complaints,
- and test if it is listening on port 10025:
-
---> $ telnet 127.0.0.1 10025
- Trying 127.0.0.1...
- Connected to 127.0.0.1.
- Escape character is '^]'.
- 220 yourhost.example.com ESMTP Postfix
---> quit
- 221 Bye
- Connection closed by foreign host.
-
-
-4. If you want, simulate a mail sent to amavisd and see if it gets delivered
- via Postfix to its recipient. Try first with a simple and clean message,
- then a message with an EICAR test virus pattern which should be recognized
- by all virus scanners (unless all scanners are disabled or not installed):
-
---> $ telnet 127.0.0.1 10024
- Trying 127.0.0.1...
- Connected to 127.0.0.1.
- Escape character is '^]'.
- 220 [127.0.0.1] ESMTP amavisd-new service ready
---> MAIL FROM:<test at example.com>
- 250 2.1.0 Sender test at example.com OK
---> RCPT TO:<postmaster>
- 250 2.1.5 Recipient postmaster OK
---> DATA
- 354 End data with <CR><LF>.<CR><LF>
---> Subject: test1
--->
---> test1
---> .
-
-*** 250 2.6.0 Ok, id=31859-01, from MTA: 250 Ok: queued as 90B7F16F
-
---> MAIL FROM:<test at example.com>
- 250 2.1.0 Sender test at example.com OK
---> RCPT TO:<postmaster>
- 250 2.1.5 Recipient postmaster OK
---> DATA
- 354 End data with <CR><LF>.<CR><LF>
---> Subject: test2 - virus test pattern
--->
---> X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
---> .
-
-you should get one of the following replies (or similar), depending on
-the $final_virus_destiny and *virus_lovers* settings in amavisd.conf:
-*** 550 5.7.1 Message content rejected, id=16968-01 - VIRUS: EICAR-AV-Test
-*** 250 2.5.0 Ok, but 1 BOUNCE
-*** 250 2.7.1 Ok, discarded, id=16984-01 - VIRUS: EICAR-AV-Test
-*** 250 2.6.0 Ok, id=17041-01, from MTA: 250 Ok: queued as 3F1841A5F5
-
---> QUIT
- 221 2.0.0 [127.0.0.1] (amavisd) closing transmission channel
- Connection closed by foreign host.
-
-You may need/want to use different sender and recipient addresses.
-The test pattern must be entered exactly to be recognized, starting
-at the beginning of a line (without indentation).
-
-Depending on the settings in amavisd.conf, the sender (test at example.com)
-and the virus administrator may have been sent a (non-)delivery status
-notification, the second message should have been quarantined, and the first
-message must have been successfully delivered to the recipient. See the log
-that is scrolling on the terminal (as set up at step 1) and check for possible
-problems.
-
-
-5. Tell Postfix to start forwarding all mail it receives to amavisd-new
- for content inspection.
-
-To the Postfix main.cf file add a line:
-
- content_filter=smtp-amavis:[127.0.0.1]:10024
-
-either with a text editor, or preferably using a shell command:
- # postconf -e 'content_filter=smtp-amavis:[127.0.0.1]:10024'
-
-COMMENT:
- The global setting of 'content_filter' in main.cf affects any Postfix
- input service (i.e. smtpd and pickup). If a more selective approach
- is required, the option
- -o content_filter=smtp-amavis:[127.0.0.1]:10024
- may be given in master.cf to selected services only, or the option:
- -o content_filter=
- may override (i.e. clear) the global setting on selected services.
-
-
-6. Do a 'postfix reload' and watch the logs - both the Postfix logs,
-and the amavisd log file (on the screen or wherever you have it directed).
-
-If you get in trouble, you only need to undo the step 5 and do a
-'postfix reload'. New mail will no longer be tagged with content filter
-routing.
-
-COMMENT:
- The messages that have been received while 'content_filter' was set,
- will still try to get delivered to your old setting of content_filter,
- and will wait in the queue until successful or deleted or expired - or
- until you do: postsuper -r ALL; postfix reload
-
-If all is fine, you may abort (^C) the process running 'amavisd debug',
-and start amavisd without a 'debug' option, making it detach and daemonize.
-There is no need to stop or restart Postfix.
-
-This completes the integration of amavisd and Postfix.
-It uses the SMTP (or LMTP) protocol for Postfix->amavisd,
-and uses SMTP protocol for amavisd->Postfix communication.
-This is the fastest and recommended method, and simplest to set up.
-
-
-TUNING:
-
-The most important tuning knob is the number of concurrent content filtering
-processes allowed. Too low a value does not fully utilize the host resources,
-a somewhat high value wastes memory and gains no benefit to the aggregate
-mail throughput, while a too high value causes system thrashing and the
-total system mail throughput starts to drop. A useful starting value is 2,
-a commonly useful range is perhaps up to 10 (or perhaps 20 on hosts with
-1 GB of RAM or more, and SA with network tests such as Razor enabled),
-but the exact value largely depends on host capabilities and the anti-virus
-and anti-spam options in use.
-
-It is imperative that both the Postfix and the amavisd-new use the same value.
-Actually the amavisd setting may be higher that the Postfix, but this serves
-no useful purpose and just wastes resources. The amavisd.conf parameter is
-the $max_servers, the Postfix parameter is the maxproc field in the
-'smtp-amavis' entry (file master.cf).
-
-Instead of adjusting the maxproc field of the 'smtp-amavis' service,
-one may prefer to leave it a the default '-', and use a main.cf option
-for the same purpose:
- smtp-amavis_destination_concurrency_limit = 2
-
-For other tuning hints, see README.performance and:
- http://www.ijs.si/software/amavisd/amavisd-new-magdeburg-20050519.pdf
-
-
-TO DO 'VIRTUAL ALIAS' MAPPING AND OTHER POSTFIX CLEANUP PROCESSING
-BEFORE OR AFTER CONTENT FILTERING?
-
-In a post-queue content filtering setup (a normal amavisd-new setup with
-Postfix), a mail message passes through smtpd and cleanup Postfix services
-twice, once before the content filter, and the second time when approved
-message is passed from the content filter back to MTA. Any transformations
-and checks done by a cleanup service are thus performed twice. In simpler
-setups this does not matter much, but in more demanding situations one
-needs to consider which cleanup instance should perform which task.
-See cleanup(8) man page.
-
-In particular, the following should be considered:
-
-- masquerading
-
-- canonical address transformations
- placed before the content filter:
- content filter will see canonicalized envelope addresses
- (e.g. external addresses)
- placed after the content filter:
- content filter will see largely unmodified envelope addresses
-
-- virtual alias transformations of envelope recipient addresses
- placed before the content filter:
- content filter will see modified (e.g. internalized) envelope addresses
- placed after the content filter:
- content filter will see largely unmodified envelope addresses
-
-- built-in content checks like the header_checks, body_checks, mime processing
- placed before the content filter:
- the usual placement, checks should be performed as early as convenient
- placed after the content filter:
- most built-in content checks should not be performed again to save time
- and prevent late bounces. An exception may be the 'placing on hold'
- of a mail message that the content filter considered a potential threat
- and inserted a header field 'X-Amavis-Hold: reason', which needs to be
- done after content filtering.
-
-- automatic BCC recipient controls
- should only be done once to prevent mail duplication. The same
- applies when virtual mapping is used a "poor man's" mailing lists.
- Adding recipients is normally placed after content filtering;
-
-- resource and rate controls
- should be done before the content filtering, and should be disabled
- or be more liberal in the cleanup service after the content filter;
-
-To exercise full control over which cleanup service will perform which
-e-mail address mapping (virtual alias, canonical, masquerading), and
-which (if any) header/body checks, one needs to use two cleanup services:
-
-- add a new service 'pre-cleanup';
-- (optionally) add options to existing service 'cleanup';
-- add option 'cleanup_service_name=pre-cleanup' to existing services
- 'smtp' and 'pickup';
-
-as described further down.
-
-If the full flexibility of having two cleanup services is not needed
-and Postfix is snapshot 2.0.13-20030706 or later, there is a new parameter
-'receive_override_options' which eliminates the need for two cleanup
-services in some more straightforward cases (not all features of having
-two cleanup services are available). The idea is to use:
- -o receive_override_options=no_address_mappings
-for main incoming services (like smtpd and pickup), and the:
- -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks
-for the post-content-filter smtpd service on port 10025.
-See smtpd(8) man page and the FILTER_README and ADDRESS_REWRITING_README
-files in the Postfix documentation directory README_FILES.
-The receive_override_options=no_address_mappings also avoids the need
-for moving always_bcc option from main.cf to master.cf in common cases.
-
-
-ALTERNATIVE FOR POSTFIX OLDER THAN 2.2
-
-Postfix can be told to feed mail to amavisd via LMTP protocol instead
-of SMTP. This is possible since Postfix 2.0 and since amavisd-new-20021116.
-
-LMTP brings per-recipient status responses and multi-transaction session
-capability, the later of which the Postfix service smtp before cca. 2004-08
-lacked. For newer versions of Postfix with "connection cache" capability
-(previously known as "session caching"), i.e. the Postfix 2.2 and the
-snapshots since cca. 2004-08, the per-recipient status responses remains
-the only (small) advantage of LMTP.
-
-A LMTP advantage with its per-recipient status responses is most useful when
-the second MTA instance (on port 10025) returns a 5xx or 4xx SMTP response
-for some but not all recipients for some reason, or if amavisd-new is
-(inappropriately) configured to D_REJECT malware instead of D_BOUNCE it.
-Neither of the two is encountered regularly on well configured systems.
-
-As it is advisable to perform most of the MTA mail checks (like mail address
-validation, header and body checks) as soon as mail enters the mailer, the
-second MTA instance should under normal circumstances hardly ever generate
-a 5xx or 4xx response. Regarding the second argument, rejecting malware
-(D_REJECT) in an after-queue setup leads to backscatter generated by MTA
-and is not a recommended setting in a Postfix after-queue and other
-dual-MTA settings.
-
-To use LMTP instead of SMTP just replace the service name (last item)
-'smtp' with 'lmtp' in the master.cf entry: vvvv
-
-smtp-amavis unix - - y/n - 2 lmtp
- -o lmtp_data_done_timeout=1200
- -o lmtp_send_xforward_command=yes
-
-(and change option names accordingly).
-
-
-OPTIONAL:
-
-It is probably a good idea to set strict_rfc821_envelopes=yes in main.cf
-to reject non-replyable sender addresses such as <@yahoo.com> straight away,
-otherwise we end up processing such mail with inability to bounce it when
-needed, effectively losing such mail.
-
-
-One can set up an e-mail addresses (a mailbox) with Postfix to receive all
-quarantined viruses so that a mailer will deal with storing or forwarding
-them, and a local quarantine directory directly can be avoided. Here is
-one way of doing it, but see 'local(8)' Postfix man page for more options.
-
-This method of quarantining might be the only method available if amavisd
-is running chrooted and quarantine is to be located outside of chroot jail.
-
-To the Postfix aliases file (or database) add an entry for an e-mail address,
-e.g. 'infected', either to forward its mail to some place, or do a local
-delivery to a file or directory, e.g.:
-
-infected: /var/spool/mail/infected
-
-(append a '/' if you prefer Maildir style mailbox),
-then run 'newaliases' (or 'postalias /etc/postfix/aliases').
-
-In your amavisd.conf file specify (note the trailing '@') :
- $virus_quarantine_to = 'infected@'; # forward to MTA for delivery
-
-Reload amavisd (amavisd reload) if it is already running to make it
-re-read its config file, and check the log file. All set! Send some
-infected mail and watch it appear at the specified mailbox.
-
-
-----------------------------------------------------------------------------
-The next section is a commented setup that can be directly appended
-to the Postfix master.cf file instead of the item (2.) above,
-in case you need finer control or better understanding.
-It describes a Postfix setup with two cleanup services, as recommended
-by the new Postfix 2.0.3 README_FILES/FILTER_README .
-
-Here is an overall picture (adapted from FILTER_README to match
-port numbers and service name as traditionally used by amavisd-new):
-
- .......................................
- : Postfix :
- ----->smtpd \ :
- : -pre-cleanup-\ /local---->
- ---->pickup / -queue- :
- : -cleanup-/ | \smtp----->
- : bounces/ ^ v :
- : and locally | v :
- : forwarded smtpd smtp-amavis :
- : messages 10025 | :
- ...........................|...........
- ^ |
- | v
- ............|...............................
- : | $inet_socket_port=10024 :
- : | :
- : $forward_method='smtp:[127.0.0.1]:10025' :
- : $notify_method ='smtp:[127.0.0.1]:10025' :
- : :
- : amavisd-new :
- ............................................
-
-
-
-# Append this to the master.cf Postfix file and edit to will.
-# It defaults to the standard settings as described in README.postfix .
-#
-# Instruct Postfix content filtering to SEND all mail TO amavisd:
-# ===============================================================
-#
-# By default amavisd will listen to both protocols (SMTP/LMTP over TCP,
-# as well as to amavis helper protocol on a Unix socket). The older method
-# using the helper program amavis.c still works, but is not recommended, and
-# is not described here. To disable amavisd-new unnecessarily listening on a
-# Unix socket, comment out the assignment to $unix_socketname in amavisd.conf.
-#
-# NOTE1: match number of sending Postfix processes (the '2' below) with
-# $max_servers in amavisd.conf. Two to ten per CPU should be enough.
-# Going beyond 20 just wastes memory and does not help with throughput.
-# NOTE2: point the 'content_filter' hostname part to where amavisd is running,
-# and match the port number with $inet_socket_port in amavisd.conf, e.g:
-# content_filter = smtp-amavis:my-amavisd-server.example.com:10024
-# NOTE3: you should restrict amavisd to only accept connections from the
-# authorized Postfix host(s) by the @inet_acl access list in
-# the amavisd.conf file, and/or by binding to specific interface, e.g.:
-# $inet_socket_bind = '127.0.0.1'; # limit bind to loopback interface
-# NOTE4: set the chroot field the same (y/n) as for your regular smtp service;
-# NOTE5: set $child_timeout (in amavisd.conf) to a shorter time than
-# the Postfix parameter smtp_data_done_timeout - see rfc1047.
-# The value in '-o smtp_data_done_timeout=1200' must always be larger
-# (with some margin) than the value of $child_timeout in amavisd.conf
-# NOTE6: the option '-o disable_dns_lookups=yes' is recommended for reducing
-# latency with Postfix versions older than 2.0, but specify preferably
-# an IP address and not a DNS name in the content_filter specification
-# (thanks to Victor Duchovni for the suggestion).
-#
-# The "smtp-amavis" transport is a dedicated instance of the "smtp"
-# delivery agent for injecting messages into the SMTP/LMTP content filter.
-# Using a dedicated "smtp" or "lmtp" transport allows one to tune it
-# for the specific task of delivering mail to a local content filter
-# (low latency, low concurrency, throughput dependent on predictably
-# low latency).
-#
-smtp-amavis unix - - n - 2 smtp
- -o smtp_data_done_timeout=1200
- -o smtp_send_xforward_command=yes
-#some more ideas:
-# -o disable_dns_lookups=yes
-# -o max_use=20
-# -o smtp_bind_address=127.0.0.1
-# -o strict_rfc821_envelopes=yes
-# -o smtp_line_length_limit=0
-# -o notify_classes=protocol,resource,software
-# -o fallback_relay=backup-filter.example.com:10024
-
-# or equivalently when using lmtp:
-#smtp-amavis unix - - n - 2 lmtp
-# -o lmtp_data_done_timeout=1200
-# -o lmtp_send_xforward_command=yes
-#...
-
-# COMMENT:
-# To provide a backup content filter in case the primary fails,
-# either use the '-o fallback_relay' as above, or use a DNS name and
-# provide multiple MX records for it. See /etc/postfix/sample-smtp.cf
-# and smtp(8) man page for details.
-
-
-# from amavisd back to Postfix:
-# =============================
-#
-# variant 1A: via SMTP, same host (or see 1B for multihost setup)
-# In amavisd.conf choose the host running Postfix and its port number, e.g.:
-# $notify_method = 'smtp:[127.0.0.1]:10025';
-# $forward_method = 'smtp:[127.0.0.1]:10025';
-
-
-# The following is the SMTP listener that receives filtered messages from the
-# content filter. It *MUST* clear the content_filter parameter to avoid loops.
-#
-# This "smtpd" uses the normal cleanup service which is also used
-# for bounces and for internally forwarded mail.
-#
-# Disable all access control other than insisting on connections from one
-# of the IP addresses of the host. This can reduce resource usage if the
-# default restrictions do lots of checks.
-#
-# NOTE: set the chroot field the same (y/n) as for your regular smtpd service
-#
-127.0.0.1:10025 inet n - n - - smtpd
- -o content_filter=
- -o smtpd_restriction_classes=
- -o smtpd_delay_reject=no
- -o smtpd_client_restrictions=permit_mynetworks,reject
- -o smtpd_helo_restrictions=
- -o smtpd_sender_restrictions=
- -o smtpd_recipient_restrictions=permit_mynetworks,reject
- -o smtpd_data_restrictions=reject_unauth_pipelining
- -o smtpd_end_of_data_restrictions=
- -o mynetworks=127.0.0.0/8
- -o smtpd_error_sleep_time=0
- -o smtpd_soft_error_limit=1001
- -o smtpd_hard_error_limit=1000
- -o smtpd_client_connection_count_limit=0
- -o smtpd_client_connection_rate_limit=0
- -o smtpd_milters=
- -o local_header_rewrite_clients=
- -o local_recipient_maps=
- -o relay_recipient_maps=
- -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks
-
-
-# The following is the cleanup daemon that handles messages in front of
-# the content filter. It does header_checks and body_checks (if any), but
-# does no virtual alias or canonical address mapping, so that mail comes
-# to a content filter with original recipient addresses still intact.
-#
-# Virtual alias or canonical address mapping happens in the second
-# cleanup phase after the content filter. This gives the content_filter
-# access to largely unmodified addresses for maximum flexibility.
-#
-# Note that some sites may specifically want to perform canonical and/or
-# virtual address mapping in front of the content_filter. However, in that
-# case you still have to enable address rewriting in the after-filter cleanup
-# instance in order to correctly process forwarded mail or bounced mail.
-
-# handle both the canonicalization and virtual_alias_maps later
-# (this will provide content filter with largely unmodified addresses)
-#
-pre-cleanup unix n - n - 0 cleanup
- -o virtual_alias_maps=
- -o canonical_maps=
- -o sender_canonical_maps=
- -o recipient_canonical_maps=
- -o masquerade_domains=
-
-# ...or leave canonicalization in pre-cleanup, but do virtual_alias_maps later
-# (this is useful if canonicalization is used to map internal e-mail
-# addresses to external, and you want the content filter to always see
-# external canonical addresses of local users, both as senders,
-# and as recipients):
-#
-#pre-cleanup unix n - n - 0 cleanup
-# -o virtual_alias_maps=
-
-
-# The following is the normal cleanup daemon. No header or body checks here,
-# because these have already been taken care of by the pre-cleanup service
-# before the content filter. The normal cleanup instance does all
-# the virtual alias and canonical address mapping that was disabled
-# in the pre-cleanup instance before the content filter.
-#
-cleanup unix n - n - 0 cleanup
- -o mime_header_checks=
- -o nested_header_checks=
- -o body_checks=
- -o header_checks=
-# or use second-stage header checks, to be able to place mail bombs on HOLD
-# -o header_checks=pcre:/etc/postfix/header_checks2
-# consider also:
-# -o always_bcc=snooping at example.com
-
-# Place the following line (without the leading # and space) into file
-# /etc/postfix/header_checks2, and use the -o header_checks=pcre:... above,
-# if you need trouble mail (e.g. mail bombs) to be placed on hold by Postfix,
-# instead of being passed to recipients:
-#
-# /^X-Amavis-Hold:/ HOLD
-
-
-# These are the usual input "smtpd" and local "pickup" servers already
-# present in master.cf. We add an option to select a non-default
-# cleanup service.
-#
-smtp inet n - n - - smtpd
- -o cleanup_service_name=pre-cleanup
-pickup fifo n - n 60 1 pickup
- -o cleanup_service_name=pre-cleanup
-
-
-
-# variant 1B: via SMTP - with amavisd on a different host,
-# also good for the case of several MTAs using the same amavisd server.
-# - re-injection (forwarding) if $forward_method is smtp:...
-# - notification messages if $notify_method is smtp:...
-# - quarantine if $virus_quarantine_to contains '@'
-# In amavisd.conf set port number where Postfix (one or more) is listening
-# for re-injected mail and notifications, and optionally use an asterisk in
-# $forward_method and $notify_method specification if host or port field
-# is to be dynamically replaced (re-injection port number is automatically
-# set to one higher than the port number on which message came in to amavisd,
-# making possible for several MTA pairs on the same host to independently
-# use amavisd, e.g. separately for incoming and outgoing mail). To prevent
-# unauthorized use of the service you should restrict the set of IP addresses
-# from which amavisd is willing to accept mail by specifying authorized Postfix
-# host(s) with the access list @inet_acl in the amavisd.conf file. Bind must
-# not be restricted to loopback interface, so set $inet_socket_bind to undef.
-#
-# NOTE1: you SHOULD also restrict Postfix to only accept connections
-# on port 10025 from the amavisd host by '-o mynetworks = ...'
-# NOTE2: set the chroot field the same (y/n) as for your regular smtp service.
-
-#10025 inet n - n - - smtpd
-# -o content_filter=
-# -o smtpd_restriction_classes=
-# -o smtpd_delay_reject=no
-# -o smtpd_client_restrictions=permit_mynetworks,reject
-# -o smtpd_helo_restrictions=
-# -o smtpd_sender_restrictions=
-# -o smtpd_recipient_restrictions=permit_mynetworks,reject
-# -o smtpd_data_restrictions=reject_unauth_pipelining
-# -o smtpd_end_of_data_restrictions=
-# -o mynetworks=127.0.0.0/8,10.0.0.0/8,192.168.1.1
-# -o smtpd_error_sleep_time=0
-# -o smtpd_soft_error_limit=1001
-# -o smtpd_hard_error_limit=1000
-# -o smtpd_client_connection_count_limit=0
-# -o smtpd_client_connection_rate_limit=0
-# -o smtpd_milters=
-# -o local_header_rewrite_clients=
-# -o local_recipient_maps=
-# -o relay_recipient_maps=
-# -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks
-
-
-A tip from Wietse Venema (2002-12-12):
-
-| If you want to filter inbound SMTP mail only, then:
-|
-| /etc/postfix/main.cf:
-| smtpd_recipient_restrictions =
-| check_recipient_access hash:/etc/postfix/recipient_access
-| ...the usual stuff here...
-| reject_unauth_destination
-|
-| /etc/postfix/recipient_access:
-| my.domain FILTER foo:bar
-|
-| That filters all the mail that has at least one recipient in your
-| domain, and does not filter mail with external recipients only.
-
-(comment: the 'foo:bar' is what you would traditionally specify
-in content_filter option, i.e. smtp-amavis:[127.0.0.1]:10024 )
+Chapter 1. Integrating amavisd-new in Postfix
+
+ Patrick Ben Koetter
+
+ <[1]patrick.koetter at state-of-mind.de>
+
+ Mark Martinec
+
+ <[2]Mark.Martinec+amavis at ijs.si>
+
+ License: GNU GENERAL PUBLIC LICENSE, Version 2, June 1991
+
+ +------------------------------------------------------------------------+
+ | Revision History |
+ |------------------------------------------------------------------------|
+ | Revision 122 | 15. Jun 2007 | PK |
+ |------------------------------------------------------------------------|
+ | Added Section on Advanced Configuration |
+ |------------------------------------------------------------------------|
+ | Revision 108 | 22. Apr 2007 | PK |
+ |------------------------------------------------------------------------|
+ | Initial publication |
+ +------------------------------------------------------------------------+
+
+ Table of Contents
+
+ [3]1. Requirements
+
+ [4]1.1. Which Postfix version is required?
+
+ [5]1.2. Catching errors during integration
+
+ [6]2. Basic Postfix and amavisd-new configuration
+
+ [7]2.1. Configuring amavisd-new for Postfix
+
+ [8]2.2. Configuring the transport from Postfix to amavisd-new
+
+ [9]2.3. Configuring a dedicated SMTP-server for message
+ reinjection
+
+ [10]2.4. Testing basic configuration
+
+ [11]3. Message filtering examples
+
+ [12]3.1. Filtering E-mail globally
+
+ [13]3.2. Filtering E-mail by Postfix service
+
+ [14]3.3. Filtering E-Mails per Recipient Domain
+
+ [15]3.4. Filtering E-Mails by Sender-Domain
+
+ [16]3.5. Filtering E-mail per Content
+
+ [17]4. Advanced Postfix and amavisd-new configuration
+
+ [18]4.1. Multiple cleanup service architecture
+
+ [19]4.2. Configuring two cleanup services
+
+ [20]5. Tuning
+
+ [21]5.1. Maximum Number of Concurrent Processes
+
+ [22]5.2. Additional Tips for Tuning
+
+ Abstract
+
+ This document describes how amavisd-new can be integrated into the Postfix
+ SMTP delivery process. It lists the necessary requirements, explains how
+ Postfix and amavisd-new need to be configured to basically work together
+ and it gives filter-examples to show how amavisd-new can be called from
+ Postfix.
+
+1. Requirements
+
+ The following requirements must be met before integration can begin:
+
+ 1. amavisd-new has already been installed and successfully tested.
+
+ 2. Postfix has been installed, configured for basic operations and tested
+ successfully.
+
+ [23][Tip] Tip
+ Independently of the configuration examples shown in this
+ document, it is advisable to set strict_rfc821_envelopes = yes
+ in /etc/postfix/main.cf. Postfix will reject any message from
+ envelope-senders, whose address can't be used to send a reply
+ to.
+
+ This avoids accepting e-mails from erroneous envelope-senders
+ that can't be informed of problems, which finally would result
+ in deleting the message - even if Postfix claimed successful
+ delivery in the first.
+
+ 1.1. Which Postfix version is required?
+
+ Integrating amavisd-new into the Postfix delivery process requires that
+ Postfix is able to delegate messages to external content filters. The
+ minimum version that provides content filtering is Postfix
+ release-20010228.
+
+ 1.2. Catching errors during integration
+
+ Chances are that configuration errors during implementation cause Postfix
+ to bounce legitimate messages. Setting the soft_bounce parameter during
+ integration and reloading the Postfix configuration afterwards prevents
+ Postfix from bouncing legitimate mail during that time:
+
+ # postconf -e "soft_bounce = yes"
+ # postfix reload
+
+ As soon as soft_bounce has been activated Postfix will treat all delivery
+ errors as temporary errors - any client that wants to send messages to
+ Postfix will keep mail in the mailqueue and it will suspend delivery until
+ the soft_bounce parameter has been removed or set to no.
+
+ Once the integration of amavisd-new into the Postfix delivery process has
+ been completed successfully soft_bounce must be removed or Postfix will
+ not generate bounce messages for legitimate mail.
+
+2. Basic Postfix and amavisd-new configuration
+
+ There are several moments at which Postfix can hand over messages over to
+ amavisd-new (before it accepts a message from a client or after) and there
+ are different filter approaches (globally, per recipient (domain), per
+ network interface, etc.) that can trigger Postfix to transport a message
+ to amavisd-new.
+
+ The transport methods - transporting a message from Postfix to amavisd-new
+ and backwards - however always remain the same. They will be described in
+ this section first. The section that follows will deal with different
+ filter approaches.
+
+ [24][Tip] Integration procedure
+ The following examples have been structured to cause minimum
+ trouble on an online mail system. The order of steps ensures
+ that filtering will be enabled at the very last moment. Several
+ tests will have been conducted to verify the delivery chain
+ works before the filter is enabled. Once enabled the complete
+ system should work at once.
+
+ 2.1. Configuring amavisd-new for Postfix
+
+ Configuring amavisd-new to work with Postfix answers the following two
+ questions:
+
+ 1. Which port should the amavisd-new daemon listen to for incoming
+ connections from Postfix?
+
+ 2. Which IP-address and port should the amavisd-new SMTP client use to
+ (re)inject filtered messages (and notifications about message
+ statuses) into the Postfix SMTP delivery system?
+
+ 2.1.1. Configuring amavisd-new for incoming connections
+
+ The $inet_socket_port in /etc/amavisd.conf parameter sets the port number
+ amavisd-new will listen for incoming (E)SMTP connections. The following
+ example explicitly configures amavisd-new to bind to port 10024 (default
+ setting undef):
+
+ $inet_socket_port = 10024;
+
+ 2.1.2. Configuring the reinjection path
+
+ Two parameters, $forward_method and $notify_method, need to be configured
+ (usually identically) to reinject messages into the Postfix mail system.
+
+ The first parameter, $forward_method, specifies where amavisd-new should
+ transport scanned messages to, while the second parameter, $notify_method,
+ specifies where notifications about scanned messages should be transported
+ to.
+
+ By default amavisd uses 127.0.0.1 on port 10025 to contact a SMTP server
+ for reinjection of filtered messages. Unless a different IP address or
+ port should be used, no modifications must be applied and this section can
+ be skipped.
+
+ In case a different IP address or port should be used, the parameters
+ $notify_method and $forward_method need to be adjusted to reflect these
+ requirements. The following example edits these parameters in
+ /etc/amavisd.conf and uses 192.0.2.1 as IP address and port 20025:
+
+ $notify_method = 'smtp:[192.0.2.1]:20025';
+ $forward_method = 'smtp:[192.0.2.1]:20025';
+
+ 2.2. Configuring the transport from Postfix to amavisd-new
+
+ Both, amavisd-new and Postfix, are able to use either SMTP- or
+ LMTP-communication to transport a message from Postfix to amavisd-new.
+ Both variants will be described in this section.
+
+ Why configure a dedicated service?
+
+ Theoretically it's possible to transport messages from Postfix to
+ amavisd-new using the existing smtp-, lmtp, or even the relay-service in
+ /etc/postfix/master.cf.
+
+ In practice transporting messages to amavisd-new requires imposing
+ transport limits on the transporting service. Imposing such limits on a
+ globally available service would impose these limits on the complete
+ Postfix mail system - it would slow down the system significantly and
+ should be avoided.
+
+ [25][Note] Note
+ The number of Postfix clients that may connect simultaneously
+ to amavisd-new instances must be limited to the maximum number
+ of daemon child processes amavisd-new starts.
+
+ If the Postfix transport client was allowed to open more
+ connections amavisd-new can handle, amavisd-new would start to
+ queue incoming Postfix connections. Postfix in turn would
+ interpret such behaviour as "unresponsive remote MTA" and would
+ itself begin to queue mail that should be filtered. All this
+ would possibly throttle down the complete system and all
+ further filtering attempts would suffer.
+
+ 2.2.1. Configuring a dedicated lmtp-client
+
+ The following example creates a new, dedicated lmtp-transport named
+ amavisfeed in /etc/postfix/master.cf. Its configuration details are
+ explained following the listing:
+
+ # ==========================================================================
+ # service type private unpriv chroot wakeup maxproc command + args
+ # (yes) (yes) (yes) (never) (100)
+ # ==========================================================================
+
+ ...
+
+ amavisfeed unix - - n - 2 lmtp
+ -o lmtp_data_done_timeout=1200
+ -o lmtp_send_xforward_command=yes
+ -o disable_dns_lookups=yes
+ -o max_use=20
+
+ [26][Important] Important
+ A noteworthy quote from the Postfix documentation: "...do
+ not specify whitespace around the `='. In parameter
+ values, either avoid whitespace altogether, ...". Further
+ details on master.cf configuration syntax can be found in
+ master.cf or master(5).
+
+ Here's a quick rundown on the settings that differ from other services
+ defaults:
+
+ maxproc
+
+ The maximum number of concurrent Postfix amavis-service processes
+ has been limited to 2 (default: default_process_limit = 100). This
+ value reflects the default of 2 amavisd-daemon children processes
+ and is a good setting to start from. The value may be raised
+ later, when the system works stable and still can take a higher
+ load. It should not exceed the number of simultaneous amavisd
+ child processes.
+
+ lmtp_data_done_timeout
+
+ Setting lmtp_data_done_timeout to 1200 (seconds) doubles the
+ default time span a regular Postfix client waits after message
+ delivery for the server to reply DONE to claim successful
+ delivery. It must be larger than amavisd setting $child_timeout
+ (default 8*60 seconds) and should add a sufficient safety margin,
+ for example to cater for periods of automatic database maintenance
+ (e.g. bayes database on non-SQL database types) which can take a
+ long time in some cases.
+
+ If the server does not reply within the configured time span, the
+ Postfix client will quit the connection, put the message into the
+ deferred queue, log a delivery failure and retry later to
+ transport the message to amavisd-new.
+
+ [27][Note] Note
+ Raising this value serves a trick amavisd uses to avoid
+ message loss in case of power outage etc. The trick
+ consists in keeping the incoming connection as long
+ open as it takes to filter the message and take
+ appropriate action (reinjection, notification,
+ quarantine, etc.).
+
+ Only when the message (or notifications etc.) has been
+ reinjected amavisd will send DONE to the client and the
+ client will close the connection. This way Postfix will
+ always keep the message in its own mail queue, where it
+ can be reactivated after a system failure.
+
+ lmtp_send_xforward_command
+
+ Enabling lmtp_send_xforward_command configures the Postfix
+ lmtp-client to forward the original clients HELO name and IP
+ address to amavisd-new. amavisd-new in turn can use these
+ informations for
+
+ o logging and notifications (macro %a)
+
+ o switching policy banks (MYNETS, @mynetworks_maps)
+
+ o pen pals functionality
+
+ o p0f fingerprinting
+
+ disable_dns_lookups
+
+ The transport route from Postfix to amavisd-new, it will be
+ configured later in [28]Section 3, "Message filtering examples",
+ will probably never change. It will - probably - only change when
+ the whole mail system is being reconfigured. The target host may
+ therefore be specified as IP address instead of using a DNS
+ hostname. This saves "expensive" DNS-request (3 lookups) and
+ improves performance.
+
+ max_use
+
+ By default Postfix reuses a service instance 100 times (max_use =
+ 100), before the instance terminates. The master daemon will
+ reinvoke such a service if required. There's no need for the
+ amavisfeed-service to have such a long life-span. Best practice
+ has it to set max_use to 20.
+
+ 2.2.2. Configuring a dedicated smtp-client
+
+ Configuring a dedicated smtp-client is almost identical to configuring a
+ dedicated lmtp-client. The syntax differences in detail are that the names
+ of parameters start with smtp_ instead of lmtp_ and that the command at
+ the end of the service invokes the smtp- and not lmtp-client. The same
+ reasons given for differing lmtp client options apply to the dedicated
+ smtp client configuration.
+
+ Here's an example of a dedicated smtp client given the service name
+ amavisfeed:
+
+ # ==========================================================================
+ # service type private unpriv chroot wakeup maxproc command + args
+ # (yes) (yes) (yes) (never) (100)
+ # ==========================================================================
+
+ ...
+
+ amavisfeed unix - - n - 2 smtp
+ -o smtp_data_done_timeout=1200
+ -o smtp_send_xforward_command=yes
+ -o disable_dns_lookups=yes
+ -o max_use=20
+
+ 2.3. Configuring a dedicated SMTP-server for message reinjection
+
+ The second service that needs to be added to the Postfix mail system is a
+ dedicated SMTP-server. It will exist only to accept filtered messages and
+ notifications from amavisd-new to transported them closer to their final
+ destination.
+
+ This dedicated smtpd server will differ in many aspects from the default
+ smtpd daemon. The most important difference is that it configures an empty
+ content_filter parameter, thus overriding any global external content
+ filtering settings in Postfix.
+
+ [29][Note] Note
+ Delegating messages to an external content filter in Postfix is
+ done using the content_filter parameter. If the dedicated
+ smtpd-daemon would not override any global content_filter
+ settings, the reinjected message would be sent of to the
+ external content filter again - the mail would end in an
+ endless loop.
+
+ The following Postfix example uses amavisd-new default settings taken from
+ the $forward_method and $notify_method parameters. These settings
+ configure amavisd-new to forward filtered messages and notifications to
+ 127.0.0.1 on port 10025; the Postfix smtpd daemon will be configured to
+ bind to that IP address and listen on the specified port for incoming
+ connections:
+
+ # ==========================================================================
+ # service type private unpriv chroot wakeup maxproc command + args
+ # (yes) (yes) (yes) (never) (100)
+ # ==========================================================================
+
+ ...
+
+ 127.0.0.1:10025 inet n - n - - smtpd
+ -o content_filter=
+ -o smtpd_delay_reject=no
+ -o smtpd_client_restrictions=permit_mynetworks,reject
+ -o smtpd_helo_restrictions=
+ -o smtpd_sender_restrictions=
+ -o smtpd_recipient_restrictions=permit_mynetworks,reject
+ -o smtpd_data_restrictions=reject_unauth_pipelining
+ -o smtpd_end_of_data_restrictions=
+ -o smtpd_restriction_classes=
+ -o mynetworks=127.0.0.0/8
+ -o smtpd_error_sleep_time=0
+ -o smtpd_soft_error_limit=1001
+ -o smtpd_hard_error_limit=1000
+ -o smtpd_client_connection_count_limit=0
+ -o smtpd_client_connection_rate_limit=0
+ -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
+ -o local_header_rewrite_clients=
+
+ Here's a quick rundown on the settings that differ from smtpd defaults:
+
+ content_filter
+
+ The empty content_filter overrides other, globally set
+ content_filter delegations.
+
+ ..._maps
+
+ Empty ..._maps override any other globally set map lookups.
+ Procedures to enforce settings specified in such maps have already
+ taken place when Postfix accepted the message from the external
+ client. Doing them again will not produce new results but only
+ waste resources.
+
+ ..._restrictions_...
+
+ There's no need to apply any already enforced ..._restrictions_...
+ another time. It would also only waste resources.
+
+ mynetworks
+
+ To avoid abuse from remote hosts, the dedicated smtpd-daemon will
+ only allow clients from 127.0.0.0/8 to relay messages.
+
+ local_header_rewrite_clients
+
+ By default this option would "rewrite message header addresses in
+ mail from these clients and update incomplete addresses with the
+ domain name". If such action has already been taken by Postfix
+ before the message went off to amavis, it should not be done a
+ second time when it reenters the Postfix mail system. Leaving this
+ option empty disables local header rewrites and saves resources.
+
+ remaining options
+
+ All remaining options either configure the dedicated smtpd-daemon
+ to be more failure tolerant or exist to avoid unnecessary use of
+ resources.
+
+ Running the postfix reload will activate the new transports (Postfix will
+ not yet send regular mail to amavisd). Combined with the tail command
+ problems can easily be detected:
+
+ # postfix reload && tail -f /var/log/maillog
+
+ If there are no problems reported, basic configuration can be tested.
+
+ 2.4. Testing basic configuration
+
+ Testing basic configuration consists of three separate tests, starting at
+ the end of the new delivery chain and working to it's beginning. Their
+ goal is to answer the following questions:
+
+ 1. Will amavisd-new accept connections at the specified IP address and
+ port?
+
+ 2. Will the new dedicated smtpd-daemon accept connections at the
+ specified IP address and port?
+
+ 3. Will a test message, injected into amavisd-new, be filtered, sent to
+ Postfix and delivered into a mailbox?
+
+ 2.4.1. Testing amavisd's host and port
+
+ A test, using the telnet command, serves to verify that amavisd listens on
+ the specified IP address and port. A successful connection looks like
+ this:
+
+ $ telnet localhost 10024
+ 220 [127.0.0.1] ESMTP amavisd-new service ready
+ EHLO localhost
+ 250-[127.0.0.1]
+ 250-VRFY
+ 250-PIPELINING
+ 250-SIZE
+ 250-ENHANCEDSTATUSCODES
+ 250-8BITMIME
+ 250-DSN
+ 250 XFORWARD NAME ADDR PROTO HELO
+ QUIT
+ 221 2.0.0 [127.0.0.1] amavisd-new closing transmission channel
+
+ If the test fails, the following questions may help to debug the problem:
+
+ o Is the amavisd-new daemon running?
+
+ o Does amavisd-new write an error to the log?
+
+ o Do the IP address and port number specified in the amavisd-new
+ configuration match the values used during the test?
+
+ o Does a firewall intercept connections?
+
+ 2.4.2. Testing the dedicated Postfix smtpd-daemon
+
+ When Postfix was reloaded, the new, dedicated smtpd-daemon
+ (127.0.0.1:10025) should have been activated. A successful connection
+ looks like this:
+
+ $ telnet 127.0.0.1 10025
+ 220 mail.example.com ESMTP Postfix (2.3.2)
+ EHLO localhost
+ 250-mail.example.com
+ 250-PIPELINING
+ 250-SIZE 40960000
+ 250-ETRN
+ 250-STARTTLS
+ 250-AUTH PLAIN CRAM-MD5 LOGIN DIGEST-MD5
+ 250-AUTH=PLAIN CRAM-MD5 LOGIN DIGEST-MD5
+ 250-ENHANCEDSTATUSCODES
+ 250-8BITMIME
+ 250 DSN
+ QUIT
+ 221 2.0.0 Bye
+
+ If the test fails, the following questions may help to debug the problem:
+
+ o Is the Postfix master daemon running?
+
+ o Does Postfix write an error to the log?
+
+ o Do the IP address and port number specified in the new services
+ configuration match the values used during the test?
+
+ o Does a firewall intercept connections?
+
+ 2.4.3. Testing the new transport chain
+
+ This test proves amavisd accepts e-mail as specified in [30]Section 2.1,
+ "Configuring amavisd-new for Postfix", filters it and finally hands it
+ over to Postfix' dedicated smtpd-daemon as specified in [31]Section 2.3,
+ "Configuring a dedicated SMTP-server for message reinjection".
+
+ The following example uses the content of test-messages/sample-nonspam.txt
+ from the amavisd test-messages to send an e-mail:
+
+ $ telnet localhost 10024
+ 220 [127.0.0.1] ESMTP amavisd-new service ready
+ HELO localhost
+ 250 [127.0.0.1]
+ MAIL FROM: <>
+ 250 2.1.0 Sender OK
+ RCPT TO: <postmaster>
+ 250 2.1.5 Recipient postmaster OK
+ DATA
+ 354 End data with <CR><LF>.<CR><LF>
+ From: virus-tester
+ To: undisclosed-recipients:;
+ Subject: amavisd test - simple - no spam test pattern
+
+ This is a simple test message from the amavisd-new test-messages.
+ .
+ 250 2.6.0 Ok, id=30897-02, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 079474CE44
+ QUIT
+ 221 2.0.0 [127.0.0.1] amavisd-new closing transmission channel
+
+ The maillog shows the delivery path. Here's an excerpt from a successful
+ delivery process:
+
+ Nov 1 11:28:10 mail postfix/smtpd[30986]: connect from localhost[127.0.0.1] [32]1
+ Nov 1 11:28:10 mail postfix/smtpd[30986]: 079474CE44: client=localhost[127.0.0.1]
+ Nov 1 11:28:10 mail postfix/cleanup[30980]: 079474CE44: message-id=<20061101102810.079474CE44 at mail.example.com>
+ Nov 1 11:28:10 mail postfix/qmgr[20432]: 079474CE44: from=<>, size=822, nrcpt=1 (queue active)
+ Nov 1 11:28:10 mail amavis[30897]: (30897-02) Passed BAD-HEADER, <> -> <postmaster>, quarantine: badh-le5gjszxowBk, mail_id: le5gjszxowBk, Hits: -1.76, queued_as: 079474CE44, 39505 ms [33]2
+ Nov 1 11:28:10 mail postfix/smtpd[30986]: disconnect from localhost[127.0.0.1]
+ Nov 1 11:28:10 mail postfix/local[30987]: 079474CE44: to=<postmaster at example.com>, relay=local, delay=0.27, delays=0.14/0.05/0/0.08, dsn=2.0.0, status=sent (delivered to mailbox: postmaster) [34]3
+ Nov 1 11:28:10 mail postfix/qmgr[20432]: 079474CE44: removed
+
+ [35]1 amavisd connects with Postfix dedicated smtpd-daemon and hands over
+ the e-mail that had been sent during the telnet session. smtpd gives
+ a queue-id of 079474CE44 that can be tracked throughout the maillog.
+ [36]2 amavisd notices it has checked and sent an e-mail to <postmaster>.
+ [37]3 Postfix' local-service logs it successfully delivered an e-mail with
+ queue-id 079474CE44 to the mailbox of postmaster.
+
+ If the test fails, the following questions may help to debug the problem:
+
+ o Does amavisd-new log errors?
+
+ o Does running amavisd-new in debug-mode report errors?
+
+3. Message filtering examples
+
+ Postfix can use various criteria to decide whether a message should be
+ sent to amavisd-new for examination. Combinations of criteria may serve to
+ create different configurations. The following section describes the
+ following configurations:
+
+ o Filtering e-mail globally
+
+ o Filtering e-mail globally by service
+
+ o Filtering e-mail per recipient domain
+
+ o Filtering e-mail per sender domain
+
+ o Filtering e-mail by content
+
+ 3.1. Filtering E-mail globally
+
+ In most cases email policies require global filtering - every inbound and
+ every outbound e-mail must be filtered by amavisd-new - before it may be
+ sent closer to its final destination.
+
+ [38][Note] Why check outgoing mail traffic?
+ Some reasons for checking mail coming from internal networks or
+ from authenticated roaming users are:
+
+ o detect an internal infected PC which is sending viruses
+
+ o detect an internal zombiized PC (or an internal open relay
+ or proxy) which is sending or relaying spam
+
+ o let the SpamAssassin Bayes autolearning feature see a
+ balanced view of all mail, including useful samples of
+ non-spam originating from inside
+
+ o make it possible for pen pals feature to function (if
+ enabled)
+
+ In Postfix global settings for its services are written to main.cf. The
+ content_filter parameter, the parameter configuring that messages are sent
+ to amavisd-new, must therefore be placed in main.cf.
+
+ The content_filter parameter requires a triplet, consisting of the
+ transport service's name (here: amavisfeed, given in [39]Section 2.2.1,
+ "Configuring a dedicated lmtp-client"), the target hosts IP address and
+ the port where amavisd-new listens for incoming connections. Following the
+ values used in this documents examples the content_filter configuration
+ results in this:
+
+ content_filter=amavisfeed:[127.0.0.1]:10024
+
+ The new external content filter will be activated once Postfix has been
+ reloaded. Sending a test-mail verifies the system works.
+
+ 3.2. Filtering E-mail by Postfix service
+
+ Postfix is able to filter messages per service. Such configuration
+ requires the content_filter not to be applied globally to all services in
+ main.cf (see: [40]Section 3.1, "Filtering E-mail globally"), but
+ selectively, per service in master.cf.
+
+ The following example presumes Postfix runs on a system offering three IP
+ addresses. In this example these are: 192.0.2.1 (WAN), 127.0.0.1
+ (localhost) and 10.0.0.254 (LAN). The goal is to filter only e-mail that
+ enters from the WAN interface.
+
+ This requires to create three dedicated smtpd-daemon instances, each
+ binding to one of the given IP addresses and deactivating the global smtp
+ service calling the smtpd command.
+
+ Additionally the WAN interface (here: 192.0.2.1:25) is configured to use
+ content_filter =amavisfeed:[127.0.0.1]:10024 - it will delegate any
+ message that enters the Postfix mail system at this service to the
+ external amavisd content filter.
+
+ # ==========================================================================
+ # service type private unpriv chroot wakeup maxproc command + args
+ # (yes) (yes) (yes) (never) (100)
+ # ==========================================================================
+ # smtp inet n - n - - smtpd
+
+ ...
+
+ 192.0.2.1:25 inet n - n - - smtpd
+ -o content_filter=amavisfeed:[127.0.0.1]:10024
+ -o receive_override_options=no_address_mappings
+
+ 10.0.0.254:25 inet n - n - - smtpd
+
+ 127.0.0.1:10025 inet n - n - - smtpd
+ -o content_filter=
+ -o smtpd_delay_reject=no
+ -o smtpd_client_restrictions=permit_mynetworks,reject
+ -o smtpd_helo_restrictions=
+ -o smtpd_sender_restrictions=
+ -o smtpd_recipient_restrictions=permit_mynetworks,reject
+ -o smtpd_data_restrictions=reject_unauth_pipelining
+ -o smtpd_end_of_data_restrictions=
+ -o smtpd_restriction_classes=
+ -o mynetworks=127.0.0.0/8
+ -o smtpd_error_sleep_time=0
+ -o smtpd_soft_error_limit=1001
+ -o smtpd_hard_error_limit=1000
+ -o smtpd_client_connection_count_limit=0
+ -o smtpd_client_connection_rate_limit=0
+ -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
+ -o local_header_rewrite_clients=
+
+ 3.3. Filtering E-Mails per Recipient Domain
+
+ Postfix is able to filter e-mails per recipient domain. In order to do
+ this the content_filter parameter must not be set globally (see:
+ [41]Section 3.1, "Filtering E-mail globally"). Instead the content_filter
+ parameter has to be associated with one or more recipient domains listed
+ in a lookup table (map).
+
+ [42][Caution] Caution
+ This filter method is not selective! It will send any mail
+ with a recipient domain listed in the lookup table to amavis
+ even if the mail contains another recipient that should not
+ be examined by the amavis framework.
+
+ If fully selective rules are required all mail should be
+ sent to amavis and amavis' own rule sets should be
+ configured to decide whether a message for a given recipient
+ should be examined or not.
+
+ When Postfix searches the lookup table and finds the recipients domain
+ listed as key, it will take the action associated with that domain. The
+ action will send the message to a FILTER amavisfeed:[127.0.0.1]:10024.
+
+ The following map /etc/postfix/filter_recipient_domains specifies to send
+ messages to the FILTER amavisfeed whenever a message for any recipient at
+ example.com enters the Postfix mailqueues:
+
+ example.com FILTER amavisfeed:[127.0.0.1]:10024
+
+ Once the table has been created the postmap command must be used to create
+ an indexed map Postfix can read:
+
+ # postmap /etc/postfix/filter_recipient_domains
+
+ Once the map has been indexed, the postmap command is used to test the
+ map. In the following example the postmap command queries for the domain
+ example.com and returns the associated action:
+
+ # postmap -q "example.com" /etc/postfix/filter_recipient_domains
+ FILTER amavisfeed:[127.0.0.1]:10024
+
+ The tested map must be added to main.cf, before Postfix can make use of
+ the new filter policy. Setting the check_recipient_access parameter in the
+ list of smtpd_recipient_restrictions triggers evaluation of entries in the
+ map - check_recipient_access is triggered by the envelope-recipient(s)
+ given by a SMTP-client in a SMTP-session with Postfix.
+
+ The following example puts the check_recipient_access rule before
+ permit_mynetworks - all clients envelope-recipient(s) will be filtered:
+
+ smtpd_recipient_restrictions =
+ ...
+ check_recipient_access hash:/etc/postfix/filter_recipient_domains
+ ...
+ permit_mynetworks
+ reject_unauth_destination
+ ...
+
+ Filtering E-Mails per Recipient Domain only from External Clients
+
+ This example puts the check_recipient_access rule after permit_mynetworks
+ - only messages sent from clients that are not in Postfix $mynetworks list
+ (external or untrusted clients) will be filtered:
+
+ smtpd_recipient_restrictions =
+ ...
+ permit_mynetworks
+ reject_unauth_destination
+ check_recipient_access hash:/etc/postfix/filter_recipient_domains
+ ...
+
+ 3.4. Filtering E-Mails by Sender-Domain
+
+ In general it doesn't make sense to filter e-mails by sender-domain, as
+ anyone can fake a sender-domain during a SMTP-session. Filtering by
+ sender-domain will probably only make sense, if messages are not filtered
+ globally, but e-mails from ones own domain should be checked for spam or
+ viruses before they leave the network.
+
+ Most of the configuration steps are identical with the ones noted in
+ [43]Section 3.3, "Filtering E-Mails per Recipient Domain", except for the
+ parameter that triggers evaluation of the indexed map. In this scenario
+ envelope-senders should trigger map evaluation. The map, named
+ /etc/postfix/filter_sender_domains this time, contains the sender domain
+ (example.com) and associates it with the required FILTER:
+
+ example.com FILTER amavisfeed:[127.0.0.1]:10024
+
+ Once the map has been converted and tested with the postmap command (see:
+ [44]Section 3.3, "Filtering E-Mails per Recipient Domain") it must be
+ added to the list of smtpd_recipient_restrictions using the
+ check_sender_access parameter:
+
+ smtpd_recipient_restrictions =
+ ...
+ check_sender_access hash:/etc/postfix/filter_sender_domains
+ ...
+ permit_mynetworks
+ reject_unauth_destination
+ ...
+
+ [45][Important] Important
+ The map must be listed before permit_mynetworks, because
+ only then it will be applied to all clients - even the
+ ones Postfix trusts, which are very likely the ones from
+ example.com.
+
+ 3.5. Filtering E-mail per Content
+
+ Postfix is able - with deliberate limitations (see: BUILTIN_FILTER_README)
+ - to search for strings in headers, the body and MIME-headers. If a string
+ matches, Postfix may call appropriate action.
+
+ The following example configures Postfix to look for the string offer in
+ Subject:-headers and delegate the message to an external content filter if
+ if finds a matching string.
+
+ A map, consisting of the search string noted as regexp-expression,
+ associates the search pattern with a FILTER action:
+
+ /^Subject:.*offer/ FILTER amavisfeed:[127.0.0.1]:10024
+
+ [46][Note] Indexing regexp- or pcre-maps?
+ regexp- or pcre-maps are and must be plaintext files. They must
+ not and cannot be converted to an indexed map using the postmap
+ command. They can be tested using the postmap command using the
+ -q command line option.
+
+ Once the map has been created, Postfix must be configured to use it. The
+ following example uses the header_checks parameter (not body_checks or
+ mime_header_checks as they apply to other message parts) to implement the
+ map into the Postfix delivery process:
+
+ header_checks = regexp:/etc/postfix/filter_header
+
+ Once Postfix has been reloaded it will send every e-mail that contains the
+ word offer in the Subject:-header off to the external amavisd content
+ filter.
+
+4. Advanced Postfix and amavisd-new configuration
+
+ In a post-queue content filtering setup, a mail message passes through
+ smtpd and cleanup Postfix services twice, once before a content filter,
+ and the second time when an approved message is reinjected from a content
+ filter into the Postfix mail system. This is because checks and
+ transformations that have been configured in main.cf are globally active
+ and will be loaded and run by any instance of these two services. To avoid
+ wasting resources, options that control runtime behavior of these services
+ should not be applied globally in main.cf, but selectively to separate
+ instances of these services in master.cf.
+
+ Checks and transformations which are performed by a smtpd Postfix service
+ itself, e.g. access controls, recipient validation, milters etc., can be
+ controlled by adding options (-o) to appropriate smtpd services. This has
+ been shown in the basic configuration examples (see: [47]Section 2.3,
+ "Configuring a dedicated SMTP-server for message reinjection").
+
+ Checks and transformations which are performed by a cleanup Postfix
+ service are trickier because in a normal Postfix setup there is only one
+ cleanup service, unlike smtpd services of which there are many. Some of
+ the more important cleanup settings are dynamically controllable by a
+ smtpd service through the use of its receive_override_options option.
+
+ [48][Tip] Transformations and checks
+ Any transformation should preferably only be performed once,
+ either before or after content filtering. When to transform
+ depends on the desired effect, for example whether a content
+ filter should see unchanged or modified mail messages. Typical
+ transformations are:
+
+ o rewrite addresses
+
+ o add BCC recipients
+
+ o modify mail header.
+
+ Most checks should also be performed only once, preferably only
+ on the first passage, when the mail enters the Postfix mail
+ system the first time. This way messages can be rejected early -
+ if needed - and will not tie up downstream resources. Checking
+ early also avoids bounces in case of negative check results on a
+ second passage after content filtering.
+
+ 4.1. Multiple cleanup service architecture
+
+ To gain more control over a cleanup service than offered by
+ receive_override_options, two (or more) cleanup services, each with its
+ own set of options, must be run. A Postfix setup with more than one
+ cleanup service is possible either with two separate Postfix instances, or
+ through a specification of services and their options in master.cf file of
+ a single Postfix instance.
+
+ The following diagram illustrates a setup with two cleanup services in a
+ single Postfix instance:
+
+ .......................................
+ : Postfix :
+ ----->smtpd \ :
+ : -pre-cleanup-\ /local---->
+ ---->pickup / -queue- :
+ : -cleanup-/ | \smtp----->
+ : bounces/ ^ v :
+ : and locally | v :
+ : forwarded smtpd amavisfeed :
+ : messages 10025 | :
+ ...........................|...........
+ ^ |
+ | v
+ ............|...............................
+ : | $inet_socket_port=10024 :
+ : | :
+ : $forward_method='smtp:[127.0.0.1]:10025' :
+ : $notify_method ='smtp:[127.0.0.1]:10025' :
+ : :
+ : amavisd-new :
+ ............................................
+
+ Procedure 1.1. Message flow with two cleanup services
+
+ 1. Messages enter the Postfix system at the regular smtpd or pickup
+ service.
+
+ 2. The pre-cleanup cleanup service performs transformations and checks on
+ these messages.
+
+ 3. The qmgr service schedules the messages to be sent to the amavisd-new
+ content filter.
+
+ 4. amavisd-new performs various tests on the messages.
+
+ 5. Messages are re-injected into the Postfix mail system, sending them to
+ a dedicated, local smtpd service.
+
+ 6. The cleanup cleanup service performs transformations and checks that
+ must be done at this stage, but omits the ones that have already been
+ carried out in step 2.
+
+ 4.2. Configuring two cleanup services
+
+ Configuring Postfix smtpd services to use two separate, dedicated cleanup
+ services requires the following steps:
+
+ 1. Create a second cleanup instance
+
+ 2. Modify the existing cleanup service
+
+ 3. Configure smtpd services to use either of the two cleanup services.
+
+ 4.2.1. Creating a second cleanup instance
+
+ The following example adds a cleanup daemon named pre-cleanup. It will
+ handle messages before a content filter.
+
+ # ==========================================================================
+ # service type private unpriv chroot wakeup maxproc command + args
+ # (yes) (yes) (yes) (never) (100)
+ # ==========================================================================
+ # smtp inet n - n - - smtpd
+
+ ...
+
+ pre-cleanup unix n - n - 0 cleanup
+ -o virtual_alias_maps=
+
+ The above leaves canonicalization address rewriting enabled so that a
+ content filter will see canonicalized (external) sender mail addresses,
+ but it disables globally configured virtual alias transformations.
+
+ Such transformations will be done later by the second cleanup service, so
+ that a content filter will see original (external) recipient mail
+ addresses. Other options may also be used as needed.
+
+ 4.2.2. Modifying the existing cleanup service
+
+ The already existing cleanup service - having the service name cleanup -
+ will be used to process messages that re-enter the Postfix mail system
+ (also for delivery notifications and forwarding as generated internally by
+ Postfix).
+
+ Cleanup jobs that already have been performed by the pre-cleanup service
+ should not be run again. The following example disables typical checks
+ that have been run before or are not needed for internally generated
+ notifications:
+
+ # ==========================================================================
+ # service type private unpriv chroot wakeup maxproc command + args
+ # (yes) (yes) (yes) (never) (100)
+ # ==========================================================================
+ # smtp inet n - n - - smtpd
+
+ ...
+
+ cleanup unix n - n - 0 cleanup
+ -o mime_header_checks= [49]1
+ -o nested_header_checks= [50]2
+ -o body_checks= [51]3
+ -o header_checks= [52]4
+
+ [53]1 The specified options disable header and body checks as these would
+ [54]2 already be performed by a pre-cleanup service.
+ [55]3
+ [56]4
+
+ [57][Note] always_bcc
+ This cleanup service would also be the appropriate one for
+ specifying always_bcc option - doing it globally would apply to
+ both cleanup services and would result in two copies of each
+ message to be sent to the specified address.
+
+ 4.2.3. Configuring smtpd services
+
+ Finally existing smtpd services on ports 25 and 587 (submission), and the
+ pickup service must be configured to send messages to the new pre-cleanup
+ service instead of a default cleanup service:
+
+ # ==========================================================================
+ # service type private unpriv chroot wakeup maxproc command + args
+ # (yes) (yes) (yes) (never) (100)
+ # ==========================================================================
+ # smtp inet n - n - - smtpd
+
+ ...
+
+ pickup fifo n - n 60 1 pickup
+ -o cleanup_service_name=pre-cleanup
+ smtp inet n - n - - smtpd
+ -o cleanup_service_name=pre-cleanup
+ submission inet n - n - - smtpd
+ -o cleanup_service_name=pre-cleanup
+
+5. Tuning
+
+ 5.1. Maximum Number of Concurrent Processes
+
+ The most important settings to tune and optimize in Postfix and amavisd
+ workflow are the maximum number of concurrent processes. The maximum
+ number of concurrent processes on both sides must be chosen with care.
+
+ If the number is too low, hardware resources aren't used efficiently and
+ delivery time will be unnecessarily prolonged. Experience tells that
+ raising the number of processes a little, will not raise the overall
+ throughput in the same proportion.
+
+ As the system resources are nearing saturation with each increase of the
+ number of processes, an increase in throughput becomes marginal, and
+ eventually even negative when the number of processes exceeds its
+ near-optimum value. E-mail throughput will decrease, because processes
+ need to wait for each other. At worst e-mail delivery stalls.
+
+ Best practice is to start with a (conservative) maximum number of 2
+ concurrent processes. Everyday use has shown that this value may be raised
+ to a value between 10 and 30 concurrent Postfix client and amavisd server
+ processes. This also depends on the overall resources the system may
+ provide, how amavisd has been integrated into the Postfix delivery process
+ and on the anti-virus and anti-spam software being loaded and used by
+ amavisd-new.
+
+ Regardless of the maximum number of concurrent processes, both sides -
+ Postfix and amavisd - should be synchronized. To synchronize both sides
+ edit, the $max_servers parameter for amavisd-new (see: amavisd.conf) and
+ the number of processes in master.cf listed in the dedicated transports
+ maxproc column for Postfix.
+
+ Both values should be identical for two reasons: If amavisd-new offers
+ more processes than Postfix will ever use, amavisd-new wastes resources.
+ On the other hand, if Postfix starts more dedicated transports than
+ amavisd can handle simultaneously, e-mail transport will be refused and
+ logged as error.
+
+ [58][Note] Controlling the maximum number of concurrent processes in
+ main.cf
+ Instead of controlling the maximum number of concurrent
+ processes of Postfix' dedicated transport in master.cf it is
+ also possible to keep the default setting - in master.cf and
+ set the following parameter and option in main.cf:
+
+ amavisfeed_destination_concurrency_limit = 2
+
+ The name of the parameter starts with the service in master.cf
+ (here: amavisfeed) that should be controlled and goes on with
+ the suffix _destination_concurrency_limit. Here also 2 is set
+ as initial (conservative) value.
+
+ 5.2. Additional Tips for Tuning
+
+ Further Tuning-Tips can be found in README.performance and the slides from
+ [59]amavisd-new, advanced configuration and management.
+
+References
+
+ Visible links
+ 1. mailto:patrick.koetter at state-of-mind.de
+ 2. mailto:Mark.Martinec+amavis at ijs.si
+ 3. #requirements
+ 4. #requirements_postfix_version
+ 5. #requirements_catching_errors
+ 6. #basics
+ 7. #basics_amavisd-new
+ 8. #basics_transport
+ 9. #basics_smtpd-daemon
+ 10. #basics_testing
+ 11. #filter
+ 12. #filter_global
+ 13. #filter_service_global
+ 14. #filter_by_recipient
+ 15. #filter_by_sender
+ 16. #filter_by_content
+ 17. #d0e968
+ 18. #d0e1038
+ 19. #d0e1110
+ 20. #tuning
+ 21. #d0e1231
+ 22. #d0e1288
+ 28. 3. Message filtering examples
+ #filter
+ 30. 2.1. Configuring amavisd-new for Postfix
+ #basics_amavisd-new
+ 31. 2.3. Configuring a dedicated SMTP-server for message reinjection
+ #basics_smtpd-daemon
+ 35. #mailflow_1
+ 36. #mailflow_2
+ 37. #mailflow_3
+ 39. 2.2.1. Configuring a dedicated lmtp-client
+ #basics_transport_lmtp-client
+ 40. 3.1. Filtering E-mail globally
+ #filter_global
+ 41. 3.1. Filtering E-mail globally
+ #filter_global
+ 43. 3.3. Filtering E-Mails per Recipient Domain
+ #filter_by_recipient
+ 44. 3.3. Filtering E-Mails per Recipient Domain
+ #filter_by_recipient
+ 47. 2.3. Configuring a dedicated SMTP-server for message reinjection
+ #basics_smtpd-daemon
+ 53. #cleanup-mime_header_checks
+ 54. #cleanup-nested_header_checks
+ 55. #cleanup-body_checks
+ 56. #cleanup-header_checks
+ 59. http://www.ijs.si/software/amavisd/amavisd-new-magdeburg-20050519.pdf
diff --git a/README_FILES/README.postfix.html b/README_FILES/README.postfix.html
new file mode 100644
--- /dev/null
+++ b/README_FILES/README.postfix.html
@@ -0,0 +1,635 @@
+<html><head>
+ <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+ <title>Chapter 1. Integrating amavisd-new in Postfix</title><link rel="stylesheet" href="screen.css" type="text/css"><meta name="generator" content="DocBook XSL Stylesheets V1.69.1"><meta name="keywords" content="amavisd, amavisd-new, amavisd-new Postfix, configuring amavisd-new Postfix, amavisd-new Postfix configuration, amavisd-new Postfix HOWTO"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="README.postfix"></a>Chapter 1. Integrating amavisd-new in Postfix</h2></div><div><div class="authorgroup"><div class="author"><h3 class="author"><span class="firstname">Patrick</span> <span class="othername">Ben</span> <span class="surname">Koetter</span></h3><code class="email"><<a href="mailto:patrick.koetter at state-of-mind.de">patrick.koetter at state-of-mind.de</a>></code></div><div class="author"><h3 class="author"><span class="firstname">Mark</span> <span class="surname">Martinec</span></h3><code class="email"><<a href="mailto:Mark.Martinec+amavis at ijs.si">Mark.Martinec+amavis at ijs.si</a>></code></div></div></div><div><div class="legalnotice"><a name="d0e42"></a><p>License: GNU GENERAL PUBLIC LICENSE, Version 2, June 1991</p></div></div><div><div class="revhistory"><table border="1" width="100%" summary="Revision history"><tr><th align="left" valign="top" colspan="3"><b>Revision History</b></th></tr><tr><td align="left">Revision 122</td><td align="left">15. Jun 2007</td><td align="left">PK</td></tr><tr><td align="left" colspan="3">Added Section on Advanced Configuration</td></tr><tr><td align="left">Revision 108</td><td align="left">22. Apr 2007</td><td align="left">PK</td></tr><tr><td align="left" colspan="3">Initial publication</td></tr></table></div></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#requirements">1. Requirements</a></span></dt><dd><dl><dt><span class="section"><a href="#requirements_postfix_version">1.1. Which Postfix version is required?</a></span></dt><dt><span class="section"><a href="#requirements_catching_errors">1.2. Catching errors during integration</a></span></dt></dl></dd><dt><span class="section"><a href="#basics">2. Basic Postfix and amavisd-new configuration</a></span></dt><dd><dl><dt><span class="section"><a href="#basics_amavisd-new">2.1. Configuring amavisd-new for Postfix</a></span></dt><dt><span class="section"><a href="#basics_transport">2.2. Configuring the transport from Postfix to amavisd-new</a></span></dt><dt><span class="section"><a href="#basics_smtpd-daemon">2.3. Configuring a dedicated SMTP-server for message
+ reinjection</a></span></dt><dt><span class="section"><a href="#basics_testing">2.4. Testing basic configuration</a></span></dt></dl></dd><dt><span class="section"><a href="#filter">3. Message filtering examples</a></span></dt><dd><dl><dt><span class="section"><a href="#filter_global">3.1. Filtering E-mail globally</a></span></dt><dt><span class="section"><a href="#filter_service_global">3.2. Filtering E-mail by Postfix service</a></span></dt><dt><span class="section"><a href="#filter_by_recipient">3.3. Filtering E-Mails per Recipient Domain</a></span></dt><dt><span class="section"><a href="#filter_by_sender">3.4. Filtering E-Mails by Sender-Domain</a></span></dt><dt><span class="section"><a href="#filter_by_content">3.5. Filtering E-mail per Content</a></span></dt></dl></dd><dt><span class="section"><a href="#d0e968">4. Advanced Postfix and amavisd-new configuration</a></span></dt><dd><dl><dt><span class="section"><a href="#d0e1038">4.1. Multiple cleanup service architecture</a></span></dt><dt><span class="section"><a href="#d0e1110">4.2. Configuring two cleanup services</a></span></dt></dl></dd><dt><span class="section"><a href="#tuning">5. Tuning</a></span></dt><dd><dl><dt><span class="section"><a href="#d0e1231">5.1. Maximum Number of Concurrent Processes</a></span></dt><dt><span class="section"><a href="#d0e1288">5.2. Additional Tips for Tuning</a></span></dt></dl></dd></dl></div><div class="abstract"><p class="title"><b>Abstract</b></p><p>This document describes how amavisd-new can be integrated into the
+ Postfix SMTP delivery process. It lists the necessary requirements,
+ explains how Postfix and amavisd-new need to be configured to basically
+ work together and it gives filter-examples to show how amavisd-new can be
+ called from Postfix.</p></div><div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="requirements"></a>1. Requirements</h2></div></div></div><p>The following requirements must be met before integration can
+ begin:</p><div class="orderedlist"><ol type="1"><li><p>amavisd-new has already been installed and successfully
+ tested.</p></li><li><p>Postfix has been installed, configured for basic operations and
+ tested successfully.</p></li></ol></div><div class="tip" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Tip"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Tip]" src="images/tip.png"></td><th align="left">Tip</th></tr><tr><td align="left" valign="top"><p>Independently of the configuration examples shown in this
+ document, it is advisable to set
+ <em class="parameter"><code>strict_rfc821_envelopes</code></em> = <code class="option">yes</code> in
+ <code class="filename">/etc/postfix/main.cf</code>. Postfix will reject any
+ message from envelope-senders, whose address can't be used to send a
+ reply to.</p><p>This avoids accepting e-mails from erroneous envelope-senders that
+ can't be informed of problems, which finally would result in deleting
+ the message - even if Postfix claimed successful delivery in the
+ first.</p></td></tr></table></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="requirements_postfix_version"></a>1.1. Which Postfix version is required?</h3></div></div></div><p>Integrating amavisd-new into the Postfix delivery process requires
+ that Postfix is able to delegate messages to external content filters.
+ The minimum version that provides content filtering is Postfix
+ release-20010228.</p></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="requirements_catching_errors"></a>1.2. Catching errors during integration</h3></div></div></div><p>Chances are that configuration errors during implementation cause
+ Postfix to bounce legitimate messages. Setting the
+ <em class="parameter"><code>soft_bounce</code></em> parameter during integration and
+ reloading the Postfix configuration afterwards prevents Postfix from
+ bouncing legitimate mail during that time:</p><pre class="programlisting"># <strong class="userinput"><code>postconf -e "soft_bounce = yes"</code></strong>
+# <strong class="userinput"><code>postfix reload</code></strong></pre><p>As soon as <em class="parameter"><code>soft_bounce</code></em> has been activated
+ Postfix will treat all delivery errors as temporary errors - any client
+ that wants to send messages to Postfix will keep mail in the mailqueue
+ and it will suspend delivery until the
+ <em class="parameter"><code>soft_bounce</code></em> parameter has been removed or set to
+ <code class="option">no</code>.</p><p>Once the integration of amavisd-new into the Postfix delivery
+ process has been completed successfully
+ <em class="parameter"><code>soft_bounce</code></em> must be removed or Postfix will not
+ generate bounce messages for legitimate mail.</p></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="basics"></a>2. Basic Postfix and amavisd-new configuration</h2></div></div></div><p>There are several moments at which Postfix can hand over messages
+ over to amavisd-new (before it accepts a message from a client or after)
+ and there are different filter approaches (globally, per recipient
+ (domain), per network interface, etc.) that can trigger Postfix to
+ transport a message to amavisd-new.</p><p>The transport methods - transporting a message from Postfix to
+ amavisd-new and backwards - however always remain the same. They will be
+ described in this section first. The section that follows will deal with
+ different filter approaches.</p><div class="tip" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Tip: Integration procedure"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Tip]" src="images/tip.png"></td><th align="left">Integration procedure</th></tr><tr><td align="left" valign="top"><p>The following examples have been structured to cause minimum
+ trouble on an online mail system. The order of steps ensures that
+ filtering will be enabled at the very last moment. Several tests will
+ have been conducted to verify the delivery chain works before the filter
+ is enabled. Once enabled the complete system should work at once.</p></td></tr></table></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="basics_amavisd-new"></a>2.1. Configuring amavisd-new for Postfix</h3></div></div></div><p>Configuring amavisd-new to work with Postfix answers the following
+ two questions:</p><div class="orderedlist"><ol type="1"><li><p>Which port should the amavisd-new daemon listen to for
+ incoming connections from Postfix?</p></li><li><p>Which IP-address and port should the amavisd-new SMTP client
+ use to (re)inject filtered messages (and notifications about message
+ statuses) into the Postfix SMTP delivery system?</p></li></ol></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="d0e157"></a>2.1.1. Configuring amavisd-new for incoming connections</h4></div></div></div><p>The <em class="parameter"><code>$inet_socket_port</code></em> in
+ <code class="filename">/etc/amavisd.conf</code> parameter sets the port number
+ amavisd-new will listen for incoming (E)SMTP connections. The
+ following example explicitly configures amavisd-new to bind to port
+ <code class="systemitem">10024</code> (default setting
+ <code class="option">undef</code>):</p><pre class="programlisting">$inet_socket_port = 10024;</pre></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="d0e176"></a>2.1.2. Configuring the reinjection path</h4></div></div></div><p>Two parameters, <em class="parameter"><code>$forward_method</code></em> and
+ <em class="parameter"><code>$notify_method</code></em>, need to be configured (usually
+ identically) to reinject messages into the Postfix mail system.</p><p>The first parameter, <em class="parameter"><code>$forward_method</code></em>,
+ specifies where amavisd-new should transport scanned messages to,
+ while the second parameter, <em class="parameter"><code>$notify_method</code></em>,
+ specifies where notifications about scanned messages should be
+ transported to.</p><p>By default amavisd uses <code class="systemitem">127.0.0.1</code> on port <code class="systemitem">10025</code> to contact a SMTP server for
+ reinjection of filtered messages. Unless a different IP address or
+ port should be used, no modifications must be applied and this section
+ can be skipped.</p><p>In case a different IP address or port should be used, the
+ parameters <em class="parameter"><code>$notify_method</code></em> and
+ <em class="parameter"><code>$forward_method</code></em> need to be adjusted to reflect
+ these requirements. The following example edits these parameters in
+ <code class="filename">/etc/amavisd.conf</code> and uses <code class="systemitem">192.0.2.1</code> as IP address and port
+ <code class="systemitem">20025</code>:</p><pre class="programlisting">$notify_method = 'smtp:[192.0.2.1]:20025';
+$forward_method = 'smtp:[192.0.2.1]:20025';</pre></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="basics_transport"></a>2.2. Configuring the transport from Postfix to amavisd-new</h3></div></div></div><p>Both, amavisd-new and Postfix, are able to use either SMTP- or
+ LMTP-communication to transport a message from Postfix to amavisd-new.
+ Both variants will be described in this section.</p><h4><a name="d0e227"></a>Why configure a dedicated service?</h4><p>Theoretically it's possible to transport messages from Postfix to
+ amavisd-new using the existing smtp-, lmtp, or even the relay-service in
+ <code class="filename">/etc/postfix/master.cf</code>.</p><p>In practice transporting messages to amavisd-new requires imposing
+ transport limits on the transporting service. Imposing such limits on a
+ globally available service would impose these limits on the complete
+ Postfix mail system - it would slow down the system significantly and
+ should be avoided.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>The number of Postfix clients that may connect simultaneously to
+ amavisd-new instances must be limited to the maximum number of daemon
+ child processes amavisd-new starts.</p><p>If the Postfix transport client was allowed to open more
+ connections amavisd-new can handle, amavisd-new would start to queue
+ incoming Postfix connections. Postfix in turn would interpret such
+ behaviour as “<span class="quote">unresponsive remote MTA</span>” and would itself
+ begin to queue mail that should be filtered. All this would possibly
+ throttle down the complete system and all further filtering attempts
+ would suffer.</p></td></tr></table></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="basics_transport_lmtp-client"></a>2.2.1. Configuring a dedicated lmtp-client</h4></div></div></div><p>The following example creates a new, dedicated lmtp-transport
+ named <code class="systemitem">amavisfeed</code> in
+ <code class="filename">/etc/postfix/master.cf</code>. Its configuration details
+ are explained following the listing:</p><pre class="programlisting"># ==========================================================================
+# service type private unpriv chroot wakeup maxproc command + args
+# (yes) (yes) (yes) (never) (100)
+# ==========================================================================
+
+...
+
+amavisfeed unix - - n - 2 lmtp
+ -o lmtp_data_done_timeout=1200
+ -o lmtp_send_xforward_command=yes
+ -o disable_dns_lookups=yes
+ -o max_use=20</pre><div class="important" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Important"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Important]" src="images/important.png"></td><th align="left">Important</th></tr><tr><td align="left" valign="top"><p>A noteworthy quote from the Postfix documentation:
+ “<span class="quote">...do not specify whitespace around the ‘<span class="quote">=</span>’. In
+ parameter values, either avoid whitespace altogether, ...</span>”.
+ Further details on <code class="filename">master.cf</code> configuration
+ syntax can be found in <code class="filename">master.cf</code> or
+ <span class="citerefentry"><span class="refentrytitle">master</span>(5)</span>.</p></td></tr></table></div><p>Here's a quick rundown on the settings that differ from other
+ services defaults:</p><div class="variablelist"><dl><dt><span class="term"><em class="parameter"><code>maxproc</code></em></span></dt><dd><p>The maximum number of concurrent Postfix amavis-service
+ processes has been limited to <code class="option">2</code> (default:
+ <em class="parameter"><code>default_process_limit</code></em> =
+ <code class="option">100</code>). This value reflects the default of
+ <code class="option">2</code> amavisd-daemon children processes and is a
+ good setting to start from. The value may be raised later, when
+ the system works stable and still can take a higher load. It
+ should not exceed the number of simultaneous amavisd child
+ processes.</p></dd><dt><span class="term"><em class="parameter"><code>lmtp_data_done_timeout</code></em></span></dt><dd><p>Setting <em class="parameter"><code>lmtp_data_done_timeout</code></em> to
+ <code class="option">1200</code> (seconds) doubles the default time span a
+ regular Postfix client waits after message delivery for the
+ server to reply <code class="computeroutput">DONE</code> to claim
+ successful delivery. It must be larger than amavisd setting
+ <em class="parameter"><code>$child_timeout</code></em> (default
+ <code class="option">8</code>*<code class="option">60</code> seconds) and should add a
+ sufficient safety margin, for example to cater for periods of
+ automatic database maintenance (e.g. bayes database on non-SQL
+ database types) which can take a long time in some cases.</p><p>If the server does not reply within the configured time
+ span, the Postfix client will quit the connection, put the
+ message into the deferred queue, log a delivery failure and
+ retry later to transport the message to amavisd-new.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>Raising this value serves a trick amavisd uses to avoid
+ message loss in case of power outage etc. The trick consists
+ in keeping the incoming connection as long open as it takes to
+ filter the message and take appropriate action (reinjection,
+ notification, quarantine, etc.).</p><p>Only when the message (or notifications etc.) has been
+ reinjected amavisd will send
+ <code class="computeroutput">DONE</code> to the client and the
+ client will close the connection. This way Postfix will always
+ keep the message in its own mail queue, where it can be
+ reactivated after a system failure.</p></td></tr></table></div></dd><dt><span class="term"><em class="parameter"><code>lmtp_send_xforward_command</code></em></span></dt><dd><p>Enabling <em class="parameter"><code>lmtp_send_xforward_command</code></em>
+ configures the Postfix lmtp-client to forward the original
+ clients HELO name and IP address to amavisd-new. amavisd-new in
+ turn can use these informations for</p><div class="itemizedlist"><ul type="disc"><li><p>logging and notifications (macro
+ <code class="varname">%a</code>)</p></li><li><p>switching policy banks (<code class="constant">MYNETS</code>,
+ <em class="parameter"><code>@mynetworks_maps</code></em>)</p></li><li><p>pen pals functionality</p></li><li><p>p0f fingerprinting</p></li></ul></div></dd><dt><span class="term"><em class="parameter"><code>disable_dns_lookups</code></em></span></dt><dd><p>The transport route from Postfix to amavisd-new, it will
+ be configured later in <a href="#filter" title="3. Message filtering examples">Section 3, “Message filtering examples”</a>, will probably
+ never change. It will - probably - only change when the whole
+ mail system is being reconfigured. The target host may therefore
+ be specified as IP address instead of using a DNS hostname. This
+ saves “<span class="quote">expensive</span>” DNS-request (3 lookups) and
+ improves performance.</p></dd><dt><span class="term"><em class="parameter"><code>max_use</code></em></span></dt><dd><p>By default Postfix reuses a service instance 100 times
+ (<em class="parameter"><code>max_use</code></em> = <code class="option">100</code>), before
+ the instance terminates. The master daemon will reinvoke such a
+ service if required. There's no need for the amavisfeed-service
+ to have such a long life-span. Best practice has it to set
+ <em class="parameter"><code>max_use</code></em> to <code class="option">20</code>.</p></dd></dl></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="basics_transport_smtp-client"></a>2.2.2. Configuring a dedicated smtp-client</h4></div></div></div><p>Configuring a dedicated smtp-client is almost identical to
+ configuring a dedicated lmtp-client. The syntax differences in detail
+ are that the names of parameters start with
+ <em class="parameter"><code>smtp_</code></em> instead of <em class="parameter"><code>lmtp_</code></em>
+ and that the command at the end of the service invokes the smtp- and
+ not lmtp-client. The same reasons given for differing lmtp client
+ options apply to the dedicated smtp client configuration.</p><p>Here's an example of a dedicated smtp client given the service
+ name <em class="parameter"><code>amavisfeed</code></em>:</p><pre class="programlisting"># ==========================================================================
+# service type private unpriv chroot wakeup maxproc command + args
+# (yes) (yes) (yes) (never) (100)
+# ==========================================================================
+
+...
+
+amavisfeed unix - - n - 2 smtp
+ -o smtp_data_done_timeout=1200
+ -o smtp_send_xforward_command=yes
+ -o disable_dns_lookups=yes
+ -o max_use=20</pre></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="basics_smtpd-daemon"></a>2.3. Configuring a dedicated SMTP-server for message
+ reinjection</h3></div></div></div><p>The second service that needs to be added to the Postfix mail
+ system is a dedicated SMTP-server. It will exist only to accept filtered
+ messages and notifications from amavisd-new to transported them closer
+ to their final destination.</p><p>This dedicated smtpd server will differ in many aspects from the
+ default smtpd daemon. The most important difference is that it
+ configures an empty <em class="parameter"><code>content_filter</code></em> parameter,
+ thus overriding any global external content filtering settings in
+ Postfix.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>Delegating messages to an external content filter in Postfix is
+ done using the <em class="parameter"><code>content_filter</code></em> parameter. If the
+ dedicated smtpd-daemon would not override any global
+ <em class="parameter"><code>content_filter</code></em> settings, the reinjected message
+ would be sent of to the external content filter again - the mail would
+ end in an endless loop.</p></td></tr></table></div><p>The following Postfix example uses amavisd-new default settings
+ taken from the <em class="parameter"><code>$forward_method</code></em> and
+ <em class="parameter"><code>$notify_method</code></em> parameters. These settings
+ configure amavisd-new to forward filtered messages and notifications to
+ <code class="systemitem">127.0.0.1</code> on port <code class="systemitem">10025</code>; the Postfix smtpd daemon will be
+ configured to bind to that IP address and listen on the specified port
+ for incoming connections:</p><pre class="programlisting"># ==========================================================================
+# service type private unpriv chroot wakeup maxproc command + args
+# (yes) (yes) (yes) (never) (100)
+# ==========================================================================
+
+...
+
+127.0.0.1:10025 inet n - n - - smtpd
+ -o content_filter=
+ -o smtpd_delay_reject=no
+ -o smtpd_client_restrictions=permit_mynetworks,reject
+ -o smtpd_helo_restrictions=
+ -o smtpd_sender_restrictions=
+ -o smtpd_recipient_restrictions=permit_mynetworks,reject
+ -o smtpd_data_restrictions=reject_unauth_pipelining
+ -o smtpd_end_of_data_restrictions=
+ -o smtpd_restriction_classes=
+ -o mynetworks=127.0.0.0/8
+ -o smtpd_error_sleep_time=0
+ -o smtpd_soft_error_limit=1001
+ -o smtpd_hard_error_limit=1000
+ -o smtpd_client_connection_count_limit=0
+ -o smtpd_client_connection_rate_limit=0
+ -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
+ -o local_header_rewrite_clients=</pre><p>Here's a quick rundown on the settings that differ from smtpd
+ defaults:</p><div class="variablelist"><dl><dt><span class="term"><em class="parameter"><code>content_filter</code></em></span></dt><dd><p>The empty <em class="parameter"><code>content_filter</code></em> overrides
+ other, globally set <em class="parameter"><code>content_filter</code></em>
+ delegations.</p></dd><dt><span class="term"><em class="parameter"><code>..._maps</code></em></span></dt><dd><p>Empty <em class="parameter"><code>..._maps</code></em> override any other
+ globally set map lookups. Procedures to enforce settings specified
+ in such maps have already taken place when Postfix accepted the
+ message from the external client. Doing them again will not
+ produce new results but only waste resources.</p></dd><dt><span class="term"><em class="parameter"><code>..._restrictions_...</code></em></span></dt><dd><p>There's no need to apply any already enforced
+ <em class="parameter"><code>..._restrictions_...</code></em> another time. It would
+ also only waste resources.</p></dd><dt><span class="term"><em class="parameter"><code>mynetworks</code></em></span></dt><dd><p>To avoid abuse from remote hosts, the dedicated smtpd-daemon
+ will only allow clients from <code class="systemitem">127.0.0.0/8</code> to relay
+ messages.</p></dd><dt><span class="term"><em class="parameter"><code>local_header_rewrite_clients</code></em></span></dt><dd><p>By default this option would “<span class="quote">rewrite message header
+ addresses in mail from these clients and update incomplete
+ addresses with the domain name</span>”. If such action has already
+ been taken by Postfix before the message went off to amavis, it
+ should not be done a second time when it reenters the Postfix mail
+ system. Leaving this option empty disables local header rewrites
+ and saves resources.</p></dd><dt><span class="term">remaining options</span></dt><dd><p>All remaining options either configure the dedicated
+ smtpd-daemon to be more failure tolerant or exist to avoid
+ unnecessary use of resources.</p></dd></dl></div><p>Running the postfix reload will activate the new transports
+ (Postfix will not yet send regular mail to amavisd). Combined with the
+ tail command problems can easily be detected:</p><pre class="screen"># <strong class="userinput"><code>postfix reload && tail -f /var/log/maillog</code></strong></pre><p>If there are no problems reported, basic configuration can be
+ tested.</p></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="basics_testing"></a>2.4. Testing basic configuration</h3></div></div></div><p>Testing basic configuration consists of three separate tests,
+ starting at the end of the new delivery chain and working to it's
+ beginning. Their goal is to answer the following questions:</p><div class="orderedlist"><ol type="1"><li><p>Will amavisd-new accept connections at the specified IP
+ address and port?</p></li><li><p>Will the new dedicated smtpd-daemon accept connections at the
+ specified IP address and port?</p></li><li><p>Will a test message, injected into amavisd-new, be filtered,
+ sent to Postfix and delivered into a mailbox?</p></li></ol></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="d0e536"></a>2.4.1. Testing amavisd's host and port</h4></div></div></div><p>A test, using the telnet command, serves to verify that amavisd
+ listens on the specified IP address and port. A successful connection
+ looks like this:</p><pre class="screen">$ <strong class="userinput"><code>telnet localhost 10024</code></strong>
+220 [127.0.0.1] ESMTP amavisd-new service ready
+<strong class="userinput"><code>EHLO localhost</code></strong>
+250-[127.0.0.1]
+250-VRFY
+250-PIPELINING
+250-SIZE
+250-ENHANCEDSTATUSCODES
+250-8BITMIME
+250-DSN
+250 XFORWARD NAME ADDR PROTO HELO
+<strong class="userinput"><code>QUIT</code></strong>
+221 2.0.0 [127.0.0.1] amavisd-new closing transmission channel</pre><p>If the test fails, the following questions may help to debug the
+ problem:</p><div class="itemizedlist"><ul type="disc"><li><p>Is the amavisd-new daemon running?</p></li><li><p>Does amavisd-new write an error to the log?</p></li><li><p>Do the IP address and port number specified in the
+ amavisd-new configuration match the values used during the
+ test?</p></li><li><p>Does a firewall intercept connections?</p></li></ul></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="d0e567"></a>2.4.2. Testing the dedicated Postfix smtpd-daemon</h4></div></div></div><p>When Postfix was reloaded, the new, dedicated smtpd-daemon
+ (127.0.0.1:10025) should have been activated. A successful connection
+ looks like this:</p><pre class="screen">$ <strong class="userinput"><code>telnet 127.0.0.1 10025</code></strong>
+220 mail.example.com ESMTP Postfix (2.3.2)
+<strong class="userinput"><code>EHLO localhost</code></strong>
+250-mail.example.com
+250-PIPELINING
+250-SIZE 40960000
+250-ETRN
+250-STARTTLS
+250-AUTH PLAIN CRAM-MD5 LOGIN DIGEST-MD5
+250-AUTH=PLAIN CRAM-MD5 LOGIN DIGEST-MD5
+250-ENHANCEDSTATUSCODES
+250-8BITMIME
+250 DSN
+<strong class="userinput"><code>QUIT</code></strong>
+221 2.0.0 Bye</pre><p>If the test fails, the following questions may help to debug the
+ problem:</p><div class="itemizedlist"><ul type="disc"><li><p>Is the Postfix master daemon running?</p></li><li><p>Does Postfix write an error to the log?</p></li><li><p>Do the IP address and port number specified in the new
+ services configuration match the values used during the
+ test?</p></li><li><p>Does a firewall intercept connections?</p></li></ul></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="d0e598"></a>2.4.3. Testing the new transport chain</h4></div></div></div><p>This test proves amavisd accepts e-mail as specified in <a href="#basics_amavisd-new" title="2.1. Configuring amavisd-new for Postfix">Section 2.1, “Configuring amavisd-new for Postfix”</a>, filters it and finally hands it over
+ to Postfix' dedicated smtpd-daemon as specified in <a href="#basics_smtpd-daemon" title="2.3. Configuring a dedicated SMTP-server for message
 reinjection">Section 2.3, “Configuring a dedicated SMTP-server for message
+ reinjection”</a>.</p><p>The following example uses the content of
+ <code class="filename">test-messages/sample-nonspam.txt</code> from the amavisd
+ test-messages to send an e-mail:</p><pre class="screen">$ <strong class="userinput"><code>telnet localhost 10024</code></strong>
+220 [127.0.0.1] ESMTP amavisd-new service ready
+<strong class="userinput"><code>HELO localhost</code></strong>
+250 [127.0.0.1]
+<strong class="userinput"><code>MAIL FROM: <></code></strong>
+250 2.1.0 Sender OK
+<strong class="userinput"><code>RCPT TO: <postmaster></code></strong>
+250 2.1.5 Recipient postmaster OK
+<strong class="userinput"><code>DATA</code></strong>
+354 End data with <CR><LF>.<CR><LF>
+<strong class="userinput"><code>From: virus-tester
+To: undisclosed-recipients:;
+Subject: amavisd test - simple - no spam test pattern
+
+This is a simple test message from the amavisd-new test-messages.
+.</code></strong>
+250 2.6.0 Ok, id=30897-02, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 079474CE44
+<strong class="userinput"><code>QUIT</code></strong>
+221 2.0.0 [127.0.0.1] amavisd-new closing transmission channel</pre><p>The maillog shows the delivery path. Here's an excerpt from a
+ successful delivery process:</p><pre class="programlisting">Nov 1 11:28:10 mail postfix/smtpd[30986]: connect from localhost[127.0.0.1] <a name="mailflow_1"></a><img src="images/callouts/1.png" alt="1" border="0">
+Nov 1 11:28:10 mail postfix/smtpd[30986]: 079474CE44: client=localhost[127.0.0.1]
+Nov 1 11:28:10 mail postfix/cleanup[30980]: 079474CE44: message-id=<20061101102810.079474CE44 at mail.example.com>
+Nov 1 11:28:10 mail postfix/qmgr[20432]: 079474CE44: from=<>, size=822, nrcpt=1 (queue active)
+Nov 1 11:28:10 mail amavis[30897]: (30897-02) Passed BAD-HEADER, <> -> <postmaster>, quarantine: badh-le5gjszxowBk, mail_id: le5gjszxowBk, Hits: -1.76, queued_as: 079474CE44, 39505 ms <a name="mailflow_2"></a><img src="images/callouts/2.png" alt="2" border="0">
+Nov 1 11:28:10 mail postfix/smtpd[30986]: disconnect from localhost[127.0.0.1]
+Nov 1 11:28:10 mail postfix/local[30987]: 079474CE44: to=<postmaster at example.com>, relay=local, delay=0.27, delays=0.14/0.05/0/0.08, dsn=2.0.0, status=sent (delivered to mailbox: postmaster) <a name="mailflow_3"></a><img src="images/callouts/3.png" alt="3" border="0">
+Nov 1 11:28:10 mail postfix/qmgr[20432]: 079474CE44: removed</pre><div class="calloutlist"><table border="0" summary="Callout list"><tr><td width="5%" valign="top" align="left"><a href="#mailflow_1"><img src="images/callouts/1.png" alt="1" border="0"></a> </td><td valign="top" align="left"><p>amavisd connects with Postfix dedicated smtpd-daemon and
+ hands over the e-mail that had been sent during the telnet
+ session. smtpd gives a queue-id of 079474CE44 that can be tracked
+ throughout the maillog.</p></td></tr><tr><td width="5%" valign="top" align="left"><a href="#mailflow_2"><img src="images/callouts/2.png" alt="2" border="0"></a> </td><td valign="top" align="left"><p>amavisd notices it has checked and sent an e-mail to
+ <postmaster>.</p></td></tr><tr><td width="5%" valign="top" align="left"><a href="#mailflow_3"><img src="images/callouts/3.png" alt="3" border="0"></a> </td><td valign="top" align="left"><p>Postfix' local-service logs it successfully delivered an
+ e-mail with queue-id 079474CE44 to the mailbox of
+ postmaster.</p></td></tr></table></div><p>If the test fails, the following questions may help to debug the
+ problem:</p><div class="itemizedlist"><ul type="disc"><li><p>Does amavisd-new log errors?</p></li><li><p>Does running amavisd-new in debug-mode report errors?</p></li></ul></div></div></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="filter"></a>3. Message filtering examples</h2></div></div></div><p>Postfix can use various criteria to decide whether a message should
+ be sent to amavisd-new for examination. Combinations of criteria may serve
+ to create different configurations. The following section describes the
+ following configurations:</p><div class="itemizedlist"><ul type="disc"><li><p>Filtering e-mail globally</p></li><li><p>Filtering e-mail globally by service</p></li><li><p>Filtering e-mail per recipient domain</p></li><li><p>Filtering e-mail per sender domain</p></li><li><p>Filtering e-mail by content</p></li></ul></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="filter_global"></a>3.1. Filtering E-mail globally</h3></div></div></div><p>In most cases email policies require global filtering - every
+ inbound and every outbound e-mail must be filtered by amavisd-new -
+ before it may be sent closer to its final destination.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note: Why check outgoing mail traffic?"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Why check outgoing mail traffic?</th></tr><tr><td align="left" valign="top"><p>Some reasons for checking mail coming from internal networks or
+ from authenticated roaming users are:</p><div class="itemizedlist"><ul type="disc"><li><p>detect an internal infected PC which is sending
+ viruses</p></li><li><p>detect an internal zombiized PC (or an internal open relay
+ or proxy) which is sending or relaying spam</p></li><li><p>let the SpamAssassin Bayes autolearning feature see a
+ balanced view of all mail, including useful samples of non-spam
+ originating from inside</p></li><li><p>make it possible for pen pals feature to function (if
+ enabled)</p></li></ul></div></td></tr></table></div><p>In Postfix global settings for its services are written to
+ <code class="filename">main.cf</code>. The <em class="parameter"><code>content_filter</code></em>
+ parameter, the parameter configuring that messages are sent to
+ amavisd-new, must therefore be placed in
+ <code class="filename">main.cf</code>.</p><p>The <em class="parameter"><code>content_filter</code></em> parameter requires a
+ triplet, consisting of the transport service's name (here: amavisfeed,
+ given in <a href="#basics_transport_lmtp-client" title="2.2.1. Configuring a dedicated lmtp-client">Section 2.2.1, “Configuring a dedicated lmtp-client”</a>), the target
+ hosts IP address and the port where amavisd-new listens for incoming
+ connections. Following the values used in this documents examples the
+ <em class="parameter"><code>content_filter</code></em> configuration results in
+ this:</p><pre class="programlisting">content_filter=amavisfeed:[127.0.0.1]:10024</pre><p>The new external content filter will be activated once Postfix has
+ been reloaded. Sending a test-mail verifies the system works.</p></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="filter_service_global"></a>3.2. Filtering E-mail by Postfix service</h3></div></div></div><p>Postfix is able to filter messages per service. Such configuration
+ requires the <em class="parameter"><code>content_filter</code></em> not to be applied
+ globally to all services in <code class="filename">main.cf</code> (see: <a href="#filter_global" title="3.1. Filtering E-mail globally">Section 3.1, “Filtering E-mail globally”</a>), but selectively, per service in
+ <code class="filename">master.cf</code>.</p><p>The following example presumes Postfix runs on a system offering
+ three IP addresses. In this example these are: <code class="systemitem">192.0.2.1</code> (WAN), <code class="systemitem">127.0.0.1</code> (localhost) and <code class="systemitem">10.0.0.254</code> (LAN). The goal is to filter
+ only e-mail that enters from the WAN interface.</p><p>This requires to create three dedicated smtpd-daemon instances,
+ each binding to one of the given IP addresses and deactivating the
+ global smtp service calling the smtpd command.</p><p>Additionally the WAN interface (here: 192.0.2.1:25) is configured
+ to use <em class="parameter"><code>content_filter</code></em>
+ <code class="option">=amavisfeed:[127.0.0.1]:10024</code> - it will delegate any
+ message that enters the Postfix mail system at this service to the
+ external amavisd content filter.</p><pre class="programlisting"># ==========================================================================
+# service type private unpriv chroot wakeup maxproc command + args
+# (yes) (yes) (yes) (never) (100)
+# ==========================================================================
+# smtp inet n - n - - smtpd
+
+...
+
+192.0.2.1:25 inet n - n - - smtpd
+ -o content_filter=amavisfeed:[127.0.0.1]:10024
+ -o receive_override_options=no_address_mappings
+
+10.0.0.254:25 inet n - n - - smtpd
+
+127.0.0.1:10025 inet n - n - - smtpd
+ -o content_filter=
+ -o smtpd_delay_reject=no
+ -o smtpd_client_restrictions=permit_mynetworks,reject
+ -o smtpd_helo_restrictions=
+ -o smtpd_sender_restrictions=
+ -o smtpd_recipient_restrictions=permit_mynetworks,reject
+ -o smtpd_data_restrictions=reject_unauth_pipelining
+ -o smtpd_end_of_data_restrictions=
+ -o smtpd_restriction_classes=
+ -o mynetworks=127.0.0.0/8
+ -o smtpd_error_sleep_time=0
+ -o smtpd_soft_error_limit=1001
+ -o smtpd_hard_error_limit=1000
+ -o smtpd_client_connection_count_limit=0
+ -o smtpd_client_connection_rate_limit=0
+ -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
+ -o local_header_rewrite_clients=</pre></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="filter_by_recipient"></a>3.3. Filtering E-Mails per Recipient Domain</h3></div></div></div><p>Postfix is able to filter e-mails per recipient domain. In order
+ to do this the <em class="parameter"><code>content_filter</code></em> parameter must not
+ be set globally (see: <a href="#filter_global" title="3.1. Filtering E-mail globally">Section 3.1, “Filtering E-mail globally”</a>). Instead the
+ <em class="parameter"><code>content_filter</code></em> parameter has to be associated
+ with one or more recipient domains listed in a lookup table
+ (map).</p><div class="caution" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Caution"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Caution]" src="images/caution.png"></td><th align="left">Caution</th></tr><tr><td align="left" valign="top"><p>This filter method is not selective! It will send
+ <span class="emphasis"><em>any</em></span> mail with a recipient domain listed in the
+ lookup table to amavis even if the mail contains another recipient
+ that <span class="emphasis"><em>should not</em></span> be examined by the amavis
+ framework.</p><p>If fully selective rules are required all mail should be sent to
+ amavis and amavis' own rule sets should be configured to decide
+ whether a message for a given recipient should be examined or
+ not.</p></td></tr></table></div><p>When Postfix searches the lookup table and finds the recipients
+ domain listed as key, it will take the action associated with that
+ domain. The action will send the message to a <code class="option">FILTER
+ amavisfeed:[127.0.0.1]:10024</code>.</p><p>The following map
+ <code class="filename">/etc/postfix/filter_recipient_domains</code> specifies to
+ send messages to the <code class="option">FILTER amavisfeed</code> whenever a
+ message for any recipient at example.com enters the Postfix
+ mailqueues:</p><pre class="programlisting">example.com FILTER amavisfeed:[127.0.0.1]:10024</pre><p>Once the table has been created the <span><strong class="command">postmap</strong></span>
+ command must be used to create an indexed map Postfix can read:</p><pre class="screen"># <strong class="userinput"><code>postmap /etc/postfix/filter_recipient_domains</code></strong></pre><p>Once the map has been indexed, the <span><strong class="command">postmap</strong></span>
+ command is used to test the map. In the following example the
+ <span><strong class="command">postmap</strong></span> command queries for the domain example.com
+ and returns the associated action:</p><pre class="screen"># <strong class="userinput"><code>postmap -q "example.com" /etc/postfix/filter_recipient_domains</code></strong>
+FILTER amavisfeed:[127.0.0.1]:10024</pre><p>The tested map must be added to <code class="filename">main.cf</code>,
+ before Postfix can make use of the new filter policy. Setting the
+ <em class="parameter"><code>check_recipient_access</code></em> parameter in the list of
+ <em class="parameter"><code>smtpd_recipient_restrictions</code></em> triggers evaluation
+ of entries in the map - <em class="parameter"><code>check_recipient_access</code></em> is
+ triggered by the envelope-recipient(s) given by a SMTP-client in a
+ SMTP-session with Postfix.</p><p>The following example puts the
+ <em class="parameter"><code>check_recipient_access</code></em> rule before
+ <em class="parameter"><code>permit_mynetworks</code></em> - all clients
+ envelope-recipient(s) will be filtered:</p><pre class="programlisting">smtpd_recipient_restrictions =
+ ...
+ <strong class="userinput"><code>check_recipient_access hash:/etc/postfix/filter_recipient_domains</code></strong>
+ ...
+ permit_mynetworks
+ reject_unauth_destination
+ ...</pre><h4><a name="d0e860"></a>Filtering E-Mails per Recipient Domain only from External
+ Clients</h4><p>This example puts the
+ <em class="parameter"><code>check_recipient_access</code></em> rule after
+ <em class="parameter"><code>permit_mynetworks</code></em> - only messages sent from
+ clients that are not in Postfix <em class="parameter"><code>$mynetworks</code></em> list
+ (external or untrusted clients) will be filtered:</p><pre class="programlisting">smtpd_recipient_restrictions =
+ ...
+ permit_mynetworks
+ reject_unauth_destination
+ <strong class="userinput"><code>check_recipient_access hash:/etc/postfix/filter_recipient_domains</code></strong>
+ ...</pre></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="filter_by_sender"></a>3.4. Filtering E-Mails by Sender-Domain</h3></div></div></div><p>In general it doesn't make sense to filter e-mails by
+ sender-domain, as anyone can fake a sender-domain during a SMTP-session.
+ Filtering by sender-domain will probably only make sense, if messages
+ are not filtered globally, but e-mails from ones own domain should be
+ checked for spam or viruses before they leave the network.</p><p>Most of the configuration steps are identical with the ones noted
+ in <a href="#filter_by_recipient" title="3.3. Filtering E-Mails per Recipient Domain">Section 3.3, “Filtering E-Mails per Recipient Domain”</a>, except for the parameter that
+ triggers evaluation of the indexed map. In this scenario
+ envelope-senders should trigger map evaluation. The map, named
+ <code class="filename">/etc/postfix/filter_sender_domains</code> this time,
+ contains the sender domain (example.com) and associates it with the
+ required FILTER:</p><pre class="programlisting">example.com FILTER amavisfeed:[127.0.0.1]:10024</pre><p>Once the map has been converted and tested with the
+ <span><strong class="command">postmap</strong></span> command (see: <a href="#filter_by_recipient" title="3.3. Filtering E-Mails per Recipient Domain">Section 3.3, “Filtering E-Mails per Recipient Domain”</a>) it must be added to the list of
+ <em class="parameter"><code>smtpd_recipient_restrictions</code></em> using the
+ <em class="parameter"><code>check_sender_access</code></em> parameter:</p><pre class="programlisting">smtpd_recipient_restrictions =
+ ...
+ <strong class="userinput"><code>check_sender_access hash:/etc/postfix/filter_sender_domains</code></strong>
+ ...
+ permit_mynetworks
+ reject_unauth_destination
+ ...</pre><div class="important" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Important"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Important]" src="images/important.png"></td><th align="left">Important</th></tr><tr><td align="left" valign="top"><p>The map must be listed before
+ <em class="parameter"><code>permit_mynetworks</code></em>, because only then it will be
+ applied to all clients - even the ones Postfix trusts, which are very
+ likely the ones from example.com.</p></td></tr></table></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="filter_by_content"></a>3.5. Filtering E-mail per Content</h3></div></div></div><p>Postfix is able - with deliberate limitations (see:
+ <code class="filename">BUILTIN_FILTER_README</code>) - to search for strings in
+ headers, the body and MIME-headers. If a string matches, Postfix may
+ call appropriate action.</p><p>The following example configures Postfix to look for the string
+ <code class="literal">offer</code> in Subject:-headers and delegate the message to
+ an external content filter if if finds a matching string.</p><p>A map, consisting of the search string noted as regexp-expression,
+ associates the search pattern with a FILTER action:</p><pre class="programlisting">/^Subject:.*offer/ FILTER amavisfeed:[127.0.0.1]:10024</pre><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note: Indexing regexp- or pcre-maps?"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Indexing regexp- or pcre-maps?</th></tr><tr><td align="left" valign="top"><p>regexp- or pcre-maps are and must be plaintext files. They
+ <span class="emphasis"><em>must not</em></span> and cannot be converted to an indexed
+ map using the <span><strong class="command">postmap</strong></span> command. They can be tested
+ using the <span><strong class="command">postmap</strong></span> command using the
+ <code class="option">-q</code> command line option.</p></td></tr></table></div><p>Once the map has been created, Postfix must be configured to use
+ it. The following example uses the <em class="parameter"><code>header_checks</code></em>
+ parameter (not <em class="parameter"><code>body_checks</code></em> or
+ <em class="parameter"><code>mime_header_checks</code></em> as they apply to other message
+ parts) to implement the map into the Postfix delivery process:</p><pre class="programlisting">header_checks = regexp:/etc/postfix/filter_header</pre><p>Once Postfix has been reloaded it will send every e-mail that
+ contains the word <code class="literal">offer</code> in the Subject:-header off to
+ the external amavisd content filter.</p></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d0e968"></a>4. Advanced Postfix and amavisd-new configuration</h2></div></div></div><p>In a post-queue content filtering setup, a mail message passes
+ through <code class="systemitem">smtpd</code> and
+ <code class="systemitem">cleanup</code> Postfix services twice, once before a
+ content filter, and the second time when an approved message is reinjected
+ from a content filter into the Postfix mail system. This is because checks
+ and transformations that have been configured in
+ <code class="filename">main.cf</code> are globally active and will be loaded and
+ run by any instance of these two services. To avoid wasting resources,
+ options that control runtime behavior of these services should not be
+ applied globally in <code class="filename">main.cf</code>, but selectively to
+ separate instances of these services in
+ <code class="filename">master.cf</code>.</p><p>Checks and transformations which are performed by a
+ <code class="systemitem">smtpd</code> Postfix service itself, e.g. access
+ controls, recipient validation, milters etc., can be controlled by adding
+ options (<code class="option">-o</code>) to appropriate
+ <code class="systemitem">smtpd</code> services. This has been shown in the basic
+ configuration examples (see: <a href="#basics_smtpd-daemon" title="2.3. Configuring a dedicated SMTP-server for message
 reinjection">Section 2.3, “Configuring a dedicated SMTP-server for message
+ reinjection”</a>).</p><p>Checks and transformations which are performed by a <code class="systemitem">cleanup</code> Postfix service are trickier because
+ in a normal Postfix setup there is only one
+ <code class="systemitem">cleanup</code> service, unlike
+ <code class="systemitem">smtpd</code> services of which there are many. Some of
+ the more important <code class="systemitem">cleanup</code> settings are
+ dynamically controllable by a <code class="systemitem">smtpd</code> service
+ through the use of its <em class="parameter"><code>receive_override_options</code></em>
+ option.</p><div class="tip" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Tip: Transformations and checks"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Tip]" src="images/tip.png"></td><th align="left">Transformations and checks</th></tr><tr><td align="left" valign="top"><p>Any transformation should preferably only be performed once,
+ either before or after content filtering. When to transform depends on
+ the desired effect, for example whether a content filter should see
+ unchanged or modified mail messages. Typical transformations are:</p><div class="itemizedlist"><ul type="disc"><li><p>rewrite addresses</p></li><li><p>add BCC recipients</p></li><li><p>modify mail header.</p></li></ul></div><p>Most checks should also be performed only once, preferably only on
+ the first passage, when the mail enters the Postfix mail system the
+ first time. This way messages can be rejected early - if needed - and
+ will not tie up downstream resources. Checking early also avoids bounces
+ in case of negative check results on a second passage after content
+ filtering.</p></td></tr></table></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="d0e1038"></a>4.1. Multiple cleanup service architecture</h3></div></div></div><p>To gain more control over a <code class="systemitem">cleanup</code>
+ service than offered by <em class="parameter"><code>receive_override_options</code></em>,
+ two (or more) <code class="systemitem">cleanup</code> services, each with its
+ own set of options, must be run. A Postfix setup with more than one
+ <code class="systemitem">cleanup</code> service is possible either with two
+ separate Postfix instances, or through a specification of services and
+ their options in <code class="filename">master.cf</code> file of a single Postfix
+ instance.</p><p>The following diagram illustrates a setup with two cleanup
+ services in a single Postfix instance:</p><div class="mediaobject"><pre class="literallayout"> .......................................
+ : Postfix :
+ ----->smtpd \ :
+ : -pre-cleanup-\ /local---->
+ ---->pickup / -queue- :
+ : -cleanup-/ | \smtp----->
+ : bounces/ ^ v :
+ : and locally | v :
+ : forwarded smtpd amavisfeed :
+ : messages 10025 | :
+ ...........................|...........
+ ^ |
+ | v
+ ............|...............................
+ : | $inet_socket_port=10024 :
+ : | :
+ : $forward_method='smtp:[127.0.0.1]:10025' :
+ : $notify_method ='smtp:[127.0.0.1]:10025' :
+ : :
+ : amavisd-new :
+ ............................................</pre></div><div class="procedure"><a name="d0e1066"></a><p class="title"><b>Procedure 1.1. Message flow with two cleanup services</b></p><ol type="1"><li><p>Messages enter the Postfix system at the regular
+ <code class="systemitem">smtpd</code> or <code class="systemitem">pickup</code>
+ service.</p></li><li><p>The <code class="systemitem">pre-cleanup</code> cleanup service
+ performs transformations and checks on these messages.</p></li><li><p>The <code class="systemitem">qmgr</code> service schedules the
+ messages to be sent to the amavisd-new content filter.</p></li><li><p><code class="systemitem">amavisd-new</code> performs various tests on
+ the messages.</p></li><li><p>Messages are re-injected into the Postfix mail system, sending
+ them to a dedicated, local <code class="systemitem">smtpd</code>
+ service.</p></li><li><p>The <code class="systemitem">cleanup</code> cleanup service performs
+ transformations and checks that must be done at this stage,
+ <span class="emphasis"><em>but</em></span> omits the ones that have already been
+ carried out in step 2.</p></li></ol></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="d0e1110"></a>4.2. Configuring two cleanup services</h3></div></div></div><p>Configuring Postfix <code class="systemitem">smtpd</code> services to use
+ two separate, dedicated <code class="systemitem">cleanup</code> services
+ requires the following steps:</p><div class="procedure"><ol type="1"><li><p>Create a second <code class="systemitem">cleanup</code>
+ instance</p></li><li><p>Modify the existing <code class="systemitem">cleanup</code>
+ service</p></li><li><p>Configure <code class="systemitem">smtpd</code> services to use
+ either of the two <code class="systemitem">cleanup</code> services.</p></li></ol></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="d0e1143"></a>4.2.1. Creating a second cleanup instance</h4></div></div></div><p>The following example adds a cleanup daemon named
+ <code class="systemitem">pre-cleanup</code>. It will handle messages before a
+ content filter.</p><pre class="programlisting"># ==========================================================================
+# service type private unpriv chroot wakeup maxproc command + args
+# (yes) (yes) (yes) (never) (100)
+# ==========================================================================
+# smtp inet n - n - - smtpd
+
+...
+
+pre-cleanup unix n - n - 0 cleanup
+ -o virtual_alias_maps= </pre><p>The above leaves canonicalization address rewriting enabled so
+ that a content filter will see canonicalized (external) sender mail
+ addresses, but it disables globally configured virtual alias
+ transformations.</p><p>Such transformations will be done later by the second
+ <code class="systemitem">cleanup</code> service, so that a content filter
+ will see original (external) recipient mail addresses. Other options
+ may also be used as needed.</p></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="d0e1160"></a>4.2.2. Modifying the existing cleanup service</h4></div></div></div><p>The already existing cleanup service - having the service name
+ <code class="systemitem">cleanup</code> - will be used to process messages
+ that re-enter the Postfix mail system (also for delivery notifications
+ and forwarding as generated internally by Postfix).</p><p>Cleanup jobs that already have been performed by the
+ <code class="systemitem">pre-cleanup</code> service should not be run again.
+ The following example disables typical checks that have been run
+ before or are not needed for internally generated
+ notifications:</p><pre class="programlisting"># ==========================================================================
+# service type private unpriv chroot wakeup maxproc command + args
+# (yes) (yes) (yes) (never) (100)
+# ==========================================================================
+# smtp inet n - n - - smtpd
+
+...
+
+cleanup unix n - n - 0 cleanup
+ -o mime_header_checks= <a name="cleanup-mime_header_checks"></a><img src="images/callouts/1.png" alt="1" border="0">
+ -o nested_header_checks= <a name="cleanup-nested_header_checks"></a><img src="images/callouts/2.png" alt="2" border="0">
+ -o body_checks= <a name="cleanup-body_checks"></a><img src="images/callouts/3.png" alt="3" border="0">
+ -o header_checks= <a name="cleanup-header_checks"></a><img src="images/callouts/4.png" alt="4" border="0"></pre><div class="calloutlist"><table border="0" summary="Callout list"><tr><td width="5%" valign="top" align="left"><a href="#cleanup-mime_header_checks"><img src="images/callouts/1.png" alt="1" border="0"></a> <a href="#cleanup-nested_header_checks"><img src="images/callouts/2.png" alt="2" border="0"></a> <a href="#cleanup-body_checks"><img src="images/callouts/3.png" alt="3" border="0"></a> <a href="#cleanup-header_checks"><img src="images/callouts/4.png" alt="4" border="0"></a> </td><td valign="top" align="left"><p>The specified options disable header and body checks as
+ these would already be performed by a
+ <code class="systemitem">pre-cleanup</code> service.</p></td></tr></table></div><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note: always_bcc"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">always_bcc</th></tr><tr><td align="left" valign="top"><p>This <code class="systemitem">cleanup</code> service would also be
+ the appropriate one for specifying <em class="parameter"><code>always_bcc</code></em>
+ option - doing it globally would apply to both
+ <code class="systemitem">cleanup</code> services and would result in
+ <span class="emphasis"><em>two copies of each message</em></span> to be sent to the
+ specified address.</p></td></tr></table></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="d0e1206"></a>4.2.3. Configuring smtpd services</h4></div></div></div><p>Finally existing <code class="systemitem">smtpd</code> services on
+ ports 25 and 587 (<code class="systemitem">submission</code>), and the
+ <code class="systemitem">pickup</code> service must be configured to send
+ messages to the new <code class="systemitem">pre-cleanup</code> service
+ instead of a default <code class="systemitem">cleanup</code> service:</p><pre class="programlisting"># ==========================================================================
+# service type private unpriv chroot wakeup maxproc command + args
+# (yes) (yes) (yes) (never) (100)
+# ==========================================================================
+# smtp inet n - n - - smtpd
+
+...
+
+pickup fifo n - n 60 1 pickup
+ -o cleanup_service_name=pre-cleanup
+smtp inet n - n - - smtpd
+ -o cleanup_service_name=pre-cleanup
+submission inet n - n - - smtpd
+ -o cleanup_service_name=pre-cleanup</pre></div></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tuning"></a>5. Tuning</h2></div></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="d0e1231"></a>5.1. Maximum Number of Concurrent Processes</h3></div></div></div><p>The most important settings to tune and optimize in Postfix and
+ amavisd workflow are the maximum number of concurrent processes. The
+ maximum number of concurrent processes on both sides must be chosen with
+ care.</p><p>If the number is too low, hardware resources aren't used
+ efficiently and delivery time will be unnecessarily prolonged.
+ Experience tells that raising the number of processes a little, will not
+ raise the overall throughput in the same proportion.</p><p>As the system resources are nearing saturation with each increase
+ of the number of processes, an increase in throughput becomes marginal,
+ and eventually even negative when the number of processes exceeds its
+ near-optimum value. E-mail throughput will decrease, because processes
+ need to wait for each other. At worst e-mail delivery stalls.</p><p>Best practice is to start with a (conservative) maximum number of
+ 2 concurrent processes. Everyday use has shown that this value may be
+ raised to a value between 10 and 30 concurrent Postfix client and
+ amavisd server processes. This also depends on the overall resources the
+ system may provide, how amavisd has been integrated into the Postfix
+ delivery process and on the anti-virus and anti-spam software being
+ loaded and used by amavisd-new.</p><p>Regardless of the maximum number of concurrent processes, both
+ sides - Postfix and amavisd - should be synchronized. To synchronize
+ both sides edit, the <em class="parameter"><code>$max_servers</code></em> parameter for
+ amavisd-new (see: <code class="filename">amavisd.conf</code>) and the number of
+ processes in <code class="filename">master.cf</code> listed in the dedicated
+ transports <em class="parameter"><code>maxproc</code></em> column for Postfix.</p><p>Both values should be identical for two reasons: If amavisd-new
+ offers more processes than Postfix will ever use, amavisd-new wastes
+ resources. On the other hand, if Postfix starts more dedicated
+ transports than amavisd can handle simultaneously, e-mail transport will
+ be refused and logged as error.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note: Controlling the maximum number of concurrent processes in
 main.cf"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Controlling the maximum number of concurrent processes in
+ main.cf</th></tr><tr><td align="left" valign="top"><p>Instead of controlling the maximum number of concurrent
+ processes of Postfix' dedicated transport in
+ <code class="filename">master.cf</code> it is also possible to keep the default
+ setting <code class="option">-</code> in <code class="filename">master.cf</code> and set
+ the following parameter and option in
+ <code class="filename">main.cf</code>:</p><pre class="programlisting">amavisfeed_destination_concurrency_limit = 2</pre><p>The name of the parameter starts with the service in
+ <code class="filename">master.cf</code> (here: amavisfeed) that should be
+ controlled and goes on with the suffix
+ <em class="parameter"><code>_destination_concurrency_limit</code></em>. Here also
+ <code class="option">2</code> is set as initial (conservative) value.</p></td></tr></table></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="d0e1288"></a>5.2. Additional Tips for Tuning</h3></div></div></div><p>Further Tuning-Tips can be found in
+ <code class="filename">README.performance</code> and the slides from <a href="http://www.ijs.si/software/amavisd/amavisd-new-magdeburg-20050519.pdf" target="_top">amavisd-new,
+ advanced configuration and management</a>.</p></div></div></div></body></html>
\ No newline at end of file
diff --git a/README_FILES/README.protocol b/README_FILES/README.protocol
--- a/README_FILES/README.protocol
+++ b/README_FILES/README.protocol
@@ -2,7 +2,8 @@ AMAVIS POLICY DELEGATION PROTOCOL (AM.PD
==========================================
Author: Mark Martinec <Mark.Martinec at ijs.si>
Created: 2003-11-10, last modified 2004-09-09, 2005-03-18, 2005-06-22,
- 2006-08-18 (added attributes: version_server, insheader and quarantine)
+ 2006-08-18 (added attributes: version_server, insheader and quarantine),
+ 2007-05-17 (allow attribute value: request=requeue)
NOTE: at the end of this document there is a description
(by Stephane Lentz) of the currently used protocol.
@@ -109,8 +110,8 @@ sender=<foo at example.com>
sender=<foo at example.com>
specifies the envelope sender address (reverse-path).
The attribute should appear exactly once. The attribute value syntax
- is specified in rfc2821 as 'Reverse-path'; a null reverse path
- is specified as <>
+ is specified in rfc2821 as 'Reverse-path' (i.e. smtp-quoted form,
+ enclosed in <>); a null reverse path is specified as <>.
recipient=<user1 at example.net>
specifies the envelope recipient address. The attribute appears once
@@ -171,6 +172,15 @@ client_name=mail.example.com
optional informational attribute: the DNS name of the original SMTP client
as obtained by DNS reverse mapping of the original SMTP client;
+policy_bank=TLS,ORIGINATING,MYNETS
+ value is a comma-separated list of policy bank names. Names of
+ nonexistent banks are silently ignored, so are leading and trailing spaces
+ and TABs around each name. The order of policy bank loading generally
+ follows the order in which information about a message were obtained:
+ - interface or socket -based policy banks (when MTA connects to amavisd);
+ - MYNETS (when client's IP address becomes known);
+ - the list as specified in the policy_bank attribute of AM.PDP;
+ - MYUSERS (when sender e-mail address becomes known);
Attributes in the server response are:
--------------------------------------
@@ -190,12 +200,13 @@ version_server=2
- new attribute insheader, see below;
- new attribute quarantine, see below.
-delrcpt=user1 at example.net
+delrcpt=<user1 at example.net>
The specified recipient should be removed from the list of
recipients of this mail. The specified address must exactly match
- the recipient addresses as specified in the client request.
-
-addrcpt=user1 at example.net
+ the recipient addresses as specified in the client request,
+ i.e. a smtp-quoted form enclosed in <> should be specified;
+
+addrcpt=<user1 at example.net>
The specified recipient should be added to the list of recipients
of this mail. Paired with 'delrcpt' the pair indicates an existing
recipient address to be replaced by a modified address, e.g. to
@@ -216,11 +227,14 @@ insheader=index hdr_head hdr_body
insheader=index hdr_head hdr_body
Similar to addheader, but specifies a mail header field to be inserted to
the mail header at a given position, index 0 implies top of the header.
- Amavisd-new currently (2.4.3) always specifies index to be 0, meaning the
- added header field is to be prepended to the header. Header fields to be
- prepended are listed in reverse order, the last one listed in AM.PDP is
- to appear at the top of a resulting mail header. Inserting a header field
- at an arbitrary position is a later addition to sendmail milter protocol
+ Amavisd-new passes a value of $prepend_header_fields_hdridx as the index
+ argument, which is configurable and defaults to 1 for compatibility with
+ dkim-milter and dk-milter signing milters. Alternative useful value is 0,
+ which may be used in absence of other milters inserting their header fields
+ at index 1. Header fields to be prepended are listed in reverse order,
+ the last one listed in AM.PDP is to appear at the top of a resulting mail
+ header. Inserting a header field at an arbitrary position is a later
+ addition to sendmail milter protocol
(introduced with sendmail 8.13.0 2004-06-20)
addheader=hdr_head hdr_body
@@ -303,7 +317,7 @@ is not part of the actual session.
===============================================================================
Releasing a message from a quarantine:
-request=release
+request=release (or request=requeue, available since 2.5.1)
mail_id=xxxxxxxxxxxx
secret_id=xxxxxxxxxxxx (authorizes a release)
quar_type=x F/Z/B/Q/M (file/zipfile/bsmtp/sql/mailbox)
diff --git a/README_FILES/README.sql b/README_FILES/README.sql
--- a/README_FILES/README.sql
+++ b/README_FILES/README.sql
@@ -1,11 +1,17 @@ USING SQL FOR LOOKUPS, LOG/REPORTING AND
USING SQL FOR LOOKUPS, LOG/REPORTING AND QUARANTINE
===================================================
-This text contains general SQL-related documentation. For aspects
+This text contains a general SQL-related documentation. For aspects
specific to using SQL database for lookups, please see README.lookups .
-Since amavisd-new-20020630 SQL is supported for lookups.
-Since amavisd-new-2.3.0 SQL is also supported for storing information
+For general aspects of lookups, please see README.lookups.
+For MySQL-specific notes and schema please see README.sql-mysql.
+For PostgreSQL-specific notes and schema please see README.sql-pg
+(which in most respects applies also to a SQLite database).
+
+
+Since version of amavisd-new-20020630 a SQL is supported for lookups.
+Since amavisd-new-2.3.0, SQL is also supported for storing information
about processed mail (logging/reporting) and optionally for quarantining
to a SQL database.
@@ -62,243 +68,42 @@ The database specified in @storage_sql_d
(SELECT, INSERT, UPDATE), and a database server offering transactions
must be used.
-
-Below is an example that can be used with MySQL or PostgreSQL or SQLite.
-The provided schema can be cut/pasted or fed directly into the client program
-to create a database. The '--' introduces comments according to SQL specs.
-
--- MySQL notes:
--- - SERIAL can be used instead of INT UNSIGNED NOT NULL AUTO_INCREMENT
--- with databases which do not recognize AUTO_INCREMENT;
--- The attribute SERIAL was introduced with MySQL 4.1.0, but it
--- implicitly creates an additional UNIQUE index, which is redundant.
--- - instead of declaring a time_iso field in table msgs as a string:
--- time_iso char(16) NOT NULL,
--- one may want to declare is as:
--- time_iso TIMESTAMP NOT NULL DEFAULT 0,
--- in which case $timestamp_fmt_mysql *MUST* be set to 1 in amavisd.conf
--- to avoid MySQL inability to accept ISO 8601 timestamps with zone Z
--- and ISO date/time delimiter T; failing to set $timestamp_fmt_mysql
--- makes MySQL store zero time on INSERT and write current local time
--- on UPDATE if auto-update is allowed, which is different from the
--- intended mail timestamp (localtime vs. UTC, off by seconds)
-
--- PostgreSQL notes (initially provided by Phil Regnauld):
--- - use SERIAL instead of INT UNSIGNED NOT NULL AUTO_INCREMENT
--- - remove the 'unsigned' throughout,
--- - remove the 'ENGINE=InnoDB' throughout,
--- - instead of declaring time_iso field in table msgs as a string:
--- time_iso char(16) NOT NULL,
--- it is more useful to declare it as:
--- time_iso TIMESTAMP WITH TIME ZONE NOT NULL,
--- if changing existing table from char to timestamp is desired,
--- the following clause can be used:
--- ALTER TABLE msgs ALTER time_iso
--- TYPE TIMESTAMP WITH TIME ZONE
--- USING to_timestamp(time_iso,'YYYYMMDDTHH24MMSSTZ');
--- - create an amavis username and the database (choose name, e.g. mail)
--- $ createuser -U pgsql --no-adduser --createdb amavis
--- $ createdb -U amavis mail
--- - populate the database using the schema below:
--- $ psql -U amavis mail < amavisd-pg.sql
-
--- SQLite notes:
--- - use INTEGER PRIMARY KEY AUTOINCREMENT
--- instead of INT UNSIGNED NOT NULL AUTO_INCREMENT;
--- - replace SERIAL by INTEGER;
--- - SQLite is well suited for lookups database, but is not appropriate
--- for @storage_sql_dsn due to coarse lock granularity;
-
--- local users
-CREATE TABLE users (
- id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY, -- unique id
- priority integer NOT NULL DEFAULT '7', -- sort field, 0 is low prior.
- policy_id integer unsigned NOT NULL DEFAULT '1', -- JOINs with policy.id
- email varchar(255) NOT NULL UNIQUE,
- fullname varchar(255) DEFAULT NULL, -- not used by amavisd-new
- local char(1) -- Y/N (optional field, see note further down)
-);
-
--- any e-mail address (non- rfc2822-quoted), external or local,
--- used as senders in wblist
-CREATE TABLE mailaddr (
- id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
- priority integer NOT NULL DEFAULT '7', -- 0 is low priority
- email varchar(255) NOT NULL UNIQUE
-);
-
--- per-recipient whitelist and/or blacklist,
--- puts sender and recipient in relation wb (white or blacklisted sender)
-CREATE TABLE wblist (
- rid integer unsigned NOT NULL, -- recipient: users.id
- sid integer unsigned NOT NULL, -- sender: mailaddr.id
- wb varchar(10) NOT NULL, -- W or Y / B or N / space=neutral / score
- PRIMARY KEY (rid,sid)
-);
-
-CREATE TABLE policy (
- id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
- -- 'id' this is the _only_ required field
- policy_name varchar(32), -- not used by amavisd-new, a comment
-
- virus_lover char(1) default NULL, -- Y/N
- spam_lover char(1) default NULL, -- Y/N
- banned_files_lover char(1) default NULL, -- Y/N
- bad_header_lover char(1) default NULL, -- Y/N
-
- bypass_virus_checks char(1) default NULL, -- Y/N
- bypass_spam_checks char(1) default NULL, -- Y/N
- bypass_banned_checks char(1) default NULL, -- Y/N
- bypass_header_checks char(1) default NULL, -- Y/N
-
- spam_modifies_subj char(1) default NULL, -- Y/N
-
- virus_quarantine_to varchar(64) default NULL,
- spam_quarantine_to varchar(64) default NULL,
- banned_quarantine_to varchar(64) default NULL,
- bad_header_quarantine_to varchar(64) default NULL,
- clean_quarantine_to varchar(64) default NULL,
- other_quarantine_to varchar(64) default NULL,
-
- spam_tag_level float default NULL, -- higher score inserts spam info headers
- spam_tag2_level float default NULL, -- inserts 'declared spam' header fields
- spam_kill_level float default NULL, -- higher score triggers evasive actions
- -- e.g. reject/drop, quarantine, ...
- -- (subject to final_spam_destiny setting)
- spam_dsn_cutoff_level float default NULL,
- spam_quarantine_cutoff_level float default NULL,
-
- addr_extension_virus varchar(64) default NULL,
- addr_extension_spam varchar(64) default NULL,
- addr_extension_banned varchar(64) default NULL,
- addr_extension_bad_header varchar(64) default NULL,
-
- warnvirusrecip char(1) default NULL, -- Y/N
- warnbannedrecip char(1) default NULL, -- Y/N
- warnbadhrecip char(1) default NULL, -- Y/N
- newvirus_admin varchar(64) default NULL,
- virus_admin varchar(64) default NULL,
- banned_admin varchar(64) default NULL,
- bad_header_admin varchar(64) default NULL,
- spam_admin varchar(64) default NULL,
- spam_subject_tag varchar(64) default NULL,
- spam_subject_tag2 varchar(64) default NULL,
- message_size_limit integer default NULL, -- max size in bytes, 0 disable
- banned_rulenames varchar(64) default NULL -- comma-separated list of ...
- -- names mapped through %banned_rules to actual banned_filename tables
-);
-
-
-
--- R/W part of the dataset (optional)
--- May reside in the same or in a separate database as lookups database;
--- REQUIRES SUPPORT FOR TRANSACTIONS; specified in @storage_sql_dsn
---
--- MySQL note ( http://dev.mysql.com/doc/mysql/en/storage-engines.html ):
--- ENGINE is the preferred term, but cannot be used before MySQL 4.0.18.
--- TYPE is available beginning with MySQL 3.23.0, the first version of
--- MySQL for which multiple storage engines were available. If you omit
--- the ENGINE or TYPE option, the default storage engine is used.
--- By default this is MyISAM.
---
--- Please create additional indexes on keys when needed, or drop suggested
--- ones as appropriate to optimize queries needed by a management application.
--- See your database documentation for further optimization hints. With MySQL
--- see Chapter 15 of the reference manual. For example the chapter 15.17 says:
--- InnoDB does not keep an internal count of rows in a table. To process a
--- SELECT COUNT(*) FROM T statement, InnoDB must scan an index of the table,
--- which takes some time if the index is not entirely in the buffer pool.
-
-
--- provide unique id for each e-mail address, avoids storing copies
-CREATE TABLE maddr (
- id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
- email varchar(255) NOT NULL UNIQUE, -- full mail address
- domain varchar(255) NOT NULL -- only domain part of the email address
- -- with subdomain fields in reverse
-) ENGINE=InnoDB;
-
--- information pertaining to each processed message as a whole;
--- NOTE: records with NULL msgs.content should be ignored by utilities,
--- as such records correspond to messages just being processes, or were lost
--- NOTE: with PostgreSQL, instead of a character field time_iso, please use:
--- time_iso TIMESTAMP WITH TIME ZONE NOT NULL,
--- NOTE: with MySQL, instead of a character field time_iso, one might prefer:
--- time_iso TIMESTAMP NOT NULL DEFAULT 0,
--- but the following MUST then be set in amavisd.conf: $timestamp_fmt_mysql=1
-CREATE TABLE msgs (
- mail_id varchar(12) NOT NULL PRIMARY KEY, -- long-term unique mail id
- secret_id varchar(12) DEFAULT '', -- authorizes release of mail_id
- am_id varchar(20) NOT NULL, -- id used in the log
- time_num integer unsigned NOT NULL, -- rx_time: seconds since Unix epoch
- time_iso char(16) NOT NULL, -- rx_time: ISO8601 UTC ascii time
- sid integer unsigned NOT NULL, -- sender: maddr.id
- policy varchar(255) DEFAULT '', -- policy bank path (like macro %p)
- client_addr varchar(255) DEFAULT '', -- SMTP client IP address (IPv4 or v6)
- size integer unsigned NOT NULL, -- message size in bytes
- content char(1), -- content type: V/B/S/s/M/H/O/C:
- -- virus/banned/spam(kill)/spammy(tag2)
- -- /bad mime/bad header/oversized/clean
- -- is NULL on partially processed mail
- quar_type char(1), -- quarantined as: ' '/F/Z/B/Q/M
- -- none/file/zipfile/bsmtp/sql/mailbox
- quar_loc varchar(255) DEFAULT '', -- quarantine location (e.g. file)
- dsn_sent char(1), -- was DSN sent? Y/N/q (q=quenched)
- spam_level float, -- SA spam level (no boosts)
- message_id varchar(255) DEFAULT '', -- mail Message-ID header field
- from_addr varchar(255) DEFAULT '', -- mail From header field, UTF8
- subject varchar(255) DEFAULT '', -- mail Subject header field, UTF8
- host varchar(255) NOT NULL, -- hostname where amavisd is running
- FOREIGN KEY (sid) REFERENCES maddr(id) ON DELETE RESTRICT
-) ENGINE=InnoDB;
-CREATE INDEX msgs_idx_sid ON msgs (sid);
-CREATE INDEX msgs_idx_time_num ON msgs (time_num);
--- alternatively when purging based on time_iso (instead of msgs_idx_time_num):
--- CREATE INDEX msgs_idx_time_iso ON msgs (time_iso);
-
--- per-recipient information related to each processed message;
--- NOTE: records in msgrcpt without corresponding msgs.mail_id record are
--- orphaned and should be ignored and eventually deleted by external utilities
-CREATE TABLE msgrcpt (
- mail_id varchar(12) NOT NULL, -- (must allow duplicates)
- rid integer unsigned NOT NULL, -- recipient: maddr.id (dupl. allowed)
- ds char(1) NOT NULL, -- delivery status: P/R/B/D/T
- -- pass/reject/bounce/discard/tempfail
- rs char(1) NOT NULL, -- release status: initialized to ' '
- bl char(1) DEFAULT ' ', -- sender blacklisted by this recip
- wl char(1) DEFAULT ' ', -- sender whitelisted by this recip
- bspam_level float, -- spam level + per-recip boost
- smtp_resp varchar(255) DEFAULT '', -- SMTP response given to MTA
- FOREIGN KEY (rid) REFERENCES maddr(id) ON DELETE RESTRICT,
- FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE
-) ENGINE=InnoDB;
-CREATE INDEX msgrcpt_idx_mail_id ON msgrcpt (mail_id);
-CREATE INDEX msgrcpt_idx_rid ON msgrcpt (rid);
-
--- mail quarantine in SQL, enabled by $*_quarantine_method='sql:'
--- NOTE: records in quarantine without corresponding msgs.mail_id record are
--- orphaned and should be ignored and eventually deleted by external utilities
-CREATE TABLE quarantine (
- mail_id varchar(12) NOT NULL, -- long-term unique mail id
- chunk_ind integer unsigned NOT NULL, -- chunk number, starting with 1
- mail_text text NOT NULL, -- store mail as chunks up to 16 kB
- PRIMARY KEY (mail_id,chunk_ind),
- FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE
-) ENGINE=InnoDB;
-
--- field msgrcpt.rs is primarily intended for use by quarantine management
--- software; the value assigned by amavisd is a space;
--- a short _preliminary_ list of possible values:
--- 'V' => viewed (marked as read)
--- 'R' => released (delivered) to this recipient
--- 'p' => pending (a status given to messages when the admin received the
--- request but not yet released; targeted to banned parts)
--- 'D' => marked for deletion; a cleanup script may delete it
-
-
--- =====================
--- Example data follows:
--- =====================
+Database schemas are available in README.sql-mysql (for MySQL)
+and in README.sql-pg (for PostgreSQL and SQLite).
+
+There are two parts of a schema, an read-only part used for lookups,
+and a R/W part used for logging and quarantining. They are completely
+independent and may reside on different SQL servers (even on different
+types of SQL server), but may also coexist in a single database if desired.
+
+Note that some databases are very well suited for lookups, but less so for
+highly concurent transactional use in logging/quarantining. Some experience:
+
+- SQLite works nicely for lookups, avoiding a need for a separate server
+ process, but its coarse locking granularity makes its unquitable for
+ logging and quarantining;
+
+- MySQL and PostgreSQL are both fine for lookups;
+
+- PostgreSQL is better suited for SQL logging/quarantining because maintenance
+ operations (cleaning of old records) are much faster than with MySQL;
+ Note that SQL logging is needed for amavisd-new pen-pals feature to work;
+
+- if using MySQL for logging/quarantining, a sufficiently recent version
+ must be used, as support for transactions is required for the R/W access;
+
+- if using SpamAssassin with its Bayes (and AWL) database on SQL,
+ bayes plugin works faster with MySQL than with PostgreSQL; note that
+ SA databases are independent from amavisd-new databases and may reside
+ on a separate SQL server, possibly of a different type. See SpamAssassin
+ documentation that comes with its distribution, files sql/README* .
+
+
+
+=====================
+Example data follows:
+=====================
+
INSERT INTO users VALUES ( 1, 9, 5, 'user1+foo at y.example.com','Name1 Surname1', 'Y');
INSERT INTO users VALUES ( 2, 7, 5, 'user1 at y.example.com', 'Name1 Surname1', 'Y');
INSERT INTO users VALUES ( 3, 7, 2, 'user2 at y.example.com', 'Name2 Surname2', 'Y');
@@ -377,70 +182,8 @@ INSERT INTO wblist VALUES (17, 4, 'B');
--- NOTE: the SELECT, INSERT and UPDATE clauses as used by the amavisd-new
--- program are configurable through %sql_clause; see amavisd.conf-default
-
-
-
-Some examples of a query:
-
--- mail from last two minutes (MySQL):
-SELECT
- UNIX_TIMESTAMP()-msgs.time_num AS age, SUBSTRING(policy,1,2) as pb,
- content AS c, dsn_sent as dsn, ds, bspam_level AS level, size,
- SUBSTRING(sender.email,1,18) AS s,
- SUBSTRING(recip.email,1,18) AS r,
- SUBSTRING(msgs.subject,1,10) AS subj
- FROM msgs LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
- LEFT JOIN maddr AS sender ON msgs.sid=sender.id
- LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
- WHERE content IS NOT NULL AND UNIX_TIMESTAMP()-msgs.time_num < 120
- ORDER BY msgs.time_num DESC;
-
--- mail from last two minutes (PostgreSQL):
-SELECT
- now()-time_iso AS age, SUBSTRING(policy,1,2) as pb,
- content AS c, dsn_sent as dsn, ds, bspam_level AS level, size,
- SUBSTRING(sender.email,1,18) AS s,
- SUBSTRING(recip.email,1,18) AS r,
- SUBSTRING(msgs.subject,1,10) AS subj
- FROM msgs LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
- LEFT JOIN maddr AS sender ON msgs.sid=sender.id
- LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
- WHERE content IS NOT NULL AND now() - time_iso < INTERVAL '2 minutes'
- ORDER BY msgs.time_num DESC;
-
--- clean messages ordered by count, grouped by domain:
-SELECT count(*) as cnt, avg(bspam_level), sender.domain
- FROM msgs
- LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
- LEFT JOIN maddr AS sender ON msgs.sid=sender.id
- LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
- WHERE content='C'
- GROUP BY sender.domain ORDER BY cnt DESC LIMIT 50;
-
--- top spamy domains with >10 messages, sorted by spam average,
--- grouped by domain:
-SELECT count(*) as cnt, avg(bspam_level) as spam_avg, sender.domain
- FROM msgs
- LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
- LEFT JOIN maddr AS sender ON msgs.sid=sender.id
- LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
- WHERE bspam_level IS NOT NULL
- GROUP BY sender.domain HAVING count(*) > 10
- ORDER BY spam_avg DESC LIMIT 50;
-
--- sender domains with >100 messages, sorted on sender.domain:
-SELECT count(*) as cnt, avg(bspam_level) as spam_avg, sender.domain
- FROM msgs
- LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
- LEFT JOIN maddr AS sender ON msgs.sid=sender.id
- LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
- GROUP BY sender.domain HAVING count(*) > 100
- ORDER BY sender.domain DESC LIMIT 100;
-
-
-
+NOTE: the SELECT, INSERT and UPDATE clauses as used by the amavisd-new
+program are configurable through %sql_clause; see amavisd.conf-default
Upgrading from pre 2.4.0 amavisd-new SQL schema to the 2.4.0 schema requires
adding column 'quar_loc' to table msgs, and creating FOREIGN KEY constraint
@@ -465,89 +208,3 @@ SQL schema to the 2.4.0 schema:
ADD FOREIGN KEY (sid) REFERENCES maddr(id) ON DELETE RESTRICT;
ALTER TABLE msgrcpt
ADD FOREIGN KEY (rid) REFERENCES maddr(id) ON DELETE RESTRICT;
-
-
-BRIEF MySQL EXAMPLE of a log/report/quarantine database housekeeping
-====================================================================
-
-DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP() - 14*24*60*60;
-DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP() - 60*60 AND content IS NULL;
-DELETE FROM maddr
- WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id)
- AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);
-
-
-BRIEF MySQL EQUIVALENT EXAMPLE based on time_iso if its data type is TIMESTAMPS
-===============================================================================
-(don't forget to set: $timestamp_fmt_mysql=1 in amavisd.conf)
-
-DELETE FROM msgs WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 14 day;
-DELETE FROM msgs WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 1 hour
- AND content IS NULL;
-DELETE FROM maddr
- WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id)
- AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);
-
-
-BRIEF PostgreSQL EXAMPLE of a log/report/quarantine database housekeeping
-=========================================================================
-
-DELETE FROM msgs WHERE time_iso < now() - INTERVAL '14 days';
-DELETE FROM msgs WHERE time_iso < now() - INTERVAL '1 h' AND content IS NULL;
-DELETE FROM maddr
- WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id)
- AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);
-
-
-COMMENTED LONGER EXAMPLE of a log/report/quarantine database housekeeping
-=========================================================================
-
--- discarding indexes makes deletion faster; if we expect a large proportion
--- of records to be deleted it may be quicker to discard index, do deletions,
--- and re-create index (not necessary with PostgreSQL, may benefit MySQL);
--- for daily maintenance this does not pay off
---DROP INDEX msgs_idx_sid ON msgs;
---DROP INDEX msgrcpt_idx_rid ON msgrcpt;
---DROP INDEX msgrcpt_idx_mail_id ON msgrcpt;
-
--- delete old msgs records based on timestamps only (for time_iso see next),
--- and delete leftover msgs records from aborted mail checking operations
-DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP()-14*24*60*60;
-DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP()-60*60 AND content IS NULL;
-
--- provided the time_iso field was created as type TIMESTAMP DEFAULT 0 (MySQL)
--- or TIMESTAMP WITH TIME ZONE (PostgreSQL), instead of purging based on
--- numerical Unix timestamp as above, one may select records based on ISO 8601
--- UTC timestamps. This is particularly suitable for PostgreSQL:
---DELETE FROM msgs WHERE time_iso < now() - INTERVAL '14 days';
---DELETE FROM msgs WHERE time_iso < now() - INTERVAL '1 h' AND content IS NULL;
-and is also possible with MySQL, using slightly different format:
---DELETE FROM msgs
--- WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 14 day;
---DELETE FROM msgs
--- WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 1 hour AND content IS NULL;
-
--- optionally certain content types may be given shorter lifetime
---DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP()-7*24*60*60
--- AND (content='V' OR (content='S' AND spam_level>20));
-
--- (optional) just in case the ON DELETE CASCADE did not do its job, we may
--- explicitly delete orphaned records (with no corresponding msgs entry);
--- if ON DELETE CASCADE did work, there should be no deletions at this step
-DELETE FROM quarantine
- WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=quarantine.mail_id);
-DELETE FROM msgrcpt
- WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=msgrcpt.mail_id);
-
--- re-create indexes (if they were removed in the first step):
---CREATE INDEX msgs_idx_sid ON msgs (sid);
---CREATE INDEX msgrcpt_idx_rid ON msgrcpt (rid);
---CREATE INDEX msgrcpt_idx_mail_id ON msgrcpt (mail_id);
-
--- delete unreferenced e-mail addresses
-DELETE FROM maddr
- WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id)
- AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);
-
--- (optional) optimize tables once in a while
---OPTIMIZE TABLE msgs, msgrcpt, quarantine, maddr;
diff --git a/README_FILES/README.sql-mysql b/README_FILES/README.sql-mysql
new file mode 100644
--- /dev/null
+++ b/README_FILES/README.sql-mysql
@@ -0,0 +1,461 @@
+USING SQL FOR LOOKUPS, LOG/REPORTING AND QUARANTINE
+===================================================
+
+This text only describes SQL specifics for a MySQL database
+and provides a schema.
+
+For general aspects of lookups, please see README.lookups.
+For general SQL notes and further examples please see README.sql.
+For PostgreSQL-specific notes and schema please see README.sql-pg
+(which in most respects also applies to a SQLite database).
+
+SERIAL can be used instead of INT UNSIGNED NOT NULL AUTO_INCREMENT
+with databases which do not recognize AUTO_INCREMENT;
+The attribute SERIAL was introduced with MySQL 4.1.0, but it
+implicitly creates an additional UNIQUE index, which is redundant.
+
+Instead of declaring a time_iso field in table msgs as a string:
+ time_iso char(16) NOT NULL,
+one may want to declare is as:
+ time_iso TIMESTAMP NOT NULL DEFAULT 0,
+in which case $timestamp_fmt_mysql *MUST* be set to 1 in amavisd.conf
+to avoid MySQL inability to accept ISO 8601 timestamps with zone Z
+and ISO date/time delimiter T; failing to set $timestamp_fmt_mysql
+makes MySQL store zero time on INSERT and write current local time
+on UPDATE if auto-update is allowed, which is different from the
+intended mail timestamp (localtime vs. UTC, off by seconds)
+
+Field quarantine.mail_text should be of data type 'blob' and not 'text'
+as suggested in earlier documentation; this is to prevent it from being
+unjustifiably associated with a character set, and to be able to
+store any byte value; to convert existing field from type 'text'
+to type 'blob' the following clause may be used:
+ ALTER TABLE quarantine CHANGE mail_text mail_text blob;
+
+
+-- local users
+CREATE TABLE users (
+ id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY, -- unique id
+ priority integer NOT NULL DEFAULT '7', -- sort field, 0 is low prior.
+ policy_id integer unsigned NOT NULL DEFAULT '1', -- JOINs with policy.id
+ email varchar(255) NOT NULL UNIQUE,
+ fullname varchar(255) DEFAULT NULL, -- not used by amavisd-new
+ local char(1) -- Y/N (optional field, see note further down)
+);
+
+-- any e-mail address (non- rfc2822-quoted), external or local,
+-- used as senders in wblist
+CREATE TABLE mailaddr (
+ id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,
+ priority integer NOT NULL DEFAULT '7', -- 0 is low priority
+ email varchar(255) NOT NULL UNIQUE
+);
+
+-- per-recipient whitelist and/or blacklist,
+-- puts sender and recipient in relation wb (white or blacklisted sender)
+CREATE TABLE wblist (
+ rid integer unsigned NOT NULL, -- recipient: users.id
+ sid integer unsigned NOT NULL, -- sender: mailaddr.id
+ wb varchar(10) NOT NULL, -- W or Y / B or N / space=neutral / score
+ PRIMARY KEY (rid,sid)
+);
+
+CREATE TABLE policy (
+ id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,
+ -- 'id' this is the _only_ required field
+ policy_name varchar(32), -- not used by amavisd-new, a comment
+
+ virus_lover char(1) default NULL, -- Y/N
+ spam_lover char(1) default NULL, -- Y/N
+ banned_files_lover char(1) default NULL, -- Y/N
+ bad_header_lover char(1) default NULL, -- Y/N
+
+ bypass_virus_checks char(1) default NULL, -- Y/N
+ bypass_spam_checks char(1) default NULL, -- Y/N
+ bypass_banned_checks char(1) default NULL, -- Y/N
+ bypass_header_checks char(1) default NULL, -- Y/N
+
+ spam_modifies_subj char(1) default NULL, -- Y/N
+
+ virus_quarantine_to varchar(64) default NULL,
+ spam_quarantine_to varchar(64) default NULL,
+ banned_quarantine_to varchar(64) default NULL,
+ bad_header_quarantine_to varchar(64) default NULL,
+ clean_quarantine_to varchar(64) default NULL,
+ other_quarantine_to varchar(64) default NULL,
+
+ spam_tag_level float default NULL, -- higher score inserts spam info headers
+ spam_tag2_level float default NULL, -- inserts 'declared spam' header fields
+ spam_kill_level float default NULL, -- higher score triggers evasive actions
+ -- e.g. reject/drop, quarantine, ...
+ -- (subject to final_spam_destiny setting)
+ spam_dsn_cutoff_level float default NULL,
+ spam_quarantine_cutoff_level float default NULL,
+
+ addr_extension_virus varchar(64) default NULL,
+ addr_extension_spam varchar(64) default NULL,
+ addr_extension_banned varchar(64) default NULL,
+ addr_extension_bad_header varchar(64) default NULL,
+
+ warnvirusrecip char(1) default NULL, -- Y/N
+ warnbannedrecip char(1) default NULL, -- Y/N
+ warnbadhrecip char(1) default NULL, -- Y/N
+ newvirus_admin varchar(64) default NULL,
+ virus_admin varchar(64) default NULL,
+ banned_admin varchar(64) default NULL,
+ bad_header_admin varchar(64) default NULL,
+ spam_admin varchar(64) default NULL,
+ spam_subject_tag varchar(64) default NULL,
+ spam_subject_tag2 varchar(64) default NULL,
+ message_size_limit integer default NULL, -- max size in bytes, 0 disable
+ banned_rulenames varchar(64) default NULL -- comma-separated list of ...
+ -- names mapped through %banned_rules to actual banned_filename tables
+);
+
+
+
+-- R/W part of the dataset (optional)
+-- May reside in the same or in a separate database as lookups database;
+-- REQUIRES SUPPORT FOR TRANSACTIONS; specified in @storage_sql_dsn
+--
+-- MySQL note ( http://dev.mysql.com/doc/mysql/en/storage-engines.html ):
+-- ENGINE is the preferred term, but cannot be used before MySQL 4.0.18.
+-- TYPE is available beginning with MySQL 3.23.0, the first version of
+-- MySQL for which multiple storage engines were available. If you omit
+-- the ENGINE or TYPE option, the default storage engine is used.
+-- By default this is MyISAM.
+--
+-- Please create additional indexes on keys when needed, or drop suggested
+-- ones as appropriate to optimize queries needed by a management application.
+-- See your database documentation for further optimization hints. With MySQL
+-- see Chapter 15 of the reference manual. For example the chapter 15.17 says:
+-- InnoDB does not keep an internal count of rows in a table. To process a
+-- SELECT COUNT(*) FROM T statement, InnoDB must scan an index of the table,
+-- which takes some time if the index is not entirely in the buffer pool.
+--
+-- Wayne Smith adds: When using MySQL with InnoDB one might want to
+-- increase buffer size for both pool and log, and might also want
+-- to change flush settings for a little better performance. Example:
+-- innodb_buffer_pool_size = 384M
+-- innodb_log_buffer_size = 8M
+-- innodb_flush_log_at_trx_commit = 0
+-- The big performance increase is the first two, the third just helps
+-- with lowering disk activity.
+
+
+-- provide unique id for each e-mail address, avoids storing copies
+CREATE TABLE maddr (
+ id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,
+ email varchar(255) NOT NULL UNIQUE, -- full mail address
+ domain varchar(255) NOT NULL -- only domain part of the email address
+ -- with subdomain fields in reverse
+) ENGINE=InnoDB;
+
+-- information pertaining to each processed message as a whole;
+-- NOTE: records with NULL msgs.content should be ignored by utilities,
+-- as such records correspond to messages just being processes, or were lost
+-- NOTE: instead of a character field time_iso, one might prefer:
+-- time_iso TIMESTAMP NOT NULL DEFAULT 0,
+-- but the following MUST then be set in amavisd.conf: $timestamp_fmt_mysql=1
+CREATE TABLE msgs (
+ mail_id varchar(12) NOT NULL PRIMARY KEY, -- long-term unique mail id
+ secret_id varchar(12) DEFAULT '', -- authorizes release of mail_id
+ am_id varchar(20) NOT NULL, -- id used in the log
+ time_num integer unsigned NOT NULL, -- rx_time: seconds since Unix epoch
+ time_iso char(16) NOT NULL, -- rx_time: ISO8601 UTC ascii time
+ sid integer unsigned NOT NULL, -- sender: maddr.id
+ policy varchar(255) DEFAULT '', -- policy bank path (like macro %p)
+ client_addr varchar(255) DEFAULT '', -- SMTP client IP address (IPv4 or v6)
+ size integer unsigned NOT NULL, -- message size in bytes
+ content char(1), -- content type: V/B/S/s/M/H/O/C:
+ -- virus/banned/spam(kill)/spammy(tag2)
+ -- /bad mime/bad header/oversized/clean
+ -- is NULL on partially processed mail
+ quar_type char(1), -- quarantined as: ' '/F/Z/B/Q/M/L
+ -- none/file/zipfile/bsmtp/sql/
+ -- /mailbox(smtp)/mailbox(lmtp)
+ quar_loc varchar(255) DEFAULT '', -- quarantine location (e.g. file)
+ dsn_sent char(1), -- was DSN sent? Y/N/q (q=quenched)
+ spam_level float, -- SA spam level (no boosts)
+ message_id varchar(255) DEFAULT '', -- mail Message-ID header field
+ from_addr varchar(255) DEFAULT '', -- mail From header field, UTF8
+ subject varchar(255) DEFAULT '', -- mail Subject header field, UTF8
+ host varchar(255) NOT NULL, -- hostname where amavisd is running
+ FOREIGN KEY (sid) REFERENCES maddr(id) ON DELETE RESTRICT
+) ENGINE=InnoDB;
+CREATE INDEX msgs_idx_sid ON msgs (sid);
+CREATE INDEX msgs_idx_mess_id ON msgs (message_id); -- useful with pen pals
+CREATE INDEX msgs_idx_time_num ON msgs (time_num);
+-- alternatively when purging based on time_iso (instead of msgs_idx_time_num):
+-- CREATE INDEX msgs_idx_time_iso ON msgs (time_iso);
+
+-- per-recipient information related to each processed message;
+-- NOTE: records in msgrcpt without corresponding msgs.mail_id record are
+-- orphaned and should be ignored and eventually deleted by external utilities
+CREATE TABLE msgrcpt (
+ mail_id varchar(12) NOT NULL, -- (must allow duplicates)
+ rid integer unsigned NOT NULL, -- recipient: maddr.id (dupl. allowed)
+ ds char(1) NOT NULL, -- delivery status: P/R/B/D/T
+ -- pass/reject/bounce/discard/tempfail
+ rs char(1) NOT NULL, -- release status: initialized to ' '
+ bl char(1) DEFAULT ' ', -- sender blacklisted by this recip
+ wl char(1) DEFAULT ' ', -- sender whitelisted by this recip
+ bspam_level float, -- spam level + per-recip boost
+ smtp_resp varchar(255) DEFAULT '', -- SMTP response given to MTA
+ FOREIGN KEY (rid) REFERENCES maddr(id) ON DELETE RESTRICT,
+ FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE
+) ENGINE=InnoDB;
+CREATE INDEX msgrcpt_idx_mail_id ON msgrcpt (mail_id);
+CREATE INDEX msgrcpt_idx_rid ON msgrcpt (rid);
+
+-- mail quarantine in SQL, enabled by $*_quarantine_method='sql:'
+-- NOTE: records in quarantine without corresponding msgs.mail_id record are
+-- orphaned and should be ignored and eventually deleted by external utilities
+CREATE TABLE quarantine (
+ mail_id varchar(12) NOT NULL, -- long-term unique mail id
+ chunk_ind integer unsigned NOT NULL, -- chunk number, starting with 1
+ mail_text blob NOT NULL, -- store mail as chunks of octets
+ PRIMARY KEY (mail_id,chunk_ind),
+ FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE
+) ENGINE=InnoDB;
+
+-- field msgrcpt.rs is primarily intended for use by quarantine management
+-- software; the value assigned by amavisd is a space;
+-- a short _preliminary_ list of possible values:
+-- 'V' => viewed (marked as read)
+-- 'R' => released (delivered) to this recipient
+-- 'p' => pending (a status given to messages when the admin received the
+-- request but not yet released; targeted to banned parts)
+-- 'D' => marked for deletion; a cleanup script may delete it
+
+
+-- =====================
+-- Example data follows:
+-- =====================
+INSERT INTO users VALUES ( 1, 9, 5, 'user1+foo at y.example.com','Name1 Surname1', 'Y');
+INSERT INTO users VALUES ( 2, 7, 5, 'user1 at y.example.com', 'Name1 Surname1', 'Y');
+INSERT INTO users VALUES ( 3, 7, 2, 'user2 at y.example.com', 'Name2 Surname2', 'Y');
+INSERT INTO users VALUES ( 4, 7, 7, 'user3 at z.example.com', 'Name3 Surname3', 'Y');
+INSERT INTO users VALUES ( 5, 7, 7, 'user4 at example.com', 'Name4 Surname4', 'Y');
+INSERT INTO users VALUES ( 6, 7, 1, 'user5 at example.com', 'Name5 Surname5', 'Y');
+INSERT INTO users VALUES ( 7, 5, 0, '@sub1.example.com', NULL, 'Y');
+INSERT INTO users VALUES ( 8, 5, 7, '@sub2.example.com', NULL, 'Y');
+INSERT INTO users VALUES ( 9, 5, 5, '@example.com', NULL, 'Y');
+INSERT INTO users VALUES (10, 3, 8, 'userA', 'NameA SurnameA anywhere', 'Y');
+INSERT INTO users VALUES (11, 3, 9, 'userB', 'NameB SurnameB', 'Y');
+INSERT INTO users VALUES (12, 3,10, 'userC', 'NameC SurnameC', 'Y');
+INSERT INTO users VALUES (13, 3,11, 'userD', 'NameD SurnameD', 'Y');
+INSERT INTO users VALUES (14, 3, 0, '@sub1.example.net', NULL, 'Y');
+INSERT INTO users VALUES (15, 3, 7, '@sub2.example.net', NULL, 'Y');
+INSERT INTO users VALUES (16, 3, 5, '@example.net', NULL, 'Y');
+INSERT INTO users VALUES (17, 7, 5, 'u1 at example.org', 'u1', 'Y');
+INSERT INTO users VALUES (18, 7, 6, 'u2 at example.org', 'u2', 'Y');
+INSERT INTO users VALUES (19, 7, 3, 'u3 at example.org', 'u3', 'Y');
+
+-- INSERT INTO users VALUES (20, 0, 5, '@.', NULL, 'N'); -- catchall
+
+INSERT INTO policy (id, policy_name,
+ virus_lover, spam_lover, banned_files_lover, bad_header_lover,
+ bypass_virus_checks, bypass_spam_checks,
+ bypass_banned_checks, bypass_header_checks, spam_modifies_subj,
+ spam_tag_level, spam_tag2_level, spam_kill_level) VALUES
+ (1, 'Non-paying', 'N','N','N','N', 'Y','Y','Y','N', 'Y', 3.0, 7, 10),
+ (2, 'Uncensored', 'Y','Y','Y','Y', 'N','N','N','N', 'N', 3.0, 999, 999),
+ (3, 'Wants all spam','N','Y','N','N', 'N','N','N','N', 'Y', 3.0, 999, 999),
+ (4, 'Wants viruses', 'Y','N','Y','Y', 'N','N','N','N', 'Y', 3.0, 6.9, 6.9),
+ (5, 'Normal', 'N','N','N','N', 'N','N','N','N', 'Y', 3.0, 6.9, 6.9),
+ (6, 'Trigger happy', 'N','N','N','N', 'N','N','N','N', 'Y', 3.0, 5, 5),
+ (7, 'Permissive', 'N','N','N','Y', 'N','N','N','N', 'Y', 3.0, 10, 20),
+ (8, '6.5/7.8', 'N','N','N','N', 'N','N','N','N', 'N', 3.0, 6.5, 7.8),
+ (9, 'userB', 'N','N','N','Y', 'N','N','N','N', 'Y', 3.0, 6.3, 6.3),
+ (10,'userC', 'N','N','N','N', 'N','N','N','N', 'N', 3.0, 6.0, 6.0),
+ (11,'userD', 'Y','N','Y','Y', 'N','N','N','N', 'N', 3.0, 7, 7);
+
+-- sender envelope addresses needed for white/blacklisting
+INSERT INTO mailaddr VALUES (1, 5, '@example.com');
+INSERT INTO mailaddr VALUES (2, 9, 'owner-postfix-users at postfix.org');
+INSERT INTO mailaddr VALUES (3, 9, 'amavis-user-admin at lists.sourceforge.net');
+INSERT INTO mailaddr VALUES (4, 9, 'makemoney at example.com');
+INSERT INTO mailaddr VALUES (5, 5, '@example.net');
+INSERT INTO mailaddr VALUES (6, 9, 'spamassassin-talk-admin at lists.sourceforge.net');
+INSERT INTO mailaddr VALUES (7, 9, 'spambayes-bounces at python.org');
+
+-- whitelist for user 14, i.e. default for recipients in domain sub1.example.net
+INSERT INTO wblist VALUES (14, 1, 'W');
+INSERT INTO wblist VALUES (14, 3, 'W');
+
+-- whitelist and blacklist for user 17, i.e. u1 at example.org
+INSERT INTO wblist VALUES (17, 2, 'W');
+INSERT INTO wblist VALUES (17, 3, 'W');
+INSERT INTO wblist VALUES (17, 6, 'W');
+INSERT INTO wblist VALUES (17, 7, 'W');
+INSERT INTO wblist VALUES (17, 5, 'B');
+INSERT INTO wblist VALUES (17, 4, 'B');
+
+-- $sql_select_policy setting in amavisd.conf tells amavisd
+-- how to fetch per-recipient policy settings.
+-- See comments there. Example:
+--
+-- SELECT *,users.id FROM users,policy
+-- WHERE (users.policy_id=policy.id) AND (users.email IN (%k))
+-- ORDER BY users.priority DESC;
+--
+-- $sql_select_white_black_list in amavisd.conf tells amavisd
+-- how to check sender in per-recipient whitelist/blacklist.
+-- See comments there. Example:
+--
+-- SELECT wb FROM wblist,mailaddr
+-- WHERE (wblist.rid=?) AND (wblist.sid=mailaddr.id) AND (mailaddr.email IN (%k))
+-- ORDER BY mailaddr.priority DESC;
+
+
+
+-- NOTE: the SELECT, INSERT and UPDATE clauses as used by the amavisd-new
+-- program are configurable through %sql_clause; see amavisd.conf-default
+
+
+
+Some examples of a query:
+
+-- mail from last two minutes:
+SELECT
+ UNIX_TIMESTAMP()-msgs.time_num AS age, SUBSTRING(policy,1,2) as pb,
+ content AS c, dsn_sent as dsn, ds, bspam_level AS level, size,
+ SUBSTRING(sender.email,1,18) AS s,
+ SUBSTRING(recip.email,1,18) AS r,
+ SUBSTRING(msgs.subject,1,10) AS subj
+ FROM msgs LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
+ LEFT JOIN maddr AS sender ON msgs.sid=sender.id
+ LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
+ WHERE content IS NOT NULL AND UNIX_TIMESTAMP()-msgs.time_num < 120
+ ORDER BY msgs.time_num DESC;
+
+-- clean messages ordered by count, grouped by domain:
+SELECT count(*) as cnt, avg(bspam_level), sender.domain
+ FROM msgs
+ LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
+ LEFT JOIN maddr AS sender ON msgs.sid=sender.id
+ LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
+ WHERE content='C'
+ GROUP BY sender.domain ORDER BY cnt DESC LIMIT 50;
+
+-- top spamy domains with >10 messages, sorted by spam average,
+-- grouped by domain:
+SELECT count(*) as cnt, avg(bspam_level) as spam_avg, sender.domain
+ FROM msgs
+ LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
+ LEFT JOIN maddr AS sender ON msgs.sid=sender.id
+ LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
+ WHERE bspam_level IS NOT NULL
+ GROUP BY sender.domain HAVING count(*) > 10
+ ORDER BY spam_avg DESC LIMIT 50;
+
+-- sender domains with >100 messages, sorted on sender.domain:
+SELECT count(*) as cnt, avg(bspam_level) as spam_avg, sender.domain
+ FROM msgs
+ LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
+ LEFT JOIN maddr AS sender ON msgs.sid=sender.id
+ LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
+ GROUP BY sender.domain HAVING count(*) > 100
+ ORDER BY sender.domain DESC LIMIT 100;
+
+
+
+
+Upgrading from pre 2.4.0 amavisd-new SQL schema to the 2.4.0 schema requires
+adding column 'quar_loc' to table msgs, and creating FOREIGN KEY constraint
+to facilitate deletion of expired records.
+
+The following clauses should be executed for upgrading pre-2.4.0 amavisd-new
+SQL schema to the 2.4.0 schema:
+
+-- mandatory change:
+ ALTER TABLE msgs ADD quar_loc varchar(255) DEFAULT '';
+
+-- optional but highly recommended:
+ ALTER TABLE quarantine
+ ADD FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE;
+ ALTER TABLE msgrcpt
+ ADD FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE;
+
+-- the following two ALTERs are not essential; if data type of maddr.id is
+-- incompatible with msgs.sid and msgs.rid (e.g. BIGINT vs. INT) and MySQL
+-- complains, don't bother to apply the constraint:
+ ALTER TABLE msgs
+ ADD FOREIGN KEY (sid) REFERENCES maddr(id) ON DELETE RESTRICT;
+ ALTER TABLE msgrcpt
+ ADD FOREIGN KEY (rid) REFERENCES maddr(id) ON DELETE RESTRICT;
+
+
+BRIEF EXAMPLE of a log/report/quarantine database housekeeping
+==============================================================
+
+DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP() - 14*24*60*60;
+DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP() - 60*60 AND content IS NULL;
+DELETE FROM maddr
+ WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id)
+ AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);
+
+
+BRIEF EQUIVALENT EXAMPLE based on time_iso if its data type is TIMESTAMPS
+=========================================================================
+(don't forget to set: $timestamp_fmt_mysql=1 in amavisd.conf)
+
+DELETE FROM msgs WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 14 day;
+DELETE FROM msgs WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 1 hour
+ AND content IS NULL;
+DELETE FROM maddr
+ WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id)
+ AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);
+
+
+COMMENTED LONGER EXAMPLE of a log/report/quarantine database housekeeping
+=========================================================================
+
+-- discarding indexes makes deletion faster; if we expect a large proportion
+-- of records to be deleted it may be quicker to discard index, do deletions,
+-- and re-create index; for daily maintenance this does not pay off
+--DROP INDEX msgs_idx_sid ON msgs;
+--DROP INDEX msgrcpt_idx_rid ON msgrcpt;
+--DROP INDEX msgrcpt_idx_mail_id ON msgrcpt;
+
+-- delete old msgs records based on timestamps only (for time_iso see next),
+-- and delete leftover msgs records from aborted mail checking operations
+DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP()-14*24*60*60;
+DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP()-60*60 AND content IS NULL;
+
+-- provided the time_iso field was created as type TIMESTAMP DEFAULT 0,
+-- instead of purging based on numerical Unix timestamp as above, one may
+-- select records based on ISO 8601 UTC timestamps.
+--DELETE FROM msgs WHERE time_iso < now() - INTERVAL '14 days';
+--DELETE FROM msgs WHERE time_iso < now() - INTERVAL '1 h' AND content IS NULL;
+and is also possible with MySQL, using slightly different format:
+--DELETE FROM msgs
+-- WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 14 day;
+--DELETE FROM msgs
+-- WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 1 hour AND content IS NULL;
+
+-- optionally certain content types may be given shorter lifetime
+--DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP()-7*24*60*60
+-- AND (content='V' OR (content='S' AND spam_level>20));
+
+-- (optional) just in case the ON DELETE CASCADE did not do its job, we may
+-- explicitly delete orphaned records (with no corresponding msgs entry);
+-- if ON DELETE CASCADE did work, there should be no deletions at this step
+DELETE FROM quarantine
+ WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=quarantine.mail_id);
+DELETE FROM msgrcpt
+ WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=msgrcpt.mail_id);
+
+-- re-create indexes (if they were removed in the first step):
+--CREATE INDEX msgs_idx_sid ON msgs (sid);
+--CREATE INDEX msgrcpt_idx_rid ON msgrcpt (rid);
+--CREATE INDEX msgrcpt_idx_mail_id ON msgrcpt (mail_id);
+
+-- delete unreferenced e-mail addresses
+DELETE FROM maddr
+ WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id)
+ AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);
+
+-- (optional) optimize tables once in a while
+--OPTIMIZE TABLE msgs, msgrcpt, quarantine, maddr;
diff --git a/README_FILES/README.sql-pg b/README_FILES/README.sql-pg
new file mode 100644
--- /dev/null
+++ b/README_FILES/README.sql-pg
@@ -0,0 +1,362 @@
+USING SQL FOR LOOKUPS, LOG/REPORTING AND QUARANTINE
+===================================================
+
+This text only describes SQL specifics for a PostgreSQL database
+and provides a schema. In most respects it applies to a SQLite database.
+
+For general aspects of lookups, please see README.lookups.
+For general SQL notes and further examples please see README.sql.
+For MySQL-specific notes and schema please see README.sql-mysql.
+
+Upgrade note: field quarantine.mail_text should be of data type 'bytea'
+and not 'text' as suggested in earlier documentation; this is to prevent
+it from being unjustifiably associated with a character set, and to be
+able to store any byte value; to convert existing field from type 'text'
+to type 'bytea' the following clause may be used:
+ ALTER TABLE quarantine ALTER mail_text TYPE bytea
+ USING decode(replace(mail_text,'\\','\\\\'),'escape');
+
+Version of Perl module DBD::Pg 1.48 or higher should be used;
+
+Short installation notes for PostgreSQL 8.2 are available at:
+ http://www.postgresql.org/docs/8.2/interactive/install-short.html
+
+In short: run: 'initdb -D ...' as user postgres, then edit pg_hba.conf
+providing restricted access to database, create users and create databases.
+
+Something like the following may be placed into pg_hba.conf :
+
+# TYPE DATABASE USER CIDR-ADDRESS METHOD
+#
+# amavis lookups:
+local mail_prefs vscan md5
+host mail_prefs vscan 127.0.0.1/32 md5
+host mail_prefs vscan ::1/128 md5
+#
+# amavis logging and pen pals:
+local mail_log vscan md5
+host mail_log vscan 127.0.0.1/32 md5
+host mail_log vscan ::1/128 md5
+#
+# spamassassin Bayes and AWL databases:
+local mail_bayes vscan md5
+host mail_bayes vscan 127.0.0.1/32 md5
+host mail_bayes vscan ::1/128 md5
+local mail_awl vscan md5
+host mail_awl vscan 127.0.0.1/32 md5
+host mail_awl vscan ::1/128 md5
+
+
+Create a SQL username (role) for use by amavisd, e.g. vscan:
+ $ createuser -U pgsql -S -D -R -P -e vscan
+
+Create databases for amavisd:
+ $ createdb -U pgsql mail_prefs
+ $ createdb -U pgsql mail_log
+
+and optionally databases for SpamAssassin:
+ $ createdb -U pgsql mail_bayes
+ $ createdb -U pgsql mail_awl
+
+The provided schema can be cut/pasted or fed directly into the client program
+to create a database. The '--' introduces comments according to SQL specs.
+
+Populate databases using the schema below:
+ $ psql -U vscan mail_prefs <...
+ $ psql -U vscan mail_log <...
+(for SA database schema see its documentation: sql/README*)
+
+
+Something like the following can be placed into amavisd.conf
+(supplying correct passwords):
+
+ @lookup_sql_dsn =
+ ([ 'DBI:Pg:database=mail_prefs', 'vscan', 'LK40.gtklkKK' ]);
+
+ @storage_sql_dsn =
+ ([ 'DBI:Pg:database=mail_log', 'vscan', 'LK40.gtklkKK' ]);
+
+Equivalent settings for AWL and Bayes databases belong to a
+SA configuration file local.cf, according to SpamAssassin documentation.
+Amavisd and SA need not use the same usernames or passwords, nor do they
+need to reside on the same SQL server.
+
+
+SQLite notes:
+ - use INTEGER PRIMARY KEY AUTOINCREMENT instead of SERIAL;
+ - SQLite is well suited for lookups database, but is not appropriate
+ for @storage_sql_dsn due to coarse lock granularity;
+
+
+-- local users
+CREATE TABLE users (
+ id serial PRIMARY KEY, -- unique id
+ priority integer NOT NULL DEFAULT '7', -- sort field, 0 is low prior.
+ policy_id integer NOT NULL DEFAULT '1' CHECK (policy_id >= 0),
+ -- JOINs with policy.id
+ email varchar(255) NOT NULL UNIQUE, -- email address, non-rfc2822-quoted
+ fullname varchar(255) DEFAULT NULL, -- not used by amavisd-new
+ local char(1) -- Y/N (optional field, see note further down)
+);
+
+-- any e-mail address (non- rfc2822-quoted), external or local,
+-- used as senders in wblist
+CREATE TABLE mailaddr (
+ id serial PRIMARY KEY,
+ priority integer NOT NULL DEFAULT '7', -- 0 is low priority
+ email varchar(255) NOT NULL UNIQUE
+);
+
+-- per-recipient whitelist and/or blacklist,
+-- puts sender and recipient in relation wb (white or blacklisted sender)
+CREATE TABLE wblist (
+ rid integer NOT NULL CHECK (rid >= 0), -- recipient: users.id
+ sid integer NOT NULL CHECK (sid >= 0), -- sender: mailaddr.id
+ wb varchar(10) NOT NULL, -- W or Y / B or N / space=neutral / score
+ PRIMARY KEY (rid,sid)
+);
+
+CREATE TABLE policy (
+ id serial PRIMARY KEY, -- 'id' this is the _only_ required field
+ policy_name varchar(32), -- not used by amavisd-new, a comment
+
+ virus_lover char(1) default NULL, -- Y/N
+ spam_lover char(1) default NULL, -- Y/N
+ banned_files_lover char(1) default NULL, -- Y/N
+ bad_header_lover char(1) default NULL, -- Y/N
+
+ bypass_virus_checks char(1) default NULL, -- Y/N
+ bypass_spam_checks char(1) default NULL, -- Y/N
+ bypass_banned_checks char(1) default NULL, -- Y/N
+ bypass_header_checks char(1) default NULL, -- Y/N
+
+ spam_modifies_subj char(1) default NULL, -- Y/N
+
+ virus_quarantine_to varchar(64) default NULL,
+ spam_quarantine_to varchar(64) default NULL,
+ banned_quarantine_to varchar(64) default NULL,
+ bad_header_quarantine_to varchar(64) default NULL,
+ clean_quarantine_to varchar(64) default NULL,
+ other_quarantine_to varchar(64) default NULL,
+
+ spam_tag_level real default NULL, -- higher score inserts spam info headers
+ spam_tag2_level real default NULL, -- inserts 'declared spam' header fields
+ spam_kill_level real default NULL, -- higher score triggers evasive actions
+ -- e.g. reject/drop, quarantine, ...
+ -- (subject to final_spam_destiny setting)
+ spam_dsn_cutoff_level real default NULL,
+ spam_quarantine_cutoff_level real default NULL,
+
+ addr_extension_virus varchar(64) default NULL,
+ addr_extension_spam varchar(64) default NULL,
+ addr_extension_banned varchar(64) default NULL,
+ addr_extension_bad_header varchar(64) default NULL,
+
+ warnvirusrecip char(1) default NULL, -- Y/N
+ warnbannedrecip char(1) default NULL, -- Y/N
+ warnbadhrecip char(1) default NULL, -- Y/N
+ newvirus_admin varchar(64) default NULL,
+ virus_admin varchar(64) default NULL,
+ banned_admin varchar(64) default NULL,
+ bad_header_admin varchar(64) default NULL,
+ spam_admin varchar(64) default NULL,
+ spam_subject_tag varchar(64) default NULL,
+ spam_subject_tag2 varchar(64) default NULL,
+ message_size_limit integer default NULL, -- max size in bytes, 0 disable
+ banned_rulenames varchar(64) default NULL -- comma-separated list of ...
+ -- names mapped through %banned_rules to actual banned_filename tables
+);
+
+
+-- R/W part of the dataset (optional)
+-- May reside in the same or in a separate database as lookups database;
+-- REQUIRES SUPPORT FOR TRANSACTIONS; specified in @storage_sql_dsn
+--
+-- Please create additional indexes on keys when needed, or drop suggested
+-- ones as appropriate to optimize queries needed by a management application.
+-- See your database documentation for further optimization hints.
+
+-- provide unique id for each e-mail address, avoids storing copies
+CREATE TABLE maddr (
+ id serial PRIMARY KEY,
+ email varchar(255) NOT NULL UNIQUE, -- full e-mail address
+ domain varchar(255) NOT NULL -- only domain part of the email address
+ -- with subdomain fields in reverse
+);
+
+-- information pertaining to each processed message as a whole;
+-- NOTE: records with NULL msgs.content should be ignored by utilities,
+-- as such records correspond to messages just being processes, or were lost
+CREATE TABLE msgs (
+ mail_id varchar(12) NOT NULL PRIMARY KEY, -- long-term unique mail id
+ secret_id varchar(12) DEFAULT '', -- authorizes release of mail_id
+ am_id varchar(20) NOT NULL, -- id used in the log
+ time_num integer NOT NULL CHECK (time_num >= 0),
+ -- rx_time: seconds since Unix epoch
+ time_iso timestamp WITH TIME ZONE NOT NULL,-- rx_time: ISO8601 UTC ascii time
+ sid integer NOT NULL CHECK (sid >= 0), -- sender: maddr.id
+ policy varchar(255) DEFAULT '', -- policy bank path (like macro %p)
+ client_addr varchar(255) DEFAULT '', -- SMTP client IP address (IPv4 or v6)
+ size integer NOT NULL CHECK (size >= 0), -- message size in bytes
+ content char(1), -- content type: V/B/S/s/M/H/O/C:
+ -- virus/banned/spam(kill)/spammy(tag2)
+ -- /bad mime/bad header/oversized/clean
+ -- is NULL on partially processed mail
+ quar_type char(1), -- quarantined as: ' '/F/Z/B/Q/M/L
+ -- none/file/zipfile/bsmtp/sql/
+ -- /mailbox(smtp)/mailbox(lmtp)
+ quar_loc varchar(255) DEFAULT '', -- quarantine location (e.g. file)
+ dsn_sent char(1), -- was DSN sent? Y/N/q (q=quenched)
+ spam_level real, -- SA spam level (no boosts)
+ message_id varchar(255) DEFAULT '', -- mail Message-ID header field
+ from_addr varchar(255) DEFAULT '', -- mail From header field, UTF8
+ subject varchar(255) DEFAULT '', -- mail Subject header field, UTF8
+ host varchar(255) NOT NULL, -- hostname where amavisd is running
+ FOREIGN KEY (sid) REFERENCES maddr(id) ON DELETE RESTRICT
+);
+CREATE INDEX msgs_idx_sid ON msgs (sid);
+CREATE INDEX msgs_idx_mess_id ON msgs (message_id); -- useful with pen pals
+CREATE INDEX msgs_idx_time_iso ON msgs (time_iso);
+CREATE INDEX msgs_idx_time_num ON msgs (time_num); -- optional
+
+-- per-recipient information related to each processed message;
+-- NOTE: records in msgrcpt without corresponding msgs.mail_id record are
+-- orphaned and should be ignored and eventually deleted by external utilities
+CREATE TABLE msgrcpt (
+ mail_id varchar(12) NOT NULL, -- (must allow duplicates)
+ rid integer NOT NULL CHECK (rid >= 0),
+ -- recipient: maddr.id (duplicates allowed)
+ ds char(1) NOT NULL, -- delivery status: P/R/B/D/T
+ -- pass/reject/bounce/discard/tempfail
+ rs char(1) NOT NULL, -- release status: initialized to ' '
+ bl char(1) DEFAULT ' ', -- sender blacklisted by this recip
+ wl char(1) DEFAULT ' ', -- sender whitelisted by this recip
+ bspam_level real, -- spam level + per-recip boost
+ smtp_resp varchar(255) DEFAULT '', -- SMTP response given to MTA
+ FOREIGN KEY (rid) REFERENCES maddr(id) ON DELETE RESTRICT,
+ FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE
+);
+CREATE INDEX msgrcpt_idx_mail_id ON msgrcpt (mail_id);
+CREATE INDEX msgrcpt_idx_rid ON msgrcpt (rid);
+
+-- mail quarantine in SQL, enabled by $*_quarantine_method='sql:'
+-- NOTE: records in quarantine without corresponding msgs.mail_id record are
+-- orphaned and should be ignored and eventually deleted by external utilities
+CREATE TABLE quarantine (
+ mail_id varchar(12) NOT NULL, -- long-term unique mail id
+ chunk_ind integer NOT NULL CHECK (chunk_ind >= 0), -- chunk number, 1..
+ mail_text bytea NOT NULL, -- store mail as chunks of octects
+ PRIMARY KEY (mail_id,chunk_ind),
+ FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE
+);
+
+-- field msgrcpt.rs is primarily intended for use by quarantine management
+-- software; the value assigned by amavisd is a space;
+-- a short _preliminary_ list of possible values:
+-- 'V' => viewed (marked as read)
+-- 'R' => released (delivered) to this recipient
+-- 'p' => pending (a status given to messages when the admin received the
+-- request but not yet released; targeted to banned parts)
+-- 'D' => marked for deletion; a cleanup script may delete it
+
+
+Some examples of a query:
+
+-- mail from last two minutes:
+SELECT
+ now()-time_iso AS age, SUBSTRING(policy,1,2) as pb,
+ content AS c, dsn_sent as dsn, ds, bspam_level AS level, size,
+ SUBSTRING(sender.email,1,18) AS s,
+ SUBSTRING(recip.email,1,18) AS r,
+ SUBSTRING(msgs.subject,1,10) AS subj
+ FROM msgs LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
+ LEFT JOIN maddr AS sender ON msgs.sid=sender.id
+ LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
+ WHERE content IS NOT NULL AND now() - time_iso < INTERVAL '2 minutes'
+ ORDER BY msgs.time_num DESC;
+
+-- clean messages ordered by count, grouped by domain:
+SELECT count(*) as cnt, avg(bspam_level), sender.domain
+ FROM msgs
+ LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
+ LEFT JOIN maddr AS sender ON msgs.sid=sender.id
+ LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
+ WHERE content='C'
+ GROUP BY sender.domain ORDER BY cnt DESC LIMIT 50;
+
+-- top spamy domains with >10 messages, sorted by spam average,
+-- grouped by domain:
+SELECT count(*) as cnt, avg(bspam_level) as spam_avg, sender.domain
+ FROM msgs
+ LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
+ LEFT JOIN maddr AS sender ON msgs.sid=sender.id
+ LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
+ WHERE bspam_level IS NOT NULL
+ GROUP BY sender.domain HAVING count(*) > 10
+ ORDER BY spam_avg DESC LIMIT 50;
+
+-- sender domains with >100 messages, sorted on sender.domain:
+SELECT count(*) as cnt, avg(bspam_level) as spam_avg, sender.domain
+ FROM msgs
+ LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id
+ LEFT JOIN maddr AS sender ON msgs.sid=sender.id
+ LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id
+ GROUP BY sender.domain HAVING count(*) > 100
+ ORDER BY sender.domain DESC LIMIT 100;
+
+
+
+
+Upgrading from pre 2.4.0 amavisd-new SQL schema to the 2.4.0 schema requires
+adding column 'quar_loc' to table msgs, and creating FOREIGN KEY constraint
+to facilitate deletion of expired records.
+
+The following clauses should be executed for upgrading pre-2.4.0 amavisd-new
+SQL schema to the 2.4.0 schema:
+
+-- mandatory change:
+ ALTER TABLE msgs ADD quar_loc varchar(255) DEFAULT '';
+
+-- optional but highly recommended:
+ ALTER TABLE quarantine
+ ADD FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE;
+ ALTER TABLE msgrcpt
+ ADD FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE;
+
+-- the following two ALTERs are not essential; if data type of maddr.id is
+-- incompatible with msgs.sid and msgs.rid (e.g. BIGINT vs. INT) and MySQL
+-- complains, don't bother to apply the constraint:
+ ALTER TABLE msgs
+ ADD FOREIGN KEY (sid) REFERENCES maddr(id) ON DELETE RESTRICT;
+ ALTER TABLE msgrcpt
+ ADD FOREIGN KEY (rid) REFERENCES maddr(id) ON DELETE RESTRICT;
+
+
+
+BRIEF EXAMPLE of a log/report/quarantine database housekeeping
+==============================================================
+
+DELETE FROM msgs WHERE time_iso < now() - INTERVAL '3 weeks';
+DELETE FROM msgs WHERE time_iso < now() - INTERVAL '1 h' AND content IS NULL;
+
+DELETE FROM maddr
+ WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id)
+ AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);
+
+On more recent testings, the following 'DELETE FROM maddr' seems
+to be faster from the one above by a factor of 1.5 to 2, and is
+functionally equivalent:
+
+DELETE FROM maddr WHERE id IN (
+ SELECT id FROM maddr LEFT JOIN (
+ SELECT sid AS id, 1 AS f FROM msgs UNION ALL
+ SELECT rid AS id, 1 AS f FROM msgrcpt
+ ) AS u USING(id) WHERE u.f IS NULL);
+
+Check also a thread 'Faster purging of SQL logging database'
+(2007-06) on the amavis-user mailing list, archived at:
+ http://marc.info/?t=118190428300003
+
+-- optionally certain content types may be given shorter lifetime:
+DELETE FROM msgs WHERE time_iso < now() - INTERVAL '1 week'
+ AND (content='V' OR (content='S' AND spam_level > 20));
diff --git a/README_FILES/amavisd-new-docs.html b/README_FILES/amavisd-new-docs.html
--- a/README_FILES/amavisd-new-docs.html
+++ b/README_FILES/amavisd-new-docs.html
@@ -39,7 +39,7 @@ senders regarding spam -- @score_sender_
<li><a href="#pbanks">policy banks</a></li>
<li><a href="#pbanks-ex">putting policy banks to good use -- examples</a></li>
<li><a href="#max_requests">$max_requests</a></li>
-<li><a href="#dkim">setting up DKIM and DomainKeys mail signing
+<li><a href="#dkim">setting up DKIM (and DomainKeys) mail signing
and verification</a></li>
</ul>
@@ -145,7 +145,7 @@ decides the outcome:</p>
<ol>
<li>a virus is detected: mail is considered infected;</li>
<li>contains banned name or type: mail is considered banned;</li>
-<li>spam level is above kill level for at least one recipient
+<li>spam level is above kill level for at least one recipient,
or a sender is blacklisted: mail is considered spam;</li>
<li>bad (invalid) headers: mail is considered as having a bad header.</li>
</ol>
@@ -185,7 +185,8 @@ or D_REJECT), the sender (non)delivery n
or D_REJECT), the sender (non)delivery notification is now prepared
in case of D_BOUNCE, and MTA receives a 2xx status (success); or in
case of D_REJECT the MTA receives a 5xx (reject) status and preparing
-sender notifications is thus delegated to MTA (not recommended).</p>
+sender notifications is thus delegated to MTA (not recommended in
+post-queue or dual-MTA content filtering setup).</p>
<p>Even in cases of mail non-delivery when a (non-)delivery status
notification (DSN) for the sender should have been prepared and sent,
@@ -195,8 +196,8 @@ effectively lost as far as the sender an
<ul>
<li>when $final_*_destiny=D_DISCARD;</li>
-<li>when mail is infected and the detected virus name matches
- the @viruses_that_fake_sender_maps;</li>
+<li>when mail is infected and the detected virus name matches the
+ @viruses_that_fake_sender_maps (unconditionally true by default);</li>
<li>when spam score exceeds level determined by @spam_dsn_cutoff_level_maps
for all recipients;</li>
<li>when mail is coming from a mailing list, as determined by
@@ -211,13 +212,13 @@ a spam score (spam level, hits), which i
a spam score (spam level, hits), which is a numeric representation of
spaminess. The higher the number, the more spamy the message is considered.
Small numbers near zero or negative indicate a clean message, colloquially
-called ham. The spam score is a characteristic of the whole message,
+called ham. Spam score is a characteristic of the whole message,
and does not depend on recipient preferences. SpamAssassin is called
only once for each message regardless of the number of recipients.</p>
<p>To determine further course of action, amavisd-new compares the spam score
to three numeric values: tag level, tag2 level and kill level. These values
-may be different for each recipient, and the further actions may be different
+may be different for each recipient, and further actions may be different
for each recipient. If necessary, the mail forwarding is split into more
than one transaction to cater for different recipient preferences.</p>
@@ -225,7 +226,8 @@ than one transaction to cater for differ
<dt>tag level</dt>
<dd>if spam score is at or above tag level, spam-related header fields
(X-Spam-Status, X-Spam-Level) are inserted for local recipients;
- undef is interpreted as lower than any spam score;</dd>
+ undefined (unknown) spam score is interpreted as lower than any
+ spam score;</dd>
<dt>tag2 level</dt>
<dd>if spam score is at or above tag2 level, spam-related header fields
(X-Spam-Status, X-Spam-Level, X-Spam-Flag and X-Spam-Report)
@@ -249,11 +251,11 @@ MUA later does with the mail).</p>
<li>mail gets quarantined (unless disabled)</li>
<li>spam administrator gets a notification (unless disabled)</li>
<li>ContentSpamMsgs counter is incremented</li>
-<li>spam defanging is done (unless disabled)</li>
<li>sender gets a notification if warnspamsender
is true and $final_spam_destiny is D_PASS</li>
<li>if message is not delivered, sender gets a nondelivery
- notification (suppressed under certain conditions).</li>
+ notification (suppressed under certain conditions)</li>
+<li>the main log entry says: Passed/Blocked SPAM.</li>
</ul>
<p>On the other hand the tag2 level just adds some mark to the passed
@@ -264,16 +266,17 @@ to act on or not. Specifically:</p>
<li>Subject header field is modified (unless disabled)</li>
<li>X-Spam-Flag and X-Spam-Status header field get a Yes</li>
<li>address extension for spam gets tacked on the recipient address</li>
-<li>and (perhaps inconsistently with the rest) the mail log entry says
- 'Passed SPAM' instead of 'Passed CLEAN'.</li>
+<li>spam defanging is done (unless disabled)</li>
+<li>the main log entry says: Passed/Blocked SPAMMY.</li>
</ul>
-<p>If a recipient (or its MUA) decides to discard the mail based on
-tag2 marking, there is no way to retrieve it later from a quarantine,
-the sender is never notified, spam administrator is never notified.
-As far as the MTA and amavisd-new are concerned, the message was
-successfully delivered. Whatever MUA does with the mail is entirely
-the responsibility and jurisdiction of the recipient.</p>
+<p>For mail below kill level, if a recipient (or his MUA) decides
+to discard a message based on tag2 marking, there is no way to
+retrieve it later from a quarantine, the sender is never notified,
+spam administrator is never notified. As far as the MTA and amavisd-new
+are concerned, the message was successfully delivered. Whatever MUA
+does with the mail is entirely the responsibility and jurisdiction
+of the recipient and his LDA and MUA.</p>
<h2><a name="quarantine">Quarantine</a></h2>
@@ -303,18 +306,25 @@ variables. A nonempty string should foll
<ul>
<li><tt>local:</tt><i>filename-template</i></li>
<li><tt>bsmtp:</tt><i>filename-template</i></li>
+<li><tt>smtp:[</tt><i>IPv4-or-IPv6-address-or-hostname</i><tt>]:</tt><i>port</i></li>
<li><tt>smtp:</tt><i>hostname</i><tt>:</tt><i>port</i></li>
-<li><tt>smtp:[</tt><i>ip-address-or-hostname</i><tt>]:</tt><i>port</i></li>
+<li><tt>smtp:</tt><i>/path/to/a/unix/socket</i></li>
+<li><tt>lmtp:[</tt><i>IPv4-or-IPv6-address-or-hostname</i><tt>]:</tt><i>port</i></li>
+<li><tt>lmtp:</tt><i>hostname</i><tt>:</tt><i>port</i></li>
+<li><tt>lmtp:</tt><i>/path/to/a/unix/socket</i></li>
<li><tt>pipe:</tt><i>argv=command args...</i></li>
<li><tt>sql:</tt><i>anything</i></li>
</ul>
<p>The <tt>local:</tt>, <tt>bsmtp:</tt> and <tt>sql:</tt> methods are the
-usual methods for quarantining. The <tt>smtp:</tt> method is only useful
-for quarantining if quarantine location is some dedicated mailbox instead
-of a local file or directory. The <tt>smtp:</tt> and <tt>pipe:</tt> methods
-are more often used for forwarding and notifications.</p>
+usual methods for quarantining. The <tt>smtp:</tt> or <tt>lmtp:</tt> methods
+are only useful for quarantining if quarantine location is some dedicated
+mailbox instead of a local file or directory. The <tt>smtp:</tt>, <tt>lmtp:</tt>
+and <tt>pipe:</tt> methods are more often used for forwarding and notifications.
+The following features became available with version 2.5.0: the <tt>lmtp:</tt>
+method, support for IPv6, and specifying a Unix socket to a <tt>smtp:</tt> or
+<tt>lmtp:</tt> method.</p>
<p>Depending on the method specified (local/bsmtp/smtp/sql) a per-recipient
setting <i>*quarantine_to</i> adopts different semantics and syntax,
@@ -360,15 +370,23 @@ possibly modified by the configuration v
<td>anything</td>
<td>sent via SMTP to a mailer for storage,
uses $notify_method to specify how to deliver to MTA;
- much like a newer <tt>smtp:</tt> entry below</td></tr>
+ much like a newer '<tt>smtp:</tt>' entry below</td></tr>
<tr>
<td><tt>smtp:</tt></td>
<td>e-mail address</td>
<td>anything</td>
<td>sent via SMTP to a mailer for storage,
- uses the specified IP address and port for delivery;
- formerly a <tt>local:</tt> method was used for this
- purpose</td></tr>
+ uses the specified IP address and port,
+ or a Unix socket for delivery; formerly
+ a '<tt>local:</tt>' method was used for
+ this purpose</td></tr>
+<tr>
+ <td><tt>lmtp:</tt></td>
+ <td>e-mail address</td>
+ <td>anything</td>
+ <td>sent via LMTP to a mailer for storage,
+ uses the specified IP address and port,
+ or a Unix socket for delivery</td></tr>
<tr>
<td><tt>bsmtp:</tt></td>
<td>anything (nonempty)</td>
@@ -1198,21 +1216,26 @@ from internally originating, authenticat
# global default:
content_filter=smtp-amavis:[127.0.0.1]:10044
+# note that permit_mynetworks only checks for key presence and ignores rhs
+mynetworks = cidr:/etc/postfix/mynetworks-filter.cidr
+
smtpd_recipient_restrictions =
reject_unauth_pipelining, reject_non_fqdn_recipient, reject_non_fqdn_sender,
reject_unknown_recipient_domain, reject_unknown_sender_domain,
- check_client_access cidr:/etc/postfix/filter-mynets.cidr,
+ check_client_access cidr:/etc/postfix/mynetworks-filter.cidr,
+ permit_mynetworks,
permit_sasl_authenticated, permit_tls_clientcerts,
reject_unauth_destination,
check_sender_access regexp:/etc/postfix/filter-catchall.regexp
</pre>
-<p>The <tt>check_client_access cidr:/etc/postfix/filter-mynets.cidr</tt>
-takes the place of the more usual <i>permit_mynetworks</i>, and it serves
-to be able to override the global <i>content_filter</i> setting by the
-use of FILTER for each of the networks (presumably internal) listed in
-filter-mynets.cidr. The final effect is that mail matching networks listed
-in filter-mynets.cidr will be sent for content filtering to tcp port 10042
+<p>The <tt>check_client_access cidr:/etc/postfix/mynetworks-filter.cidr</tt>
+preceeds the <i>permit_mynetworks</i> (which uses the same cidr table,
+but ignores the righthand side), and it serves to override the global
+<i>content_filter</i> setting by the use of FILTER for each of the
+networks (presumably internal) listed in mynetworks-filter.cidr.
+The final effect is that mail matching networks listed in
+mynetworks-filter.cidr will be sent for content filtering to tcp port 10042
(the FILTER setting in access map), authenticated non-local mail will be
sent for content filtering to port 10044 (the global setting), while all
the rest will be sent to port 10040 (as specified in catchall filter).
@@ -1220,7 +1243,7 @@ they take precedence over the global set
they take precedence over the global settings, but the FILTER rules take
the ultimate precedence.</p>
-<p>/etc/postfix/filter-mynets.cidr :</p>
+<p>/etc/postfix/mynetworks-filter.cidr :</p>
<pre>
127.0.0.0/8 FILTER smtp-amavis:[127.0.0.1]:10042
10.0.0.0/8 FILTER smtp-amavis:[127.0.0.1]:10042
@@ -1337,7 +1360,7 @@ feeding a content filter (typically this
'smtp-amavis').</p>
-<h2><a name="dkim">Setting up DKIM and DomainKeys mail
+<h2><a name="dkim">Setting up DKIM (and DomainKeys) mail
signing and verification</a></h2>
<p>The goals of DKIM and DomainKeys are:</p>
@@ -1346,11 +1369,12 @@ signing and verification</a></h2>
<li>protection against message tampering.</li>
</ul>
-<p>A DKIM draft states the following, which applies to its predecessor
-DomainKeys as well:</p>
+<p>A DKIM standard (RFC 4871) states the following, which applies
+to its predecessor DomainKeys (historical: RFC 4870) as well:</p>
<blockquote>
-<p><i>DomainKeys Identified Mail (DKIM)</i> defines a mechanism by which email
+<p><i>
+DomainKeys Identified Mail (DKIM)</i> defines a mechanism by which email
messages can be cryptographically signed, permitting a signing domain
to claim responsibility for the introduction of a message into the
mail stream. Message recipients can verify the signature by querying
@@ -1362,7 +1386,7 @@ possession of the private key for the si
<p>A gentle introduction and deployment guide is available at:
<a href="http://antispam.yahoo.com/domainkeys"
>http://antispam.yahoo.com/domainkeys</a>.
-Except for some minor details, it applies to DKIM system as well.</p>
+Except for some details, it applies to DKIM system as well.</p>
<p>With added support in Postfix 2.3 for a milter protocol, it became
possible to use with Postfix many of existing milters (mail filters)
@@ -1379,8 +1403,7 @@ offering support for <i>DomainKeys Ident
offering support for <i>DomainKeys Identified Mail (DKIM) Signatures</i>,
and <a href="http://sourceforge.net/projects/dk-milter/">dk-milter</a>,
offering support for <i>Domain-based Email Authentication (DomainKeys)</i>.
-The DomainKeys (DK) is a predecessor of DKIM, as recognized by
-draft-delany-domainkeys-base-06:</p>
+The DomainKeys (DK) is a predecessor of DKIM, as recognized by RFC 4870:</p>
<blockquote>
<p>The <i>DomainKeys</i> specification was a primary source from which the
@@ -1389,21 +1412,22 @@ for deployed implementations written pri
for deployed implementations written prior to the DKIM specification.</p>
</blockquote>
-<p>At the time of this writing it appears the <i>dkim-milter</i> is more
-reliable and better maintained than <i>dk-milter</i>, which is slowly
-fading into oblivion. Similar holds true in the world of Perl modules:
-there are modules <i>Mail::DomainKeys</i> and <i>Mail::DKIM</i>,
-both of which can be used by SpamAssassin. Again the <i>Mail::DKIM</i>
-(by Jason Long and Anthony D. Urso) seems to be of higher quality
-than the older <i>Mail::DomainKeys</i>. SpamAssassin makes it very
-easy to use each or both of them (for verification only), just by
-enabling the already provided plugins.</p>
-
-<p>Despite DomainKeys slowly giving grounds to DKIM, the DomainKeys
-is currently still in use by several large players in the Internet
-world, so this section will describe how to integrate both of them
-with Postfix and amavisd-new (an after-queue content filter) into a
-mail system.</p>
+<p>The <i>dkim-milter</i> is more reliable and better maintained than
+<i>dk-milter</i>, which is slowly fading into oblivion and is no longer
+actively maintained. Google.com has already switched to DKIM, and Yahoo
+is following shortly. Similar holds true in the world of Perl modules:
+there are modules <i>Mail::DomainKeys</i> and <i>Mail::DKIM</i>, both of
+which can be used by SpamAssassin plugins. Again the <i>Mail::DKIM</i>
+(by Jason Long, based on initial work by Anthony D. Urso) is of higher quality
+and actively maintained, compared to its predecessor <i>Mail::DomainKeys</i>.
+SpamAssassin makes it very easy to use each or both of them (for verification
+only), just by enabling the already provided plugins. Note that recent
+versions of <i>Mail::DKIM</i> support also DomainKeys signatures.</p>
+
+<p>Despite DomainKeys giving grounds to DKIM, the DomainKeys
+is currently still in use by some players in the Internet world,
+so this section will describe how to integrate both of them with Postfix
+and amavisd-new (an after-queue content filter) into a mail system.</p>
<p>Mail signing and verification is a two-part job: signing of originating
mail (or mail being redistributed) from our domain, and verifying signatures
@@ -1487,7 +1511,7 @@ both results of signature verification c
<p>Let's begin by starting both milters, each in two instances, one
dedicated for signing, the other for verification. For security reasons
all milters should be run under a dedicated username, certainly not as root,
-not as user amavis and not as user postfix:</p>
+not as user amavis and not as user postfix or mail:</p>
<p><i>dk, verifying:</i></p>
@@ -1525,12 +1549,12 @@ signing key and the same selector. Gener
signing key and the same selector. Generating a pair of public
and private key and publishing a public key and a policy in DNS
is described in the documentation of each milter and also in
-DKIM and DomainKeys drafts.</p>
+DKIM and DomainKeys RFC documents.</p>
<p>We are not specifying option -i to milters, the default of
-i 127.0.0.1 suits our setup just fine, as mail to be signed is
coming from a content filter, usually on a loopback interface
-from IP address 127.0.0.1.</p>
+from the IP address 127.0.0.1.</p>
<p>Now we can tie both verifying milters to a Postfix smtpd service
listening for incoming mail:</p>
@@ -1568,12 +1592,30 @@ protect its own signature in its signed
protect its own signature in its signed data, but the newer DKIM fixes
this omission and does protect its own signature.</p>
-<p>Default content filter is to be amavisd, listening on port 10026
-(intended for signing). Locally submited or SASL-authenticated mail will
-go to a content filter on this default port 10026 and will be signed on
-its way out. All other mail (incoming) will be diverted to port 10024
-for content filtering by a final catchall FILTER, and will never hit
-the signing milter:</p>
+<p>As a sidenote, attaching milters to sendmail would use the same order
+of invocations: signature verifying milters first, content filters next,
+and signing milters last, for example:</p>
+
+<pre>
+ dnl Verifiers:
+ INPUT_MAIL_FILTER(`dk-filter-v', `S=inet:4442 at 127.0.0.1, T=R:2m')
+ INPUT_MAIL_FILTER(`dkim-filter-v', `S=inet:4443 at 127.0.0.1, T=R:2m')
+
+ dnl Content filter:
+ INPUT_MAIL_FILTER(`amavisd-milter',
+ `S=unix:/var/amavis/amavisd-milter.sock, F=T, T=S:10m;R:10m;E:10m')
+
+ dnl Signers:
+ INPUT_MAIL_FILTER(`dk-filter-s', `S=inet:4444 at 127.0.0.1, T=R:2m')
+ INPUT_MAIL_FILTER(`dkim-filter-s', `S=inet:4445 at 127.0.0.1, T=R:2m')
+</pre>
+
+<p>Now back to Postfix setup (or any dual-MTA setup). Default content
+filter is to be amavisd, listening on port 10026 (intended for signing).
+Locally submited or SASL-authenticated mail will go to a content filter
+on this default port 10026 and will be signed on its way out. All other
+mail (incoming) will be diverted to port 10024 for content filtering
+by a final catchall FILTER, and will never hit the signing milter:</p>
<p><i>main.cf:</i></p>
<pre>
@@ -1595,33 +1637,38 @@ the signing milter:</p>
</pre>
<p>In SpamAssassin all that is necessary is to add (or uncomment) lines
-in any of the .pre files (e.g. in local.pre, or in init.pre and v312.pre):</p>
+in any of the .pre files (e.g. in local.pre, or in init.pre and v320.pre):</p>
<pre>
loadplugin Mail::SpamAssassin::Plugin::DomainKeys
loadplugin Mail::SpamAssassin::Plugin::DKIM
</pre>
-<p>Perl modules Mail::DomainKeys (version 0.86 or better) and Mail::DKIM
-(0.19 or better) need to be installed.</p>
-
-<p>The following SpamAssassin rules (in local.cf) work fairly well, giving
-verified mail a little bit of advantage and slightly favourize mail from
-some popular domains, and encourage people to start signing their mail.
-Possible signed spam can be counterbalanced by other measures.</p>
-
-<pre>
- score DK_VERIFIED -1.5
+<p>Perl modules Mail::DomainKeys (version 0.88 or better) and Mail::DKIM
+need to be installed. Note that Mail::DKIM starting with version 0.20 also
+recognizes DomainKeys signatures, so that Plugin::DomainKeys is not needed
+any longer, and in fact its underlying module is not supported any longer.
+It is advisable to stick to the most recent version of Mail::DKIM.</p>
+
+<p>The following SpamAssassin rules (in local.cf) work fairly well,
+giving verified mail a little bit of advantage and slightly favourize
+verified mail from some popular domains, encouraging people to start
+signing their mail. Possible signed spam can be counterbalanced by
+other measures.</p>
+
+<pre>
+ score DK_VERIFIED -1.1
score DK_POLICY_SIGNSOME 0
score DK_POLICY_TESTING 0
- score DKIM_VERIFIED -1.5
+ score DKIM_VERIFIED -1.3
score DKIM_POLICY_TESTING 0
# DKIM and DK-based whitelisting may be used reliably:
- score USER_IN_DKIM_WHITELIST -3.0
- whitelist_from_dkim *@friends.example.com
- whitelist_from_dk *@friends.example.com
+ score USER_IN_DKIM_WHITELIST -4.0
+ score USER_IN_DK_WHITELIST -4.0
+ whitelist_from_dkim *@friends.example.com example.com
+ whitelist_from_dkim *@intl.paypal.com paypal.com
</pre>
<p>Another suggestions - penalize mail claiming to be from Yahoo
@@ -1640,10 +1687,10 @@ or Gmail but was not signed by their off
header __L_FROM_Y4 From:addr =~ m{\@yahoo\.(ca|de|dk|es|fr|gr|ie|it|pl|se)$}i
meta __L_FROM_YAHOO __L_FROM_Y1 || __L_FROM_Y2 || __L_FROM_Y3 || __L_FROM_Y4
header __L_FROM_GMAIL From:addr =~ m{\@gmail\.com$}i
- meta L_UNVERIFIED_YAHOO !DK_VERIFIED && __L_FROM_YAHOO && !__L_VIA_ML
+ meta L_UNVERIFIED_YAHOO !DKIM_VERIFIED && __L_FROM_YAHOO && !__L_VIA_ML
priority L_UNVERIFIED_YAHOO 500
score L_UNVERIFIED_YAHOO 2.5
- meta L_UNVERIFIED_GMAIL !DK_VERIFIED && __L_FROM_GMAIL && !__L_VIA_ML
+ meta L_UNVERIFIED_GMAIL !DKIM_VERIFIED && __L_FROM_GMAIL && !__L_VIA_ML
priority L_UNVERIFIED_GMAIL 500
score L_UNVERIFIED_GMAIL 2.5
</pre>
@@ -1656,18 +1703,20 @@ the other receiving on port 10026 and fo
$inet_socket_port = [10024,10026]; # listen on two ports
</pre>
-<p>The 10024/10025 path will be controlled by a default policy bank,
-the other, dedicated to mail needing to be signed, will use a policy
-bank (arbitrarily) named ORIGINATING:</p>
+<p>The 10024>10025 path will be controlled by a default policy bank,
+the other (10026>10027), dedicated to mail needing to be signed,
+will use a policy bank (arbitrarily) named ORIGINATING:</p>
<pre>
$forward_method = 'smtp:[127.0.0.1]:10025'; # MTA with non-signing service
- $notify_method = 'smtp:[127.0.0.1]:10027'; # MTA with DKIM signing service
+ $notify_method = 'smtp:[127.0.0.1]:10027'; # MTA with signing service
# switch policy bank to 'ORIGINATING' for mail received on port 10026:
$interface_policy{'10026'} = 'ORIGINATING';
$policy_bank{'ORIGINATING'} = { # mail originating from our users
+ originating => 1, # indicates our client, introduced in amavisd-new-2.5.0
+ #
# force MTA to convert mail to 7-bit before DKIM signing
# to avoid later conversions which could destroy signature:
smtpd_discard_ehlo_keywords => ['8BITMIME'],
@@ -1681,7 +1730,7 @@ bank (arbitrarily) named ORIGINATING:</p
banned_filename_maps => ['ALT-RULES'], # more relaxed rules
spam_quarantine_cutoff_level_maps => undef, # quarantine all spam
spam_dsn_cutoff_level_maps => undef,
- spam_dsn_cutoff_level_bysender_maps => # bounce to local users only
+ spam_dsn_cutoff_level_bysender_maps => # bounce to local senders only
[ { lc(".$mydomain") => undef, '.' => 15 } ],
};
</pre>
@@ -1698,6 +1747,13 @@ feeding mail to be signed to amavisd, bu
feeding mail to be signed to amavisd, but this would require setting
up two such services, one with the option and one without.</p>
+<p>Note that 8-bit to 7-bit conversion may break SMIME or PGP signature,
+so if mail signing is in use, it may not be desirable to let Postfix
+do the conversion, and it may be acceptable to take a risk that a remote
+MTA will clobber signatures if it decides the mail text is to be converted
+to 7-bits QP. The only reliable solution in this case is to configure
+MUA clients to stick to 7-bit characters/encodings only.</p>
+
<p>The following text from the Postfix documentation file MILTER_README
<em>should be disregarded</em> -- amavisd <em>is</em> 8-bit clean,
and we want Postfix to do conversion to 7-bits on the signing
@@ -1709,9 +1765,9 @@ advanced content filter example.</span><
advanced content filter example.</span></p>
<p>While testing how the configured system plays with some mailing lists
-(such as postfix-users or SpamAssassin users list), one has to keep in
-mind that amavisd-new caches spam checking results of recently seen
-message bodies, so a mail going out to a mailing list is not yet signed
+(such as <i>postfix-users</i> or SpamAssassin <i>users</i> list), one has
+to keep in mind that amavisd-new caches spam checking results of recently
+seen message bodies: a mail going out to a mailing list is not yet signed
as it reaches a content filter, but the SpamAssassin verdict is remembered
at that point (claiming the message is not signed). When this message
with unchanged body comes back from a mailing list, this time signed
@@ -1727,38 +1783,43 @@ interoperability problems with earlier v
interoperability problems with earlier versions:</p>
<ul>
-<li>use Postfix 2.3.3 or later (fixing minor 2.3 problems with milter);</li>
+<li>use Postfix 2.3.3 or later;</li>
<li>amavisd-new 2.4.3 polished some corner issues on modifying mail header
when releasing from a quarantine and defanging, and added some goodies
affecting DKIM and DomainKeys to facilitate integration;</li>
-<li>Mail::DomainKeys 0.86 or later must be used, the last couple of versions
- squashed several bugs, one at a time. Version 0.84 dropped the use of
- unreliable Email::Address (which could cause deep recursion in evaluating
- regular expressions, bringing processing to a halt).
+<li>Mail::DomainKeys 0.88 or later must be used (currently at 1.0), the last
+ couple of versions squashed several bugs, one at a time. Version 0.84 dropped
+ the use of unreliable Email::Address (which could cause deep recursion in
+ evaluating regular expressions, bringing processing to a halt).
Nevertheless, one patch is still needed (
- <a href="http://www.ijs.si/software/amavisd/Mail-DomainKeys-mark.patch"
- >http://www.ijs.si/software/amavisd/Mail-DomainKeys-mark.patch</a> ),
- hopefully to be included with the next version;</li>
-<li>Mail::DKIM 0.19 is very solid, it is the only component where no bugs
- or design flaws were found during experimenting; in fact, this module
- helped to discover bugs in other components;</li>
+ <a href="http://www.ijs.si/software/amavisd/Mail-DomainKeys-0.88-mark.patch"
+ >http://www.ijs.si/software/amavisd/Mail-DomainKeys-0.88-mark.patch</a> ).
+ I believe it has been incorporate in the final version 1.0.
+ The Mail::DomainKeys perl module reached the end of its life with a
+ version 1.0 and is not expected to be maintained any longer.</li>
+<li>Mail::DKIM is very solid, it is the only component where no bugs or
+ design flaws were found during initial experimenting; in fact, this module
+ helped to discover bugs in other components; use the latest version,
+ currently 0.25;</li>
<li>SpamAssassin 3.1.5 added whitelisting based on verified DomainKeys
signatures, similar to what was already available for DKIM whitelisting
in earlier versions of SA;</li>
<li>dk-milter 0.4.1 needs a patch as described in the Postfix documentation
file <a href="http://www-cmi.ijs.si/doc/postfix/MILTER_README.html"
>MILTER_README</a>;</li>
-<li>the dk-milter 0.4.1 seems to be neglected lately and is rather buggy
- compared to dkim-milter. Nevertheless, with command options as given
- in the above example, it does its job sufficiently well in a described
+<li>the dk-milter 0.4.1 is neglected and is rather buggy compared
+ to dkim-milter. Nevertheless, with command options as given in
+ the above example, it does its job reasonably well in a described
Postfix + amavisd-new setup, so that it may be deployed for a
- production use; see its bug tracking system for details;</li>
+ production use; see its bug tracking system for details (and there
+ are more bugs than noted in its bug tracking system);</li>
<li>dkim-milter 0.5.2 is already compatible with Postfix, no patch is
required; the 0.5.2 also adds a missing body hash check (omitted in
0.5.1 and earlier versions) and fixes "relaxed" body canonicalization
algorithm; because the other end may still be running 0.5.1 or older,
it is best to avoid "relaxed" body canonicalization, so choose
- <i>-c relaxed/simple</i>.</li>
+ <i>-c relaxed/simple</i>. At the time of writing the current version
+ is 0.8.0 and brings it up to currency with RFC 4871.</li>
</ul>
<p>Instead of signing mail with <i>dkim-milter</i>, the same can be
@@ -1766,21 +1827,22 @@ calling a Perl module <i>Mail::DKIM</i>,
calling a Perl module <i>Mail::DKIM</i>, i.e. the same modules as used
by a SpamAssassin DKIM plugin. As the <i>Mail::DKIM</i> turned out to be
a reliable and quite efficient module, this may be a good alternative
-to <i>dkim-milter</i> (which is also quite good).</p>
-
-<p>On the other hand there exist a <i>dkfilter</i> SMTP-proxy by the same
+to <i>dkim-milter</i> (which is also quite good, but slower for signing
+larger messages).</p>
+
+<p>On the other hand there exists a <i>dkfilter</i> SMTP-proxy by the same
author, which calls a Perl module <i>Mail::DomainKays</i>, which in turn
-is not recommended for processing of entire messages because of its
-design limitation which requires loading the whole message into memory.
-The use of <i>Mail::DomainKays</i> from within SpamAssassin does not
-represent such a problem, as messages checked by SpamAssassin are already
-limited in size.</p>
+is not recommended because of its design limitation which requires loading
+the whole message into memory. The use of <i>Mail::DomainKays</i> from within
+SpamAssassin does not suffer from this problem, as messages checked by
+SpamAssassin are already limited in size.</p>
<p>Under FreeBSD just install both milters (mail/dkim-milter and mail/dk-milter)
from freshly updated ports, and add the following to <tt>/etc/rc.conf</tt>:</p>
<pre>
milterdk_enable="YES"
+ milterdk_uid='dkimfilter'
milterdk_profiles="verifier signer"
milterdk_verifier_socket='inet:4442 at 127.0.0.1'
milterdk_verifier_flags='-b v -H'
@@ -1789,6 +1851,7 @@ from freshly updated ports, and add the
-d example.org -S mysel -s /var/db/domainkeys/mysel.key.pem'
milterdkim_enable="YES"
+ milterdkim_uid='dkimfilter'
milterdkim_profiles="verifier signer"
milterdkim_verifier_socket='inet:4443 at 127.0.0.1'
milterdkim_verifier_flags='-b v'
@@ -1799,7 +1862,9 @@ from freshly updated ports, and add the
<p>Recent startup scripts supplied by both FreeBSD ports already ensure
that both deamons run under their dedicated usernames, supplying
-a default value for option -u.</p>
+a default value for option -u. The mail/dk-milter port also already
+includes a patch to make it compatible with Postfix; (the mail/dkim-milter
+doesn't need such patch, 0.5.2 and later is compatible with Postfix).</p>
<p>Mail transformations as performed by some mailing lists are probably
the most challenging problem facing DKIM and DomainKeys deployment (and
@@ -1813,7 +1878,8 @@ of another type of mailing lists is curr
of another type of mailing lists is currently Mailman, which modifies
mail body, but at least it is stripping original signatures and adding
a Sender header, so that original signatures do not appear to be broken,
-they are just missing, and mail may be re-signed.</p>
+they are just missing, and mail may be re-signed - not too good, but
+could be worse.</p>
<p>Several big players are already signing mail from their users
or employees: Yahoo! (worldwide), Gmail, eBay, Earthlink,
@@ -1826,9 +1892,9 @@ Similar to other schemes designed to pre
Similar to other schemes designed to prevent faking of sending address,
the DKIM (and the DomainKeys) encourages mail submission only through
a domain which is used in the <i>From</i> address - although there are
-other possibilities for roving users, especially in the DKIM system.
+other possibilities for roaming users, especially in the DKIM system.
People will need to become aware that their best choice is to submit
-mail through their home domain to prevent their messages being treated
+mail through their native domain to prevent their messages being treated
as second-class or appear suspicious.</p>
<p>Note that some spam is also being signed by DomainKeys or DKIM lately,
@@ -1838,19 +1904,40 @@ case such mail can be easily filtered if
case such mail can be easily filtered if desired), or they are using
a short-lived temporary domain (perhaps through <i>domain kiting</i>),
which can be counteracted by black lists of few-days old freshly
-registered domains. Adding a small negative spam score to successfully
-verified mail will encourage people to start signing their mail,
-benefiting legitimate senders and recipients, while signed spam can
-be counterbalanced by other measures.</p>
-
-<p>Signing and verifying mail is a good mechanism for companies to reliably
-whitelist mail from their partner companies or frequent clients.</p>
-
+registered domains (such as <a href="http://support-intelligence.com/dob/"
+>http://support-intelligence.com/dob/</a>). Adding a small negative spam
+score to successfully verified mail will encourage people to start signing
+their mail, benefiting legitimate senders and recipients, while signed
+spam can be counterbalanced by other measures. Signing and verifying mail
+is a good mechanism for companies to reliably whitelist mail from their
+partner companies or frequent clients.</p>
+
+<p>Some references:</p>
+<ul>
+<li><a href="http://www.postfix.org/MILTER_README.html">Postfix
+ before-queue Milter support</a> (original Postfix documentation)</li>
+<li><a href="http://antispam.yahoo.com/domainkeys">DomainKeys:
+ Proving and Protecting Email Sender Identity</a> (original
+ DK <i>Yahoo!</i> document)</li>
+<li><a href="http://www.rfc-editor.org/rfc/rfc4686.txt">RFC 4686</a>:
+ Analysis of Threats Motivating DomainKeys Identified Mail (DKIM)</li>
+<li><a href="http://www.rfc-editor.org/rfc/rfc4871.txt">RFC 4871</a>:
+ DomainKeys Identified Mail (DKIM) Signatures</li>
+<li><a href="http://www.rfc-editor.org/rfc/rfc4870.txt">RFC 4870</a>:
+ (historical document) Domain-Based Email Authentication Using
+ Public Keys Advertised in the DNS (DomainKeys)</li>
+<li><a href="http://ietf.org/html.charters/dkim-charter.html">IETF
+ charter: Domain Keys Identified Mail (DKIM)</a></li>
+<li><a href="http://www.dkim.org/deploy/">DKIM Deployment
+ Reports</a> (at <i>Mutual Internet Practices Association</i> - MIPA)</li>
+<li><a href="http://jason.long.name/dkimproxy/">Mail::DKIM and dkimproxy</a></li>
+<li><a href="http://sourceforge.net/projects/dkim-milter/">dkim-milter</a></li>
+</ul>
<hr />
<p>
<i><a href="http://www.ijs.si/people/mark/">mm</a></i>
-<br />Last updated: 2006-09-29
+<br />Last updated: 2007-05-30
</p>
<p>
diff --git a/README_FILES/images/1.png b/README_FILES/images/1.png
new file mode 100644
index 0000000000000000000000000000000000000000..7d473430b7bec514f7de12f5769fe7c5859e8c5d
GIT binary patch
literal 329
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%U7FWJ+ at F{I*F
zXx~QO0|p!{`<Hx?($n1Ql(x9!rI~U{@0x=gQ&>C}X^4DKU-G|w_t}fLBA)Suv#nrW
z!^h2QnY_`l!B<x?#cqSRfyQ~Q1@<4<=dkIYNUpteZJn&Cq*Q53=dm6u at dFh)Vs8$H
zYyn#5T;dv05}cn_Ql40p%HW`(tm&DXn4apJn4X!Otze>Oq-UXEX{m2up>JTQkX)2m
zTvF+fTUlI^nXH#utd~++ke^qgmz<wFahKu_pmADY<1*9p5=%;f=9yX~8kifU8kifT
zBpMl|nHd_JCMTsBr<xd=n3|ZSS!QG at z6I*y1L+C?TFYQ)RAFeOZJ=$?lDoSW6h at w|
KelF{r5}E*jd}poz
diff --git a/README_FILES/images/10.png b/README_FILES/images/10.png
new file mode 100644
index 0000000000000000000000000000000000000000..997bbc8246a316e040e0804174ba260e219d7d33
GIT binary patch
literal 361
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%U7ui4YZF{I*_
zYTrUp7DWMP<%u3TLY;HO{NlKy4{a+D>WtZ~+OvdJMW|Y+^UT?O-M{rKJsmzxdayJ{
zDCQA!%%@7Jj$q%-wf8e0_jRx8Dqi$}^?K=?6FriQFLv>>oc^CE+aVHhW<k9IkK}je
z)E~Fj9=KEXbDR6NEa}#hinfbWM4tz3wYI;z$T{gWXSN{FInE`n5hcO-X(i=}MX3x9
z8p at iUxrynizKQ9X$=M1fdPaH{3YM1o1{V4Th6>3=nZ+fQ4!M=ZC7H>3sl|FJr3LwU
zC3?yExf6FO?f at F61vV}-Juk7O6lk8Yg;}bFaZ-|HQc7A<qOp;Yd77z3Qj&$Gd8$#e
sMWU&|d~IH!E<TX10HC!DhDH^JM%o721}(X}Ye8Y?>FVdQ&MBb at 01?u8M*si-
diff --git a/README_FILES/images/11.png b/README_FILES/images/11.png
new file mode 100644
index 0000000000000000000000000000000000000000..ce47dac3f52ac49017749a3fea53db57d006993c
GIT binary patch
literal 565
zc%17D at N?(olHy`uVBq!ia0vp^JRr=$1SD^YpWXnZI14-?iy0V%N<f(LUYdLbP>{XE
z)7O>#600DeuDZ?5tOl@<Yeb1-X-P(Y5yQ%LXFNdWfP`F9i<65o3raHc^B9CV6_tVF
zTp)4h{DR7&%=C;BhBf?Wtbn3?V9}z~M37R2lFZyx2Gj5SoFFHFWP<b4O3D+9QW;_k
z{)3crgALCtPR>aLDlKBzeqc*KP?!TG9G+N|mzkH&a6tN5JIrap`K3k4sSIELH@}7{
zbI#ArP0cG|_;zO6exMi|NGv3?Bqx<&#k))D3=E76o-U3d6?01a*RE<Q5IA~1o at W+|
zeN%0>rx|xsgsX_Se4>0(^ZG=g6nTf{n>n|)oXrvG;d*sv!Rx8NKXw~GKQrfW2Z#K%
z(zm<(1Xz487qoSBWXxKo;u#Wpx2-KPB-B;j!R7U?dwJ%u*K=>P9r#=$;wp38cK-SN
zz-g)1Uw{2uC+ at oNbyeiF+WX7`EDLj^l^!uH$*kIce|aX8088T{joX(CWj@>7H};%#
zM55Y8&Na~Bn9k{2dW;>9EB32+Ub^@(rYxHA0Yk;S_WBRC_1nv0)lYxQZ+*S%^rkm&
zJ62 at QI{maLtR>Nc=kT^v_m0YWuXok$ja#H))t5Z$Sz?0yq<<&Hb=FLIzKeB{2rzUQ
NJYD@<);T3K0RWit*meK_
diff --git a/README_FILES/images/12.png b/README_FILES/images/12.png
new file mode 100644
index 0000000000000000000000000000000000000000..31daf4e2f25b6712499ee32de9c2e3b050b691ca
GIT binary patch
literal 617
zc%17D at N?(olHy`uVBq!ia0vp^JRr=$1SD^YpWXnZI14-?iy0V%N<f(LUYdLbP>{XE
z)7O>#600De9$%>2LVd81Yeb1-X-P(Y5yQ%LXFNdWfP`F9i<65o3raHc^B9CV6_tVF
zTp)4h{DR7&%=C;BhBf?Wtbn3?V9}z~M37R2lFZyx2Gj5SoFFHFWP<b4O3D+9QW;_k
z{)3crgALCtPR>aLDlKBzeqc*KP?!TG9G+N|mzkH&a6tN5JIrap`K3k4sSIELH@}7{
zbI#ArP0cG|_;zO6exMi|NGv3?Bqx<&#k))D3=E76o-U3d6?0lAD6T$~AkljMm;g&+
zLQtEI{z(b#FjMJm^$XX$5nsb)tRwM<QQJFAP<Pu*-OU`IQk#$V=RR+{`}*m<g7?Aa
zD=Yc?orD+~KL4DP8W|{ZH?Q1Au6o|}tgW*+R6N}RTjuw;Ek6GE;u3|Ock^z){r0dR
zLPzZI!w+S<ZRel=S at Z7qzx0*svQKaFTRwS;5l53n-}Tp9-~Yb;TGV0Qa}~uFh6lUe
z<!$F~VsKb~*;1}Qa~4yC-gFgH&D7$HJ{L=C_utQ+^{i5^;reS<2U#Ary>ajNns+H3
zFKiL;oMa-;prI1JI#i<V^2;xE`=#3wCD<BPt<szBJ at 5IdRZS_8(>C9HGhO{doGUPN
zG(cg|RJT7~<mh&$NkOO1rWw!l at wvPu*P6#!k!63ty<`6!{%&V{z?{voy0pXJWEW5m
NgQu&X%Q~loCIB`n?b84N
diff --git a/README_FILES/images/13.png b/README_FILES/images/13.png
new file mode 100644
index 0000000000000000000000000000000000000000..14021a89c2ed3d4881afea6e3a315bce4f95efce
GIT binary patch
literal 623
zc%17D at N?(olHy`uVBq!ia0vp^JRr=$1SD^YpWXnZI14-?iy0V%N<f(LUYdLbP>{XE
z)7O>#600Dep5bGK9wD%hYeb1-X-P(Y5yQ%LXFNdWfP`F9i<65o3raHc^B9CV6_tVF
zTp)4h{DR7&%=C;BhBf?Wtbn3?V9}z~M37R2lFZyx2Gj5SoFFHFWP<b4O3D+9QW;_k
z{)3crgALCtPR>aLDlKBzeqc*KP?!TG9G+N|mzkH&a6tN5JIrap`K3k4sSIELH@}7{
zbI#ArP0cG|_;zO6exMi|NGv3?Bqx<&#k))D3=E76o-U3d6?0lADE2oM$h5_OXgX at i
zAvR?_0}Jz+vmO&3O<80tv2j9YPS5+pZjY2bO4OKKkNtc3zC=po`|bmG at 5Jt?{v>?w
zxo8ImgJ5dpv}Kv0p<Rp4q#57d#_H(6EokbyId1*O$yc+kn(zt=3U+98DMe1>Z*P8V
zG5KVR;W3HXe#;XjwoN)<DbLq_{BdC5l%q+Bv(%o)gkFtX|Ge_v%aTbdo at YM)Dciks
zUGy~71_chb=7V|LSHJ#MxBvb3YKBLTKQc^sUTJ-F+glaSX{VRI-o-HQym|xwVFRnV
z=U#u!QaxUH?6G0;8V#;Mk<!@zt8V9QXL$DcXNJkEvfcBZ|J--~{de=(XLrBLU|zKx
z7&>~Ou&A-ip7rfDlka7V=lPu->#k?Z9G{lDa#dDjn9T9%r(5?4-a7u@?ay_F51bRf
WuQ~NZ+SwSWlfl!~&t;ucLK6U{l=~3?
diff --git a/README_FILES/images/14.png b/README_FILES/images/14.png
new file mode 100644
index 0000000000000000000000000000000000000000..64014b75fe2e84d45ed861974c72462727979360
GIT binary patch
literal 411
zc$@*70c8G(P)<h;3K|Lk000e1NJLTq000aC000aK0ssI2*%!;O00006VoOIv0RI60
z0RN!9r;`8x010qNS#tmY5_13m5_18JBDn_u000McNliru(ghI at 10bk)oHzgg0VqjC
zK~#90eULq>!ax*-PXaQ9e~6^e1gu=a6a&KSz}bR`+prYG9ayB$BDjWGfIE;t#wl!+
zR3S(jA%y#i_ at eOOedXoc%RQe%L;wH~k+s%ZI~)!<=dD%?4MaplaU9QPGski2q3`>r
z(}{j at 0a$CLl+)={2vLWml*i-oa5#J}DW$gCZB<pv<q`nr{PlXR*Xw at 2pXWIdxkXWE
zt%D#i#sFxy+ffu1MFHUXd}^%$xR1x9EX&1Wk)|nt-EP<EbZ)ns5Te`d-tTt+?sz=@
zE|TZ@`F!>~Z!(!M#)2St|1_V^0qpmCrBof=Y&NUas at LmfSw=)4B4f;8Fu)(eFsv24
zJzXxBrayquXcR?J<H`9po+Qcjdi6YybIurRc0Qqh_yL28ryr7W9)17-002ovPDHLk
FV1h9xuy+6e
diff --git a/README_FILES/images/15.png b/README_FILES/images/15.png
new file mode 100644
index 0000000000000000000000000000000000000000..0d65765fcf13dcfd87914744dec8bda115e4adf1
GIT binary patch
literal 640
zc%17D at N?(olHy`uVBq!ia0vp^JRr=$1SD^YpWXnZI14-?iy0V%N<f(LUYdLbP>{XE
z)7O>#600De0j~t#c`vY#Yeb1-X-P(Y5yQ%LXFNdWfP`F9i<65o3raHc^B9CV6_tVF
zTp)4h{DR7&%=C;BhBf?Wtbn3?V9}z~M37R2lFZyx2Gj5SoFFHFWP<b4O3D+9QW;_k
z{)3crgALCtPR>aLDlKBzeqc*KP?!TG9G+N|mzkH&a6tN5JIrap`K3k4sSIELH@}7{
zbI#ArP0cG|_;zO6exMi|NGv3?Bqx<&#k))D3=E76o-U3d6>~}_7 at lrQ5NYjSFT~-R
zAZE&9xvf*zamyF>sA#Fy1<S6 at S#@;bv5X?#Me{G}XBavz(Oo{hN$KPZGYO}fn)E%-
z=bU~h*ulZTpkyV(cl@#8?6YRG&pIt^2yt<7nfKgOs`tx`lPS+W|NQ;eFQ&1>;`y$1
zuXpL4{#3SmtB$wG(bG>Cty+~DdEdYJV1k8=Ti})$z3i<~GRM1*CK=82nDk_qo}i)9
ztsJwiMQ^@WGqfcd9CKS7xHG11Z`|&?d<@k#a at Sv%n$2c)WOz_x=fC`N*y_|-X)8mx
z4ht|$PTd)yvuyJ1w`I#OpL|+WC}ThWeAZU32F|7n at 4quv%zOTO-(!pIw{uTSndk!y
z9eYq%Y>nzY{`mLbd#}IlJ8b!U*0P8FIk)`;YAomaEzjGo-IbISt~K>*)!)OKYm3gs
nUO)T%^Vcd{U4|c=f0y$gS6sJdl7PnwpivB-u6{1-oD!M<RmK1q
diff --git a/README_FILES/images/2.png b/README_FILES/images/2.png
new file mode 100644
index 0000000000000000000000000000000000000000..5d09341b2f6d2ea2d1d5dad5d980f14b4b05dfd2
GIT binary patch
literal 353
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%U7uiDeaF{I*_
zY~MoB!v;Jq_9iAeraB^DxMCM?*l<CcgRL#CWz7XaCU0JD*3gsgOR9H$FWqtbTK3!B
zJ0Ax>xaY7e*=hH)_rZeB4|imU1$R#1`!P>&$poQl;nzm}mD5ZFopaX|GsS%q*{P~<
z;WtmO%lhToBL0i}yfkaOt?EN=nkLNGuU`ywhI5H)L`iUdT1k0gQ7VIjhO(w-Zen_>
zZ(@38a<+nro{^q~f~BRtfrY+-p+a&|W^qZSLvCepNoKNMYO!8QX+eHoiC%Jk?!;Y+
zJAlS%fsM;d&r2*R1)67JkeZlkYGj#gX_9E3W at 4U_nw*@Ln38B at k(iuhnUeN2e<LqY
k7avGh0MJ?nL!$~qBW(k1gO=RgwV<%`boFyt=akR{0Qb~(A^-pY
diff --git a/README_FILES/images/3.png b/README_FILES/images/3.png
new file mode 100644
index 0000000000000000000000000000000000000000..ef7b70047158970cf4e09f1bab2954d39c2d596b
GIT binary patch
literal 350
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%U7ufo&CF{I*_
zs^4nS!wx*cCSRl)+v1jPG1#CjBFFQFi+#6SLc)r~1q-#8t4+IinLl*r&Q<lBxJ20s
z#Un&6EqGq|j?0_pf#{;GO&1tq3xr=CoL%_g^!y*j*VU%jzPP<#S6O2E&#&(q`sU3H
zw%V0+F!xQ+IZ3rF>F0kK0(Y1u|9Rc(19XFPiEBhjaDG}zd16s2gM)^$re|(qda7??
zdS-IAf{C7yo`r&?rM`iMzJZ}aa#3b+Nu@(>WpPPnvR-PjUP@^}eqM=Qa(?c_U5Yz^
z#%Y0#%S_KpEGY$=XJL?(l#*y<Ze*5{l9-%ok(gwWXkeITYMGL1l$w;1#POG}6sU_2
iq$>buErX#^g`ttQfwn<Q?(SMpz<Ij*xvX<aXaWEo(|9KU
diff --git a/README_FILES/images/4.png b/README_FILES/images/4.png
new file mode 100644
index 0000000000000000000000000000000000000000..adb8364eb5d21ecdd4086e16110b62ddcb42aa4a
GIT binary patch
literal 345
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%VYpr?ytNX0GL
zUPYl+0|95{iBls|Qd8FlbRJQf7_<1(>3r>K)tuC)r#2`iJ>Prt42#Ndx#Uc~1)>aw
z3jE at Q4|!9Z%lVv}<iEVg^%4quAaPKo`?T5njMAwhb3$fi)!D?aJ+$e`&YIH)e$M>-
zc=48cF7H)t`(Ck`^+mtha~Np7bBSw2NpOBzNqJ&XDuaWDvZiNlVtT4?VtQtBwt|VC
zk)DNurKP at sg}#BILUK`NaY>~^Ze?*vX0l#tv0h4PL4IC|UUGi!#9fLzfW~Qojmu2W
zODrh`nrE42VU(7fm~5G9U~HM3l#*m_WNcxOXkuz<Xl!JXX4<g(M;TBTA4pdK&{_sV
bqY6VKZ3AtCmfYR7pfK}v^>bP0l+XkKd?9q`
diff --git a/README_FILES/images/5.png b/README_FILES/images/5.png
new file mode 100644
index 0000000000000000000000000000000000000000..4d7eb460021e845981861d77614539314f553993
GIT binary patch
literal 348
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%U7ugufMF{I*_
zXx~D?BL+My{9ICPQc_Z at Sf_vZD#0ez##W~t at 077%!Nb3vIq7BJSSC%EKYHG`>EX4g
z+-vfUhb0A>b04=Im{6XiQd1v%r%>h0$G8U7E1If8OQ!N~xOYY5h0NDT$p9(iZ?Q&e
z18-(+l~J8O`)kc}e&uL$eW&>P-#`~Qm$*ih1m~xflqVLYGB{``YkKA;rl<NQre`K+
zE12jR=~*aPTIw5E=o=U+Bo}2CmsC3BRu-3JChMgZ>!p+y<mZ*>CFkc(+ at -h!Xq*<<
zxXkpt#FA2=d1<L;mIfwf#>VEBsYynrsitN|Y01eJ$;p;U#>wWX2KP5v&I9V=1L+C?
eTFYQ)RAFeOZJ=$?lDoSW6l|WZelF{r5}E+ at z;aIj
diff --git a/README_FILES/images/6.png b/README_FILES/images/6.png
new file mode 100644
index 0000000000000000000000000000000000000000..0ba694af6c07d947d219b45a629bd32c60a0f5fe
GIT binary patch
literal 355
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%U7uh!GWF{I*_
zXs at EsVFMl(aW1i0Y+P)=xEuevCGHE5h=|CVp+1G}7w5vZrlwVA@}B2?>*)Bra at SU#
zmiz#bR~{$s<it3Y`rTM7vigH!_v%d1grx^I-22U(a_Fw$ZS^N_t}odCT<7rVX{mBD
zKd$UZU+Cwva*5A-8{Yq66Tg1wu-{y4 at rrAQbR^Ir&Lyr9CBgY=CFO}lsSFMp%9 at _J
ziRr1niRqci*$O6lMtT+smX`Vk7WxK;3du#8#U+&vxs}BwnaO&o#d;~F1^Ia;ddc~@
z6L%@@02-$SHZC(gFR`Q)XkLnuX^Od_xmk*td7^=_MPgdAVX~>2si{S(aY|Z}Vd7tb
nuUmn-_&~Y>fYve?8dVq?X&Y!8wB+ut1%;lctDnm{r-UW|sZ)Hv
diff --git a/README_FILES/images/7.png b/README_FILES/images/7.png
new file mode 100644
index 0000000000000000000000000000000000000000..472e96f8ac36862c5645732f2fff19d06ee11a8e
GIT binary patch
literal 344
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%U7ugKHIF{I*_
zYTsJMRs)_E|1hOU#Z8;UI at i4Mnd-k!bi&j%2e~{_o=nv4^w{_06Z53Sv);B(UT{5?
zH~y6Dg#wr9;+r0~<m{FH;5^$PMEk_4)!Iw=*EMoZ5)E@~%ep&xL34GF|5VE#ImPlm
z?kDR#Pgd>u%w%U~xZhnMEEs6JbBSw2NpOBzNqJ&XDuaWDvZiNlVtT4?VtQtBwt|VC
zk)DNurKP at sg}#BILUK`NaY>~^Ze?*vX0l#tv0h4PL4IC|UUGi!#9fLzfW~Qojmu2W
zODrh`nrCEbVQgk$XkwI at Y+{_8nv`N>YGIaQkz#0QY at Te9lBQ<)awbq0A4pdK&{_sV
bqY6VKZ3AtCmfYR7pdj;f^>bP0l+XkK9#?Y1
diff --git a/README_FILES/images/8.png b/README_FILES/images/8.png
new file mode 100644
index 0000000000000000000000000000000000000000..5e60973c213b37df93666c5a00724f34493974ae
GIT binary patch
literal 357
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%U7uin$eF{I*_
zY~MoB!v;L8{9IDgq}bTnq@>u&S`V$cAh at R~F=4 at V4jxkzlaQrcFYWK{)(`o5XZnut
z=nE4SU2g1ZW%;@@I$>_e3F8a=8WK~|CVXt1DqisQxtIX|`<?0gJCe5^yeVUR&~Vy~
z{dxvlr`UDv_!a+ZGp}~&TFYZk-v41(Jl}GcThd!Tpi`VnTq8<?^V3So6N^$A95j?Q
zJ#!P&Q+*TDGn2CwO!SQOEEFs)^$jfa4Ga~Mi!zH#Djjkwi%T+-^-_!VQc4T*^GfuR
z^K&QeQrrPFP77>YW_n&?Nh#1gQ}d)$LrYTw(_{nVG)tp2V+#}WG*e^KRLdkoLz7g?
pn<m-LKwW$wT>(IA84Qgo42`r6v<+Hvch`c#&(qb<Wt~$(695p&cuxQT
diff --git a/README_FILES/images/9.png b/README_FILES/images/9.png
new file mode 100644
index 0000000000000000000000000000000000000000..a0676d26cc2ff1de12c4ecdeefb44a0d71bc6bde
GIT binary patch
literal 357
zc%17D at N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQ<As43j at QdUS%U7uin$eF{I*_
zXrH23ivbU-yIWYmdRn}l|TM|3{nIuXYhxpPu>NRqa;^5&H%t0&v*|C|wdb9$wI
zR at +N9#RIowg at Uqn&z-__Tzhhz!sG|vTxA7?=O|Y?u(d4T{!RM9c7chr6d%1?R=i16
z?@Ic{f32<?<=Wp~dhLI<h(}0&{b%-`0E?uXmwt-_o#I^L8c`CQpH@<ySd_}(prNek
znVXoN>YJFJnVhX)qGzOMp<rpLZ(yNsV5pE>lv!L->5yAlT#}irms+fsQd*FoSE84k
zpF44v;trs3T43Wc)AJHbN`dAXo0u6Hr<$gkq?lM38ycjV7+5A5Sr{ayr5c%-n;95g
oF*H#D>f!_G3IJNmU}#ifXryhRZP1dtyA~9Fp00i_>zopr09=-KssI20
diff --git a/README_FILES/images/blank.png b/README_FILES/images/blank.png
new file mode 100644
index 0000000000000000000000000000000000000000..764bf4f0c3bb4a09960b04b6fa9c9024bca703bc
GIT binary patch
literal 374
zc%17D at N?(olHy`uVBq!ia0vp^4Is?H1SEZ8zRdwrEa{HEjtmSN`?>!lvNA9*>Uz33
zhE&XXd(lylL4oIh!GZnHecj|txT>yO8>^qY%(y?B;Tppl#t7yOYze#vq#8^aMzDZb
YLK^d5CO(feU_df>y85}Sb4q9e0Be<Nq5uE@
diff --git a/README_FILES/images/caution.png b/README_FILES/images/caution.png
new file mode 100644
index 0000000000000000000000000000000000000000..5b7809ca4a9c8d778087522e5ce04b6e90099595
GIT binary patch
literal 1250
zc%17D at N?(olHy`uVBq!ia0vp^5+KaM3?#3wJbQ|Pftew|C&U%V<=|it5MYpyU{F+K
zFf?Rva$<;zVn|MA$j)XcE at r5%W at u>vqT-$&hMpcE*)wGd!;~q-Q>IkUnZqz=PVt;M
zK*p3gbLK2v%CK~4^3tV1#?q}@8MbbX+PXD)>(;G%_cH9=n|$sZ!?|<ku3cleb`3~g
zyLXS_-aW&6_bl(-b9(-q;rVmJ=g%#lKY#xIJ;VF=@4tU%`2Jns`*(@&-wnTi|Nj3!
z!~g#R|Nl$;9|fa;svv&(lNewgU`+CMcVXyYmGuB}K6<)1hE&`topige$$_U$`s0Ch
zNg`*~o$^fH=@@W3A*td&`{rUN*SDd|<Db9xXV2KrVk3X<i)4bpW%Y)KhQF7W7x4An
zdBXXXku7}MiMdiGR;tCN57jo8>m2ZS_435}A8DL7{!Kpdr)m1838t0{o|PT2YB(Mf
z$Lq#w&7Sall?VG{FZEN57P6hq=7tTk-||IyZ8|2S{r-8<CL^bbi!8Pum_K_7Um*W-
zmFwkQFN3_Va&IZqJeL<V<6y!1k|p<R7n~8?B-wZL-An%YVM#I~i$Z2qZFhOTfN$gW
zvd0eR-`Y+o3fT2;`tqeI(xqRg&u=X2uc~JEd!1uFJyA;x=y~T7*NBqf{Irtt#G+IN
z2MuLS&)mfHRNut(%;anZ6Fnn63k6F{eFF=914D)6qRirwN{8Ia;*!i{z0_j8l+uFy
zyb`_S{M?DV6n6lP(*hfpnVy$eQVKLL+1$X`!ra2xATiM-H95sJ(InB>DAB^y$jHJt
t$vDOAM}X)ype{a;t^lC53`QmuMkd+ at +6G0@oQ6Od22WQ%mvv4FO#neOWIq4^
diff --git a/README_FILES/images/draft.png b/README_FILES/images/draft.png
new file mode 100644
index 0000000000000000000000000000000000000000..0084708c9b8287c51efa6b40b8d492854191455e
GIT binary patch
literal 17454
zc$_tR2{=^W|NoT|WlJhqk}M^AjjW-ulMzK`Y+0sELiT;BCQ9~Ylq?_X6b8vIODWq_
zWHMu4vKvep`_}(V-{0T!c;?=7&v~D7-rMW-zRU9)2z`zt{6`=N;xIJOF^8Z7G2p+N
z^#EuwE=L)GpM&?VnOuXQckxGe9S%XzS;$c5+O2?$rI8`Op<3Ki2S}ZWsMl})dwG2>
z+RaQoSB}hCQyrhg8gX_r{C4RrxPsT$w}+&mK~ymWJ?n;<AaQP?r>rkoj{(`)bM>We
zLrt`rrbL<Pe=Rg$TmL$nLQMKBQqlr9P4=(kxQKzYY;aghqu9xSIvY-F)0>9O&^fQn
z9!EZwtzQvqm4%+{5cl(Vw#E+nha&LzX{;~uO+FK3wkeU5YCfrav7nM1RfK25KrQP{
zrY<6rqvXNKpCl{{y5m$MR%veJDzjE;zHnB>#DE#9_HlyCai;#edU->lQW=3b-U~&z
zt%m;9g1>k&1ry<b`dge7SF1$-9kaTbaD=)I%!>78NPs!7t%Z(NFKuHj56^AfrMW{|
z%ACi86xdtk*^N<nYyt~}om2;l3d|C4N>UF at 2Mq^Tx;6hQo6boSqjO!-^<0<MyA;j>
zNg at e7T#Vn*#GZH&@Eexc!zBuS&m{J^a`!@zy}?{6D+Y=<rNJ}E=beGITvTR(em~+L
zWP|fV<Nbg4#{PbzFRrH_zV7nnp%T<W2;bj|!umZ0Q!2NrS6Z|CGeg67&J3ym{YEg0
zRdO+QEW<N+91Jf%!nVW(HWwh6>1g)@$PM6Dtn|1A^q`;6!vtoJXh`gFVEpdLa|y9~
zg9EkA;v8zdM9vsvtu>A*YVwf{OpsDeBxgLBardh$TtJ&IG;ZS%^qV<6(@|`7Bfu4c
z#{Q)Appi*En!iSvppfU-(UTtB(73nQ>P$d9aF4S&PAx+Rp at 4X7+(8;IG_IM@^Jo0q
z0ccl%P3Xj^9MsahyG4(}#vKU;n at P!yjN_s4LlOL{<{vzvhd$x^yQkZKEGZhgoDZNu
z<2?p*FBQ&nK<!`DeWn5qGmZaeFt_<f;y}x>AClEaKg2^1l{IF7ZAh>ay4T=ppRL)T
zc7|;>4O|LR$lGgceWWk(&l6Z8nExnng?>ZrtgOHh4R8bI7PHNlQTxv_{GNe}+P`63
zdy-y*Cx9AzPhHA6q_TvqCcB4nK~Q{1w|2O;>mHR-rhad-<gl3`vsTxquiD{HgM!3?
z6%5FcimQqX^HGco>D7dsGBZ=n7$EcDQyTBi8I?UgAk+M~HT<a8Hq%$o$Nb!@sQn{!
zl7CJTON5V3R!?-$HUgMY>|7S&9nBEI)mpw5y9fb8O8)k*PURw_z7y*YfzyVXH2!V=
zwN)7xLFR~1`VXR8jAS%iQZX)%vPnT{YHAW)Cq%(yeqZh3N+x+#O*XQjV>_~>@V#wa
zM0oUsJJc^##^x6D14Z$*#i4O%cl+r~3gvcNqoyv<hwK)@Wh+%)MJCjFYE>UBXfZw>
zOGY&t&rMT0=Lx`X$}eZ})JqdV%<hiWm%hk7i)YRQQM at h|0+%bPV%qMQ+i21QlH9Ri
zhk7qFELVg7GQ4<EyCZPUBP=|^)1!cQCxaDet^d(C*E!di;KSPD9MHQL7jsU8Ix`*X
zg}AHu^y)Ml;Ut)@nR>`m>jV<;c&!gt7N?sIg*rDedc5riJ#-Pl49TBePO%Z-OZtUz
zgESK$>BrV=c6i3ALJgjvYgd=pQen_%v_Bi6bOBQq>bwxJ#)$o6TlCrPWge6P;<m+?
z+8S-J)6b4_^d5VMut61|zTEN+?8&K0Yta5cE|zh~=<-0ifN?0rIFxL}p8V*w-JPw;
zHVdQXGu^rTR$}$>8cE9Sugt*+r)c^VdyR7VKkw+*S^?hi7T-|%dUk+%wBWXB`&#V5
z2+ at GRKotbMIASLwUrAGFWP=Clz0&=jzI53M;c~1n{<>>sf31K17y>d#RgmbW7h)>O
zrLug1 at S|k()d&?Jw*_W0sa);W;Fazt$JY2Cu=CDl;Q=kj3TYz!Ln&2pqP_uZ`IF?I
z3!Lm*$y at x!%bueP^8!mW>!VO-Ky6l>+tt;^H+T;npf!Ppnk*cDeW~jh+pNIaYP{dp
zF?PuO+}dg^`p>1V#to3AXratNHPtm!s~gk`zpGq(Z^iMA`?2c{Vd=oHo_&Ftw94Da
zrkS=kl8onE-G;W$Y*dr?_xFPBCr5$Gbf${8=ACD9nvb2u_l?frkMDx?B!H?@>BR4i
z>Vr_dMS`scH#3~!`-i5cE-`9XeqAY^D~YjaS0JEP1aDg3b*po}lv*LJh5*47bj^S{
zf|)X_-Ox23n1#o5RY-S&_(>!^2C-LFdK^{Tg>fYmw^sN?Wmp0UC94WKUw+Iag`UN8
zDPa#loS~^bMsEYN&_CYOVhs8A7|T?jdku`d8k9t8x;LHo4Fk4(=8qF&vc^J8QIYD*
z{wRpxmJ#>sbBE|8>eE-*VPRpi#9pfL at KYmxXqmpU;&mG?wSYPG>8ojTQ`34VZB2Zc
zk!*y{WPK3BQ;uHN6tZAI)5GG_C0#JnB~79K;-!-^?jrah_g_U!R<5=<aXQIW at LpIt
zkQg>ozD7YAUv2F5oMAxW=11mC$FbM5fr9Oxf~Scg{nxwz8b#1H`)A;;nX`devr!{y
zL(fH56B#+vekHM|<dt8s+fXTWCUAZGqV+ch47bc0%-OQO7ap+ux9O0_*wI9YPC`*l
zycos=d3!WZjDiOFtR?(xj{n8ZFc09#n>E3v-)j%C37I>t!9ZZ`zM!x5L}NK+-}xc|
z#{9>dH;geX%qmP`f$Y_A2|d>TJ9+5yv*)eRfBz0$vUpUxy?KJX*;a$WI+SJa&x!W?
zUIl^6$m&n}*LH#62vNtYt3MVi|BX;CiPHEl^Sdd~Slm;y<<>zbhVnSESdH3|X%Nig
z7Ytn9H2yo{w&IG_CBZB#md-IWHBYak-piUbAtm$#-=eI^GW3+79G*8Uwp!ZfEA+Qk
zNrdApdS9>3aPHt|O-(=|3{K_^Z<<Zf0f~r!ZK;JOVG-BA(n$}F9yxn0%Wp3~645{H
zHD51V9jg5{pDDkMRGKa3Gx)27F`Hq|O?Ivq^8Y6O?<Tv{7aT*1PWc&rgicjhdYb|y
zc_&4{UAI?XPn_e<hJlOnsZPcU?u;3 at 1n_7xHsu$CjlBK3o3YjPXsjMIv`jOZ<cqL%
zG*!S>S_q#8n%VpF4$nu;`4da{=L~ACou~UhOE3sf|8X9Efg{<BNk{IODm;+2LJ&25
z;ruNc<#I~rK+w_EYse^9 at 9;7vsI+|Wh}l~zN^<(`Si^<rqN3>B9uzRgGa<HE*1#fs
z*>RM at FvnRhF<9<&h-L*ypZ#ACYJD|u`P|*7abEa13OL5IlWeiJ4=*k6Vc=45eyt1x
zm&J~*jD3*CfmY#StNbLGO8Ca_yeRMO*MI}QJ9t#f#$Zmd%~LSafiKci at O9z9wqM-z
zBVXj$`O0(oOgc}ocG^7kZq@}q)w_)(y;RE^ytdo1mLT~*JIvPD<_ni9m<e32ci%ES
z=sj7xA;U%hTv6vS)^5 at kezX&#dsQgoCn<C+-1onQST+J+SPExzD4VdHFY4W1Sv)F2
zFQ))kbV7^)v`t_sSK$eY_25BWC+Q$OevvYoq!{bIW4+G<av-DmbM1!2==5siKS=_)
zk#>`9*po5A9_UUCVGXqQHwd&?g8eKT&FSLMfC`Z&)<Wf?(#BH%iDxUz8xG9c<eP9}
zlj)czSh><nT)wDcpb0<%4dB_`v+et}tBLQzR6iW%VQ7=}t3KS7 at FG_k0WXWUa=LC|
z at sso?FNuYl6Kq{ME_6)OM%zAF<I=+XCIG+#A~}=<{%x;1^(*(ck4=s?Xc;iGKEm3y
z_FdReBf(5at9|6uXG|g-Eo!E0p>Gfh!r~rTv$|$te_59KL91N2q7MnydH8FW-Ll72
zpq&Ny`(DN#HMh(%ek^a;HGq&~IA{A`63h5eaCSpm8_cO57#ixE)ng<%utryATPjMk
zwsJL`3k9sPc4K?{V!Mj6Qulj>=bE)RGj|?-kRu-g4)Q{p7`C^adjD2%dQ>&JOXa!P
zYE%AJnP7|MOduH*Y|?{Cx3jYYiD`q!`VU1?4^HC+%M*9HZcEWMOCW9JrU?jxxdn+8
zF&Ry#*kVn=$*?G%28T~FU|0^9;Ww1Ki#VVnbI&@lXOl7bROdW^f%!7&d)9T0B_-7t
ziAmT|HQXJh->&I}#OX`yPbh*Tll9Vib${}%?mb~C*Z*o7-Zd8F!lNn<MAACu{R=UU
z?L5PeK_Yx4zbsRT$-T+06umK~5b^njsT2>tKk!zp_wNje98Wpm`8T4zVGLbniwV4J
znXs9BrU&CLbkfhBfUoO+{g?!MauzW6UDRWY_*?GA-7wu5dZ(i?m(YU&wk>}(wXuO)
z>(RLH!n1uAIP(MPzM!Df(|-Fe(J4F1{~9{CFg*MSK)kBctPsgP4?z{c{nXhj9UG|)
zRQ#+%z1wYQ**^@N1F{lPyQqXmSDj$vh~isj;XdVbodjzT*sw!ni{(skgRoR>AYx1#
z!cf3)Pe4iX*`;%Sb3^4%4fUQGuh{BG02pWO{&bs?Bw5OU*P9(pCbhjBYYrcKO%j8J
zIMoVh{RE)aO%I{fDzU{vpv$`#<?#>ZdA+lGkhsJiE3i(#7QW-%fk?aVsF?>+Hckls
z7Wacx(crMK?-h%rzf|9p25Dk_>4 at FQ21XiGdK?=JKr(DwrR65v+7+Gh8r!A at pu|sa
zVZMphsL`ZBCT00aoB2dfxc*;rg*5X|5(y^rcV04Qp0`yAsA<u>V+&Kjs|dRS2tj|b
zyb7VJw5W)xs``cd3<&CWu;?8v_~c2TvHo}HutDlJX4b%E_`=vW3v}m1C#E2U=IL=8
zw8Q}`Ek8wSE*)fxZTP&;1pxWo3x=J`0{$v8x7eQR2G-zu_9y!iq%JWCF8p~}yD!At
zVsvZ<#ugj(WhI^uvUz?HZ|G9;Zm;<c^w7O?-i>0K+++3Jw- at TI+}Z}?sUX}z#jplr
z<2VmgC^_Ujc7un%6zGoUlrk8q{5b;_vHCP4`#U`Y8z}|0%_-7fU(!UwV^+3#lsQ34
zH^X)U!eeh|*MU(6TUJhx-y5BR+58|65@<}&<Mi1&H|sGvFrIyL7#cZ46HibF at Ez4!
ztF#WN(L<dnnBODd-R+9qbbx0&S)LxBpM$M?o_1QOmn9|z at p}Rn7aU9?tp<z28TIDm
z=rdjb`0~rK2FDmcuM&IVSg)2%o9AlifrCwOgy<EZ+hZU at EbTN&8>=kekeuFuv4-&8
z1-`bov*9s=|8o`2|Lm#Q>PmKC);*-9MXEyN8_pPwOFCddxxZy|$}B8WXX`T9CVXFU
zmnM=J$?E})0USuzDzdH_BT1bw6Jnnn?H2 at B7I&~GlSuHpC85IAtk<r_x*h9n!MPjG
znf|aj$*vO10Z#9pLAc~pU1>`}BY8oF<|Xfg)q at hI<%*@ms-mkK35LWOI+&n#v=C0m
zv9NNcD7HrVwvHkh6{*Jx4`=x{^a8Lj?(}%9oxy2aiZ2eWYj)v?E4~AB5Y#u<8XFsZ
zZBNZqIX_Z#8h1}5v4Z;Y6Wgnviu**qyeizM^qEVyL04 at 9=117;bEby3Zd$2bLx{4&
z`B@$4Si2)s^=`BruVjVNhl0kuF#%ATNBF-!zp7T@{fd_ym?d?Ds&8i6ss!i%Eme&@
zY3v}QmNp&u9XB2?f}a3D6~KEB-b^nomnfp0kj7_cXQQW>+kq!EEuPCMGqfJ)wTT=`
zB>l!FuzcWa|Fg8 at mR?Bkm~wA>DNQUY>Z*vDsbE}w1TL48yS_WBae$D~se)mp(uwX`
zh*nSo{YDG6r=_Je{(aUdd!g4}&wC)g15*o%#HQ<{pVe*@U$%p<V}g}TM4wEs?Z)nc
zWq||}@Y8o>XlTeO+|t#SS6BVC2O$-UYrHYX{Fg<A<2Ea%B at dJSeU>iI&b5nlb+e at a
zAC^7kN}#mwHRj_>N?o1UzoE0w;U^A?aBTKf6Y>`u!+TJ#XSe2kF14|@_H69IO#bY)
zpj^?z*14#-FP{~?_^HQwBkak8n&JEE)hCV5l^0zkR8>_|>mB&oJX7{QWsEJfI<woL
zK4N4WBvh_~<o_iAMFnAzNHPRI-ds{;eqA%<DQJ-KEyAuk71Z4}-!S%mMNV(Ax?$?X
z0Tkgb*v^o?+fG*x4+^RutwHA2Q|mG5=cKLEo>aTY9(P#4c&Qp<VdI&+B)1_AoRxVN
z4vmRHhM9F$RJ-9Z(Qc59yWUAFBA at bk?F3Q)y9fss>;)*QRnuMg1;&ApS%~3_ET!h&
z|7+LLQ}ci=?Dy|GVf<WY^B-}SRC#!K0Qe7iT2z#kmDRM}yfQ}v4D<_X!NWnuX}tHK
zfB(!SFaC{l|2l_`0d5f${wqeogmlYd?VIsey87<gT;{fKXi&RCdkcE`Nk&^NE9Pv%
zzx>hW8B+U9nplBJr at jA%k`iWh#`t0n6{~wYLpDdK)Vg)igpBnOnh63AeC0smVn{&R
z4Nuj~h5Dvx|F}tIlbhIvRJOnsUfp*(YdJ?;7cj%mKpY^$J1WWBTU%RgOO0Y#=rgxe
zTk`H7u<wwauDDNCejQ_OI(ZVrR!Yg2pv`!>;Dix19qAWEl*PBd`Yy4^-lQ456^(h6
zOxipq!f{ZMy>%<Z?&kG-$o1XbU9B$|LTc6}!>2AsEuFqr+?U!?_zlv1%J$xll1igK
zLul1{h=#vy&CC%rQbSWy6IVfmHr2Z)l{G>H>^i*-(_0F1u1<VX{kmIu4!;4S_Hk0+
zFW~5ksmbJRAJtU!)<Z?<8WEH6A9__CboH&7OAD9~fV%Kx?ZVro at fV5ah<72IM|yRx
z#7(Mq&(e{5Blls5UK@!$wBePsA2bF<X<^xJ3J(lKzJk at rwQv7`nN&Euf<02JE99c6
zrM;bMYw5&<=ol~yBsd05H{H=S19kiYM(Sx5>=7{~@GIA0r?0*ajcZHFt-G*Mk^Dz)
zB{nY#QhHIESFfg`=NE9;DnCZ;x6`yCbV3S4C}6VzHC%6cMw=p>zrRhryR58C>UDLD
zLUZ`v^<#(%J!8LX&;8E-21(G|>K+oObt|fK9;6?c?B=wVppcjYSCVJ9<k_X}749^p
zeZP0xUwmr7mQq#nGfCrNjod`S*SGevY4xSz<^G{PxQw&1_Q75A(nK3YSc}v}%$>`V
z%1s|Wl<c0Cj{Y at 2-{~)5LV|U_|7vBtChUFiWcX4&*YH?<i8L|)^`lFwVIa?*e%?sm
z`f$l0y%2dsF5H<)!=!IFr5RE?=-qrbDOHs(-_~kcgnVfS6r)iNV9)Es)>KMVyF9y>
z>eJ+g?dbjD3rC)Fs5nXua&1 at a2dgw{Te>Yt6{P_L*MF7R)1WO5-~nX3Z?ZW?lCDd1
z`g`mRd2<Unlkr4NL+6 at Hm_vZT*qbdCWl7|00B@}pi?Yk;o=x>l%BVjUM!Al(4Xv|r
z_VvxscaWVL#mRgWZ60!^dJpn>t7aO^jX2Z_^e3^m-9=Vk%13*8Z)!*es$dStMZ?ou
z^7iKBl>F*!D#@W^M5Hx}y6NqTZdOuU4FiZTknFg<w&pwSyQ2PgZ>J{NI;da0n~YL%
zH11WMF}L_tCvE)dNBpjPUmhR^jjljZjh=$`_V#$ppziC9`25$`{SMG}?7obYq6v+0
z3mvDd7_4k{6u>aAY}yM!-Kw_jym!M%qSb at dzI_>ssjwlC#*#~eP2IaO1MhUmFQV#{
zl-o04+=o;Y1JOI~6klDlgYNYG$OjxUmzT}0+6DXQ>ck6s=2mIw(-VA~vD6BKIoJ(D
z=64S(P8vtg|8^T4OP-D44W}x3uxZDANm;saZn$o7Cnz0 at U`YjHUEM4!TIEaLDprv_
z&#=m{oxf)>9Vk15(`~9p at o(qfv$J&5?Xy`oX|8!tr3c49G6H;HF+n`j()II|%X>f8
zj#>Zl$4M5HRPpp$SQuIeb#jE4;A0gS<$EV6MI1#=3~eXiL~|U&^yKs(bZ+j-vtPn;
z8?J=rSK&nUoZCR*h8%BN3vDdJpPoi=fh8qk(g&-T>fO;ceZHZpnKmP-9+#G7hH!yb
zuMLRIGk4Q%(kH9;0z}qUVJ3{5+0Q|+)hZ)jpko4qf}(SNHe|wXZ-kEd+J3dDO6bHO
zOT?e9ReuYrNUdM3bqf%97&qa5Ru+wvy<EkW-0}iXoIiPqA{G3?JycFJS;^8U{4{gg
z`~tz!%}_3A{FE*fa`REGfl1+7k(sRP(pCtrio!|zR^ur+y-5 at bt-cYigb;2w<NN~@
z`^JR<fuD6<ZE*z|7pi8?8A-nVz0MQRD&O&W;mqv#lnL&QT{<yFAux-!eED$hCty~g
zdFFZ>S*fy{ykF>ax-5?)g>6sL)$LDO%)P~50+0HEC4V4iPTfA~FsbZkdz(_#U`==A
zV?kbST#Kz06CfG{KN7T;34Y=cYTtkQBIc}KX*A&4*;!eRr4zlyIhx`GmI|GV#7k{0
zo}!G)h{5(0^U$$0wB(Mbp27XOspF`<5oaGh7B&pBO4s{zn2tDyP8Z{5dqo&`ZBbDn
z=G at mOPk#1fC*^ny at ML`A9|K1HWns~`%`Olr@(F{PRjgc`e3$loDKKj~<roqW!V^yw
zHFU)}K7+!ILZPIpp}V$FX5X&MWjb;muV<7IcubO=%mDFA=NIN at _hv7Bj!a<rg(fWR
z?1M-{H{E4GhF~Yjo+pTp<@PBX9 at IrM73zDcDzaj6!6M{6ncpcow}bvcni#-2_^BFz
z;{8^6kP18nZ);S%E=0x|4<^vCc2=&_Jr|!ZY$`D~OU(LpNIE2jRDQAqz5M_(+foQ0
z46<hNB+SrKW~i8t8Shxlm(D9QZbMzL^5Qg3tt-;S)O3@&_!!z%ZZ(Xs^#G4}yGTQQ
z!Qq2Ir~6k_j$e=KPvX$C8F_Mh<t-fXeDsGrGz_r3Uu7WN+E2fe9+G_W7rH97_~_`%
z9!j&lD{y4MY_uB)Rf+MUMpDSES+UhbkOdYQkY7T3ifc`e&Sl;0 at oze_J&HZKQuF<9
z(uo`9MXGgDHB-;xaxYk=`92KUc_frD^5J%5M$AT}smm-iN7!J_zd|~hq`y#KT4bc_
zw+-Ne5qdXZaZJHKd`GsctWZ=u9)G6+t7onqZpy+}hyleyWD$x5Vc^0FkS8iIYix9s
ztYp{cI|4I_`1E5ePF$ncxbRKE+}K0ks;}C~i~_Mm!vI_9*07?AqR!6F=3dAc?z_F7
z+H>^z{a3{}HzJ(#NpjC7NXvdsS+jt4mDm^-RN<amg&UTWuuT%$tL9cwr<u>^{E^&O
zgo4ir=#|zzz7 at THL}!+bP{o<iWAs*<=)98#777hLPw3e^k_sz7|38Sbf8nc|Ii-B~
zwvq(To1C#kZ87vw-qS4p8Y~F-q(DhB%lc=&n-+c3UP$j_hM9p`Bg;oK^*HwaoJlbc
z%pw*M0<+NU!+q;+Vz4HC1Lk1I7|*1V(6Ljx6Y}|_;(zjD6}r2-jqeR~xe*Oqb{U%}
zEGlZEQRI&EM}D+GcvQ*x2bfzC;q=@KI3YdZ+MgBFdk at 0(bpXBaYePxtyO*5y>_ZWm
z;;2e2_V?!$Pt~1tgUd|`CBM&pk;@l(?fd5sR|#p at 88M>}x!!Ch$Uf2Y_W4HRG$ML>
z&SxcWcr2NwYhht(YKkWTHCT^Eq3zvy_%(AI1p0-MdTZ<J)hnt<T&D;rUU&cF at oiD$
zEg6e7<+$vo;>GRf4f4Yri}7SsoMBcW#>7x:_MLk}LFf_^1kZfVq6I8D3D1Zw0L
zM*RK7pa7up+-Ic8dL_K!oVE0<-&eXc>8}Y%o-4Kiq?!h2f=o1=e1K3FcPrRGHXyl2
z3H7}c*VhM9``gB~PzO_1%$>A+v?Q@&P0hH*O)9plX7?`7wjGhLw(3e>v*_-GJ$w0W
z6m=~M(Iwoh;1jwiY00bmiLEi=%3}iUz)q_s|DCZ6LDd1 at k@)6zQfW4DR=bwln*&-u
z7sCg){c at AhuNK3qlCGzR(FF`#?i~MWD0+?uswn9|sdc~4xF%-pDkG${@|27Ud)<8V
zl>ar(FsnLo{D-05N4tOiZoiC at jed+}3DYrfGB##0C5Z|6TqwYiAaRwcqrQs8_hJ%9
zX9 at z1Y;`w(7~JgL4t?KHN at P65_HRbOnFSm(Vbvl`B`FK_i0YpKCDOPc)MKOjMr$>v
z7nDn?<mBX}{JX2kWM5m#+FH914}bs3RM_JrzB7^6t^``;q8!&AJxr}z2+WFdNHD2&
zMla-?%tyJ>)zf}Fi;~N&THe{OPm;)a^v%prRPfc>Vm$&r=-OzFo4ynfjQte!uWn1#
z{7xSHm%d{Z#9-XaaDEYkcT!xT0#irF|NpxHlAE6n-urWV#iFWe-K&nlfjl1I(7=7X
z6v2`}nN1UWIaEF3d^+_yO3CDXMo0p1W<W_O;ZRawC=WlQMsKb!r2>F8B?SQrJY?z~
ze16}`70_atp(LyhoY^(s#_EnXpgC5zkIa3$_moiO)m2t9(pB+*ZoP7i>}V)w$L>rq
z<>3$DON9+5pXd%9E3^MVee%`(wneHPqMN?Ey<NXm`}Ds^7II!vq!W3t+HI#P9-kGg
zA4J4sjN~+96f7*haQBMA#t&+0KGfU(WM^+bow7slm{p^z8^2CnRIguKs^x7x;K{@P
z at acqmt;o>x&po1kY0^XlK3U-|{f_q=x1nmcnjl}$;aT=M<+E2ED=Pe`!FAm*lS|Ll
zOi1L7Uoj7At2>R8dm^lN&pHr11-oaxhpN at Ssl3uLmBhaQAfmo|_B0!Za(g$A<>_{_
zBXb?+bDEX9y&Y9-EiA?}tqojS4u at s@u65x1SN^j3ToJ>OD!NhSX`(qyrkl?{Jpmke
z^4M_$mqc^KJ)|YY_oWAi8V=h3t^HKLW4nMr`_vei1!m=^R#ZqA7j<pegJgvhXX6O?
z)O}(ZcOQP#b!lkGnqOq8q5ZX)dy7(7<-_c1x8n#Gg>E{LNG!FU(6O-a%1`A2-T-OZ
z);^K08r+tJCv43(Qb;IQ)2pLjwV54$Ge=rycFz*0E7oiep2OeuwXLCMkWo9z`Z})P
z)M4B#nohNb<5-+qOIb6k>H5ZK at J{_yW$V#iNfzv9NvlodODs+r*49C?cXf+-NB&NK
z5c6s_FkOPMrr!P05{jQxW`EL*8G#tg#dYa!TG&j^qZU at oZi_42+EqN}y>v2^@b|;$
z3+=FY=@&_6E-EhOD?|QwYTR67-ckpt#Q~|afk+$x-R2Z>I+g`~F8_~E8Jmou4aGNW
zwt}*XHw}M2fmSp#<2AQ1ekFxjn&<o?&yK_`)nwru>uEb{9w}ATv>n at UXQuV1c^ChP
zhXdHa5w2872iNXd?ay~jybK*vYy7w&x4ODoNp(*0ZrW+G*Zh6MqHo^kNZss(ht61-
z#;jb<bd1Ejs@>m`G=<+yw4Em5$dC2`yOi)fN?3OWu4XPoLv1VY?;4d7)_x))6x0ma
z32Qy>%-yKhXe1JcM&3ovEc_zAth at d|?ex#RRSV~r{`eF5?6$S{g{eDsL9Ar^BP|=7
z3a!EB+G?hG^`#)A&--`*0?<`aLLDUiR|zZGs`Zlhnu<&|j-8yFQiVXKn$!#!yt(M3
zYGGl4E7%@loq6)r`|tkVkba_4TwXHEJi$O=$@z0wy2NeKSqNa*<XFhho_{w(Kv{_i
z at AA)21~vOYyZ$36bIQ(8DI&S~F|5C=nKOcRtatO&Y^GyH<9pcYLs`vV>ARFNdw2Vv
zs3G5x(&SE*l+YSgPMkVMwm;mm!HT&s?T}(!?XQuEa5yp-pP%LTl4PDvQ_W8cbt2mv
zX;dVTB0R9L;dn3N at Izb5P$?2 at 3&S6ED&^_*hBJSzv?&LgI&&!xu(<E_L7 at 27P@Pru
zVLZrtkkzw5yB5p2(<uJR&-Qw8Gl?4^>sx2TqwFj2?6gRz%Xe27wlC`4qDUPnsrN$5
zA4I0}ii_PoO`OEDvCBY-FMb(Dn<&$GU&!P<6_lG;xKCtCFO{u*4rs{Ks(N>;{NmmS
z<aqK&E3b%(smp+rXf^gE8mX#&bAYNb;Q)+t)5wIB(jss`N2-2S^1yExm(V@<(I{fe
z)ZF#+X at k)j%JW6vtPz~H;gXQB?zX{G&F>r2ao#CA>6J!k<c#wjOGKwlc%~cDm001k
z(mn1~$K4xoK2gJB!f++i(caTat?=ElLG at j4->OCrI+$hQu+*@s0lxosMn+7X&G|<e
zxamb|SzZ;6<L2=I(}@1X at qr`fIT=r7oMMu%Cl&MtXX7kOhuSl;-HvD->pgr8UflfJ
zv9x(2CHGCy#q+N3@?REy6L{#ZWNM`zgSAtxyT>*Ui}aiB;5aX=HWPTJT_LvS(6jdM
zXDVt>vF&}|o_BV2J1{Z==X~X-c%^3}^b(Do`?;;gJx@`dZJ~j0L!G`!r&n>dE*ZC4
zR{o%ALxl&-2Bz~))+qnXa#ZrVB$L^4SAyn at oSAz6f`Kvezc`dDDR`Nt*}TZO5Yu-8
zgznpzXoO^Rbne62#z2a%WTQ$;_4;1R7d+qeex$-(G=g|nLh=NOj2cM%)>+{EZFSq2
z=?2y($Y4&G at SOYkYzhx*&rR4{QC*&0o8n6?V+Ol^E|TN%PQr0y2B}^NFt%ZE$eE92
z at B`EN-oTZ|*rp#{A${R>_1Ols_d{<R{g~vaP5Rlo(P1zr at QwaKLesfc?cF65ShMqn
zLIKDYZq{+b5ZQQPE7+=}x_(5x+jZak(s1VJ(76?L|0?4P0trlr{H+odUQ6o{L{L2C
zcJF+w at 6jmI5_zzcY95*wamPIPh%iUquz%ttFGWlrhvrP<)i-F`LC7k`iMoxp7_n(Q
zi_dej=Gm6@>Y7TI5_H%lPvE&#J0enhLrw}`1K>2I`r*zBn+$q2Jpp$)&pb37t5A61
z$n9+1`cfyEljV9{@vpIoJ at H7*?fS6RicQ0z|B8#(LQP*Q{Vq?*?s%)3-7U7tTv8<!
zWjC_yDI|lGUs at 1um1pk^FJam>+{(!io5>NbUa53)bzAZHkt=O{d&#Nd?l!CURs@%R
z;9UMAv*jp)$7vGkwuNs-ZaJ7ad;pVfb(uq5;*Y+!k-?l|AqI6>#EtN+LVDsvFU^JU
zB3HhxR at LjtHIH$HdCg3c7Q75pC)>wvM~SvHoE1IHd9N_a;h~r0_>JCht at 9CBJEsBk
z0~7Jvyibcm2^*grc3zT$1ogkaVmXtre>sRYH*II=I^sLh)#UCJQ|u(@|9K)&07b2K
z8|Xh+OA$Fk+WAtNO$ES}x`W|!N-y;E)l at ronZ&7?$=<4^z%e|1!F1uOV at hx%AmfLh
zT-OzTPwND8FrZ+K!o)d!fr~nV8CnNfn>~_QVwLeGOUc-Bku;Gx&i5$eQ9<ecwp8%q
zBWBeudXRm7q~FUPwrsAwSO-3`KHQd~?|QmHqWHUDmi_#c)A5Ibhl8AHPvgy&Z*@oQ
z>rs4X>4DR_z*%bX?Ff(s9-c~b;8XTut{r)lHP~=AZ<tdqpvBC+;HNx0f`3`CT8Wk}
z7(2aiU$Bm8cv*{Uc~eDrTH^s+4Y?#L%8hWx&={f;Ur1Z~`zoGwI~?SvAICrVya4%N
zTChlA=7yT at Ul8iIRjx9g?KsGseE5R-YLk;5#}Za>s%K+w8%h{VOt3T9d0A|HLid^h
zbB%J(adi|(oP??eu+TTyEXJE9^sNtJ7c8m?8jBUhrWgXX-tO;PUDpv<taP)Vmy_1u
zrpmy}#qIfO$M*o^L*5wa?st`bmjU at cO4WE$Z)E^IeL^f}3pzOb?EDYhV6|@pbvBo(
z`t}y*3apac)YwQU%OO;OGIIMJz>a3wc3+$Dvk8A~!aP-edTogkPwk*%5+x(rHP{k+
zT9{eqzxmkC`dwXBbY1fLGnM_EuHM|#RBGSnMEFMEolHz2c6?Uv?oF!xfU6>-qy+AD
zo$}HIb&lM9LaYMic>T)L-LjD{)zX6Dcl;StlU<~dBk;f3z-14os3%jEVwJ@;@1%(p
z(tTrhEREJe<p!ix8hI0G3p3M{X=;2npICg^W$Px%l}B1(3j6o1zn6XRSMvdK`k#p2
zw(3Ty#Z69oosRrCR-C^V;`6%4!a~tRbSw>>HlKRfEL=BiS?qPHdGB;h%=z|=fqzdb
zh85jCRWr@^PO)z48F)xuM-=O1C%GR-Nm@`C<rw*6lgx>_&+m1rey$oP==1b at z3kfx
zX`22aa&<yqBj=#i^ND4d&dJ(R*MbyE2Et=_C=f1HkWn)*<{oH!+9}PH&Af{^Gf|;1
zyM{a^FuJhxH=5_n7p71DC5>kNB)JU*T at 5%};OQaECPYBr37L0F80nePthrbrmqy%p
zv2$~KgXO`qgt!BAQjOXzc=LOo;!gh?L4e|>DLR=uN3kvbBe6>K1BTzrp$Uayp{Meb
zqin0efUK{qc=caS!PO)6zNGDY7o=3akR#mV8QQ<pa3NZ~TXRn@{|}P;*}+sFoj3Sq
z|0(`$eVRlOu_A?h85t{cv{s0vx{}h^)YLSjb+`F%eT1@^p*}WfTR?1eqm<c-zr1hk
z+)hwvnnIf)_0_><g}dp6)2e{@3#R(`Jnkc at p#;2tn=Pl7iXsvJCDHaPMACy`<xvg+
zfq{u-hJb*>RrO+m?cbA|K76>Lt at iG#IZ?j$x~(yFfLV7PX5u!q9@?B1ZrrlW-J9r`
zef%0cN<_R$$JdN^!GpRPPq^o0zz*m(+ at SqMP&$?qgB%loE3L0V?BA?-)Y;yGbNY}*
z=wPbVHPBz`P%(jf%mv-Dk at U57CjQ=RNY*_!P{C`!B_h<|G_%T#S8mDnH<Z?`_7O4~
zI}MpyJwDOETuMyhPshf!H>}yV%G+dDv$Y7rgAQ}1l488{5ruEi!%r{q|Bg)<1uhF?
z6Kd|is~1uE=URXgE(3SsQr5qsOP>98#@p*~eT3gNk(@_&-sEqHP0;a5hAv1!_3roV
zLv}U#GS=qAQ$BK1_g(HB#jbuAQ&|=&OtZ$F=oB%#lzpF&+NVRIK1utVF*~p?zHuck
zZx>J5i8-kV<31zOsQG-ll4j>_=y)1$Vbn=knE4g(-1)hiin(LDgrPgzil<;>*3r7U
z(4|$*e5`M8aBAB&W-M>GLJHKm+huDM8zZ-^fy7GOQ$r)kE!B6A8!-H!&O9j6=oG9e
zN3 at 3%FOF77)787_TCKV(d1BuEbak%Qvtwr^Kei4(@&4J<Qq4LaI^==aF`i>Q#!TW5
zD*iJ|eeOnBlV=CDo|jaC^8?NOcb at ->*T&%Bub6PQ%-JnE{e5wwTa7cj4I)2b;!lCN
z%qO98LzlBGSf3;7siZ6(RbqvDw<-SSEwhaiY!94bqz#4J&3aXnzTE)5c(6X_){{tC
zme!r at m9S@Zjb41j>(xFn%~BQ2e+vv;&Yr>gsQ&84m at 4|1@#>o0vZx|q!|u(q+oQUj
zHG<O%q3~;<Qa&P*$s1VcZ%f%|q?`)6V-knQ$`Z{7Kld8U`HPiTkuRNTs?3rRAk^+z
zC--PQV4eT at 8EKl_gRr4R+$AjS*hMcEY}(N^5_+_-tn&w-zQucl7bf>;*wkco-r1~>
z*m)#;AIx%m=&^*sI!9{R-gG5{Ze1v73a?1+$&x01ow`+7(9vW!R`qjWVd&gzEK9Dg
zcxDOT0){8(IPGr5(}UI9J+cTmvw~>;&1yG9a!+DH_8;-00^5)Qj#w;<qn^Qe_hOAp
zJgNu^;obu6m;)#yW#Foz6I3AK)Vbjoye(?5xB|I{!bmgTS^6t%oG?v{Tl3<49;>CP
zD<L!cs{x!@S>k`^YJdCn!I?CCTkF$QrL`C=LR2(A9&Hc8hFMt|zWop=Tnq+r_v-V8
zgQY<KvCg+ftKbb?Aa3lc=a*NEi_)y~+Am}nZMAv^!Xrr}ucnDee>j}EMeQzCpk-YQ
zPp!u#KUca2TJ#?rjSgpf?=goraDg^?TSqq22s4bKn at 2=q!;flef_Yn%UMBAkd)@fF
zc8WNG5dQ0N3AZ(0AsyO`Q;_KP=HZ8$;Nk>aZR+2Q!cJ6iRaPNp>%^K;e5?YBDovc)
z5O9i7Fp?xxDd_EwkBXk}uib!|(A7;zl^+D5 at xdJ7IN+c)Kh?XFP9}Y(#cAW_8ux#t
zH=QfU$=^&G3+9!;SCP|*x})Y^{E&v~aI$CCJ*VpwUlNR?_3WW{KfCNLT8L%g+XAKf
z9|%>p6q!FZjCUFWv*h}!qnZ=sJ}%5xyG>7n=-IdkmFq`YJr+OlTRDzA$arZ46 at 54S
zqol{8xkVZp6O&lh*i>rmKt^K-=#v>4ED^5HqbkMT at qU|BHbQvtB&_7>n(;Xm6`}Mx
zTE{vY9}}|O(<2$(B%>-*h;BpQ>FRNL(RW)^h?m|9p*e%8LtCd+IH1=Cm-7pJa0 at Df
z&Hq(h>89sia9&Vd`IHVu{^ntyFQrC0%+(`wZ>olyF3i`Lih5)nc}3DSTXc+Jf<73X
z&qt5^+zP>oTUd{MAE_kkn*DogqcU5j0|T*2CPt0RvxknwPu4DKRjBm_YOre~ae-O4
zElxNb4;c4FMz4{eIJXT`EzFVN;9z4X11(Dw<5q(<3085fi>$OJ%n);TtaW5k86^3|
z$aiZ?$~g2Z_6Yr#<_>L*gepAgxu*vdRW+G_!?Jo?ks>R}c#>o{y+5c64E^mE&ZY6{
z?bU2Xh3L<+?FR@*V`FDvoP~v3sM_C?55hg#8kq#i-?M@$Li?w^{*2fPa-D8Vd+eKD
zNSfz0)kbz~VNS1z;F$!wvd#pO2Oaoi(_TCf_Jh&Y?Pb$2>CUe##W>I(z0&CwkYJB{
zmOovcC at bE6-DRyN|4igH{y}PeoHH&x%me!EEX>sqm^D<L8Zx+TSA9M>shb|6ni->@
zz#b~6B9kYyZ#NgBp=6Q=mfc?dWZ9i(TdM74A?6om7Z*k&PXB6{&f4pM1a%fguA1B&
zs#Y7BzYGz8qgCVvQG?B*Ggu&eF6aEe*Ycjt(mHQ=TUelx?xAu<hio)uX3>cxi034Y
zIS$*Dc%9kW6*#<fKIdgkLRI0IfY!>n(E2aQJw$`n_WhCovXoRon`eq4(|qwDZmLqS
z$*@@oQkQPiAChPH at bLJ^gL1U`Wyx9D6E5SLSViu=DP!M}oj$8llwldJwD+dyKE2d~
zjAm*H`f-&an?|f?yan;Jp^-*!oRN14sn`ZikF^?ioEe&NZ;5AUQn^dRg|03aeqTF{
zh|an<ptq{5 at -566O5&wx%Up*O*9wMH?%0`csrHSE51ZeY`}=CPE`@+{J$*l`(6BHn
ztzw93KP6>+?#y|-T-u9}YY1ioLmkt~YBz93!;(SmW5I|y8t&9s>OI<}U`%Qe6$Qq%
z`1#ZP6Lc!cCw0M?7KEFoy3_}9kNHodi%KTmR|gj3c}cbuLsYR0;=}if5@}=1_>tXS
zTDj~|=nVIl_tIK$;-y~+t<C0zM6ntjUyAReR<HEJCYpGC+zr$BvPZQXtHTqv5OcW~
zx++o*eV%w?#7{~uOe7Ug at f9cJogyX<9qz>F3ZIU^KG@}@k4Q?q)^96E7qV9iI643H
z at B9vNz9ju5LA~YEqF?>tjQc at +$2;qY at bu2&B2XBTV29NF=n*R4wlNS}e~IT;d$@p8
z{x#_pJe?79{<fzUPfQEOee*{B&}V^a24*Iv6>|U=e3Fx4KP-b}!OeoQ-N9%fuaCqi
z$WCvtl-moa*X at c|^7aUy($XjLi3S%LAP)Dlt-Vx{u}>yJe&<)6s9tpegYR?@K7dvh
z8O~Yr>WXln-G(ANF at b-%l1R~87^vaI>ML=!CDEwpT;)uJgP-J|Yy8bGasiuT>)E{8
z{eH0wj$qdM`sr%<uDA~p*WKOyPRPJl4r#{tOy}Qwp0qbO>TF<GMyPft5O;hIGp&}>
zcTymA^G*zs6U&)E3LP`nJ0pe%%xT`od<qtk$@=HbN=grkntMSk=jJ)JloPrW!B3M^
zM+1?oiA?>cXx);gjl*6(pUr(F?ixzF60V=w at vh9Uj0tA$=W2faDxfF_O)6!)-FW57
z3L(){rOCdy)b;e?QJeBm1Ok84j2BGuGB5<bRh$!_(6d03ZiqR#`5mJYqY|0Hd1d+A
zE-YemD at Z*M#nmX7u()24coavvB(%I{;ePBAewt>CV1AaIcJFxJ at OMR%_Ybbm%!`*)
z0ob^5jlRlLb2M<jGQ<RQN!U4joNdX_!h+{wmKVzPYPUDV7exhF_Th8m#_>fP<E_UT
zEZhR^rs6z<Qk{r0#%oaz!cS^D{-AM;kK1S at 8O%WkrhF*AZF~e at _SqEqWsdQVjH~em
zb9X{!D{TNl<AbC2Mym9h?hXsI<kYIfJ;rigM-A_xpI5D~ttoz$q$pjC&Mmq)mORa1
z)xv%Df-Q+E0bR*GudNp59r!{Pc_qG$dTG?qUua5bUHw+86!E4cls&OW;3Co7GB>zr
zH~)D_QBeRVgl0 at o62y|&^O{~SOGFKAG=}g#n at v#g;&1U(w-ZSOJY{QMni%q+Rl|g|
z6%uzo=MbF)asK*v at ewx82#KSs?+C2A4_%2|f|n_1a*VT{d7E!I$J*z+B&Q$c at H@0(
zjX6JSz47_m2iIt8<;u`z*T@>vJ!@U6k1B=`n$fBdH!05^nAJ|YCV>m8r>Y85YRFei
z@~Yz8T7+A!4R3s1^RGU^wxmOXWsT~U-3u-OBoK+qT9_}ybX7Q)2((x(2W$uMPve}@
z3|ww6)Mw2Ow9i#$B%W|^3 at pMkzPn?3KqVjMruPKrRN*Rm0}-Yqm`SpNMD;a~X;nPq
zg;<vIy4P9P_IXe^uDZs>k22 at Ym1!Hes~wKYUsi5%dZ%MqxSwYmpDSY%8v2w4GeGPp
zi)iCa^oJG}ox{@2+ at U|46NXb|X|J*CMy at g*H1%%x=1>wWcCt2w at D9mfnNaSEt@sk7
z2d9?TV4#8w@!hZ;i4X;-9lzFtCU#7C<FJ^I{J%TJD&6x<wBtmKs|*{*S$K9-CB at F2
zN3|UI{JK*&UESnn<z>>Z(u~5J>{=Nbfqq1FSf(Q;L878UdSv<4^yK9X at 0ux?B8Y-U
zF3e00DH#puqWj at SP>k2ht~jDb5=cLLAjfrKk#FigMbWtro6dv^wr{?`thpg>YBN6@
zZq`l at oT~(@z@(q^5Vj6U2V5{>b5AP2p)@ZN at tRyOZ5*}p6zLk}@UpP8?@(h<8l!`4
zN2<9kWowQ%*#uZWKzZ!R1HQ<>tW3E-E6xM6IvQmOJ<zW-f7 at A1o_<|n74Ch&<PX?4
zm<eb1=-iG5v4S;zc=MJSuUB#K)*Ytt at tAm9%~PRcJ98)6Tb;iowqB2=C(L^|BlY0S
z{YU9zfA_}rOJ#KhRD`$yJ~D39wF<aXy`UDwyp-+izgxoLa$1P?Holt{sjVLEc4SAU
zmgQa7{^Sgelz$JUiapxnf<7?avERR{bowm5 at q75BAA@BIRx7D#avz8O8(mVKQNu?$
zoW+ZZP$M&Kj80xxtK$Vcb4S*GVSe&q?E}V#Lm`{al5v!btl3vVsyv+9L3}JgXiHr;
zg?MQ>^k9W}Msbx at h!SX#Q>r?Pk8j~io~=u!!N)eFKOcZ<nuY?Sx8|qn2K#kf-3Zgx
zY>@k2`BgDG2}42wYdOoDygCYv4|fME-!#R`bbC8}-QF16pPnj(paool<xkREDPnJD
zq5ui@$K)Us<sJ$n>Eq-L=hi&-)w>0ak3gIwW}1nlw at K^o)+n!LUpewag7qE=$r*?w
z1qotBs&09eU)H^74D at hr`IUi}>Q(Y!?gdMPG4Z1Gmr at C6{JUs3+Dlad<?0ku+}De6
zXM_Gba}@KXzJwQztlsCyE_^=W4n3Uya|}U77w8~f+Jy@>*+b(!MHqv&#$*s{*J!Df
ziLXY`LzgA(R~d-5ZKHIdcBn<>@Xqx_62 at w5UHtuz5Iod!;;;yzWByi&bRqg}W$|k!
zsFn?*i=bsIv~_ZhhQxyo_ZS`atn1$5i`U2yG;t~agi7N*YC{7?kQu5GjH*0`_a~yH
zj=m$?V8yMV at r0cO%SHWaYbj%g$Ofhs895O`fBb`Na&tDQYm}#?@=vU-auUj(uz}5Y
zI2lMS%-bfC{-b<&g}7VA1RZS_A<T&mak(yaT`rkQ;e_);Hh<3aMrVw&y?t`q?lWJj
z(gDbZ?O5-{TZs4Pr}-z9OT(EnGyK^g?niLcp?T2;8`%`k_{4#^wbaVw7Z9XVn_#(`
zsC#|1G4)g at eYkUZ41$hrCwA5PV{Jl9kj?Kd0MBBH{|TvltKDeaZptpid^=G|4$Nw(
z7ltAR#qv+=jV2<SH;rVIW>t>&0v+Fdk{lD<o=4kBJ>E-3ZPx=Ilex#;x*D8JVtlwZ
ztyQzFMz-aBnrwXldKNnra^U~2)PWJF5*yoI+CxxHHSJ45)@Xv*>b8<T<BK?kWb(QQ
zVV6RhTE*x87HanlCGs(kK;zoH3<Jox5E at Da+CBFQ`r8gdA<yPM8X*poH}<v%Tna$!
zR{AkVa2<OJY{8sV^@F-o&-j-6eJmTa$$i^E%1brHQ*fx-He87D=?%4v8QQ#1pI|vv
z89u3OCwuCAnq!&*^!t1pqsg=)WZ1CzrFl`&MN}Kqk{lphHdV<*M!C9Cd=Xd9yrrh&
z7=1?>eGj9k^sU9IGU>xN&GncgG~a1DbYg10&(gcz!%pg&{q$X$GKV5ezcB3Qhj+8S
z&6bHq+EN~nIUu{Gv%rs*KFPDU$}1!JWZE%&PIFWisBHZh=xle7FS3bd>+9o6cwOjl
z1w{K0$s3mu7PH$|U+V4qu at K|3z!Cwi^Mbx%&xWwF#3U?BguLHLaNb0k_EO32E*%OA
z5``e<Z;W>VN(-xGqaSD1Go6juV<^dE=1NYgxrN2Mn at Z5Z_e8LlRgUc@|JVOM`UaJ`
zoi-|wG{ztKaYDV)yo>K9?;)WP(A<<=2aFTxP-`PwdeTO+AVvX=#GyMsxj?L#F@}}6
za9F=up_dAUjC()OfuT?XslO^Cz9V*CgE!%tnwkKOxb2RJ5Y%TYtNc!_RpPN4nFtUT
z56&efSVjqN;IL;RKRUo&3DcilfahOHL>Lq^&sNeA3W>9UX1t4XEReO~#r!@I at 4@y9
zHQT{9Fg6ZIz(^*)@1s``Uu3o0h(u+TB}@H_+=sp<Pzy5;JI42A297@{U7|*@lJN#`
zVh>ba2h&CgoyA{`^^z*c%F62BKMpzmm#O(HZ>9O~PV>HODdYQ+QtI#5XrPRT2Rki$
z&vmYm_Ic_t=>X{Ljzho8KWn27o!$V<!-{dgs;H<xQvk|M3S(?nMuyDfHRun2KYj$t
z^?H_IX*gBcjKv)<b(v*}=ocGRUXP<_TvG9@!|)4-Iybqx08O3qNijk#R%QXdF#a;`
zMDP75#?6uz2O-(~zEUs10U|`z$Qi3uo3r>d(s`lLmuGXB?wpFVGJMfN#hBtvF9I3s
zleHp*bKfy~bxhDgN9)Lo{<g-`(=5<4?c2TQih_U;NSYN>klP86nF*q>&W8`E<6lLw
zKp$E&R>S-6hoR1eq)Pxn<^Lt^bG`maGS>#?ituNh-#9P}%NvFw%H)Ul>f|R!Nid+r
zh%)Y8`;|-*5{Go12h)T+P1px`g!8v^U0SaIHWho4u>XjX;TH#1pVFhe&RiDi><$Jd
z at WHq5HFN;>0rhGdjESvefL)(8#I6!@zcCukbq&TuKK*NSz^;bocKz$AXD}v{^sg}x
zyDEBx at udzF7m*Ow6Zub5*SLG*hr}2+eUv%jL8&eO8gsaNT at MlGE2J6Cyzs$Ckcur1
zZD9cxCwK&ZM`e=sK^kal{N<bc%s;mnROz7=w$k{!_dkV;<_mehB>6-$wR}9*yR6cQ
zQT+lz*veB<jLXmk_+WO_zav)oVBTS`|5I=C^1tOij7?ZyZeuJLqH*_%?7n2&(;63r
zt0s7Op?Flg=?NRm&@*8Xq3?K=MzKn=>aGkE<fvzGcukhEhZZH)`5xTA;bW6;y4*fA
fH33hIpzUei;ql$=`Ca at S0)K|O2%S>+oyY$Vk%bk6
diff --git a/README_FILES/images/home.png b/README_FILES/images/home.png
new file mode 100644
index 0000000000000000000000000000000000000000..cbb711de712dcf06597a3a8a3d95f6fefda1f245
GIT binary patch
literal 1156
zc%17D at N?(olHy`uVBq!ia0vp^%0SG|!3-oLGuzY{7?>FXd_r6WdIS`E6g8)48qP6v
zn&afP)GK*ya`E2cmc1=K$9krmo3ixW(yjNl9=mt!-1BqS-d}tE{`vdw at Bfd2Q7|mP
zr*})-fmwnv$=lt9p at UV{1IU@|>EaktacgbhMZUud966f8a~4$m|G#l@<TjnyOZ%p@
zFh6=~QF^dc=X>FyJ&SiAo$05TFlo-aAio6P7q#s(Rdse}e(pcKHn-#J!fq~2w%=m%
z=T^<))ZV$hk#`oe^QM*Tb8T*ZK3m8p9%@|>b!(dZ+$(>o#JsXA*FO-HQE2Aqd0_Im
zvS$7YQ-eFpD&K6*-u+Ww?=dUa8Tsp3T84Ll&T}qtjVKAuPb(=;EJ|f?&`{R&%uP&B
z^-WCAOwLv?(KFJsP_VSrH?Yt*FjPn`$}BFabjYnNF3C*ROD)z*DJ{s)E742N&z-nS
zaR<;iEwFK!>3NAIr9krxO-+oAjZ)JrO_CB#jf{+pO^lOF42{fE%nXc8(<}|&ynopM
l)WrwV6#%rB!PvCI&{W$%+d!z!?GsRj!PC{xWt~$(6956ca#H{R
diff --git a/README_FILES/images/important.png b/README_FILES/images/important.png
new file mode 100644
index 0000000000000000000000000000000000000000..12c90f607a1b27ddde0a7d922ae255e8c90e883e
GIT binary patch
literal 722
zc$@*!0xkWCP)<h;3K|Lk000e1NJLTq000;O000;W0{{R3*Qw3Q0002wP)t-s5C8xW
z5D*v`7%3?!IXO8(LNQueDPv<8LPA1PQc`PcLSti6b8}K-V`FP;YkPZpjEoqJj5)o%
z2(7Ilt*tqnoLY>Gdz_qOoSb{LwMw<MQoX%erKMx7t!u5Vd*0pv*47x_-XY%JG2Y%e
z=jRCT?*Q-b5dZ%G|NjX8{}AWrDd*=g at 9#1H{~^Z4TGrN5&dzhz)??PzYwzzu=jUVR
z=X->Ngp7=gl$4aErG%}mjHRWNrKOy`y at b8JoTa6ut*xc4t*y1SwY|N)#>U3Z&d%1>
z*52OU=jZ3|@9+2b_y7O at t<uBZ00001VoOIvLub$pVgLXE7fD1xRCwBLk_%J9Fbu~t
zMe#9}p}^G1OWaUY6qJ>TMkk%M`~Kg at u8&dg_P^_0l3yQb639!jLZt^Lx<-O17UeeJ
z-|=!77W(jGx&e#?FOku-g<Jzp-ot34u$w9%;xcgKxjltCsRZ;&?e_q7l@$g*0<e|7
z{|KNq)bD%>KofoU0$~4M+dhLFueEauP`}l7LV=;lsOdn%WHure=x;k`m0(bF&MU#)
z-qv#^n8(MjB|ykioqII#+`g4no-MU=BK|Sahu_3M_-d*=7hq=~t?^}A)G7<qE4_om
zrrI*YUs-nSD*wS0r0A7w6n3nBjAsj4wia8DbCdK1SzE)9VLKTKJ^Ae~_HRnWm=GbP
zW`O<xonvgTQf<sq0000(bVXQnQ*UN;cVTj606-`sDM at W%XJt)cXK7<=AT%yAE;t}L
zIWI6cFEBA6V{&P8ba_B+d2 at 7WV=iTLE at X9ZZ*FuhV{dJdx*@m#001g<MObuGX=iR>
zbairN0An*{V`DL9V>K}|HDobmIW;*pIW=W9HaTQ6F*7kTGI9=S at Bjb+07*qoM6N<$
Ef=i}M4FCWD
diff --git a/README_FILES/images/next.png b/README_FILES/images/next.png
new file mode 100644
index 0000000000000000000000000000000000000000..45835bf89ac0eca3ad69c4d8397cd5edad2f8782
GIT binary patch
literal 1150
zc%17D at N?(olHy`uVBq!ia0vp^%0SG|!3-oLGuzY{7?>FXd_r6qS{MX+1QdG|HK%AA
z&M|bF<K(r}D`;&{^4jF=t=TPmTY8T5OgT4Y>A9tAudUsBZ|kvp$Id-JckTVP=kK4t
z|Nj2}C>RAp85k~3cnHiAj7i?^E({&4vK~NAucwP+NX4za6CZLNR^Vv~eH62Kn_64-
z|L|B5w$*oU%r{bWDv~dEWUiZEps9Sd!R+07pT#eXKJVW2V#-k$hpBnrf;OG$S$b4B
z&|6ho at O!Cu>THROJ9pc8A{}kNzF~e-D9VzSH$h>(q)2#>wCv at X8>8g|SjF#GtzW=y
z^t$olN0Vn;H_lj5xnAe^n*D#S3tQ!=U5z-t=?&0f&Lyr9CBgY=CFO}lsSFMp%9 at _J
ziRr1niRqci*$O6lMtT+smX`Vk7WxK;3du#8#U+&vxs}BwnaO&o#d;~F1^Ia;ddc~@
z6L%@@02-$SHZC(gFR`Q)XkJosl7W#?szqW-s;O~Ol1ZASNm`OYnz=!Wd9p#GMRIBV
noNS;jK9H^eptTIfrWJ;!+6LMNLUnGRfHDl8u6{1-oD!M<e<f(U
diff --git a/README_FILES/images/note.png b/README_FILES/images/note.png
new file mode 100644
index 0000000000000000000000000000000000000000..d0c3c645ab9af6318035b026dd86944b9ddc9114
GIT binary patch
literal 490
zc%17D at N?(olHy`uVBq!ia0vp^5+KY0Bp8m$B&h%?rX+877Y2q^y~;)m42&$EE{-7;
zw^Ao5PC9JB<666~wbh%Oo7tJ0_rGWfi({a(qjR96>_P4H$tJl=Pxd^1p6`8M?=jE(
z!hchf6<8Mi`NHe6`iia69>!lxe4C2-S>+Wvj;l63k!MNSb9|1+f-i>irDi|a^uLl%
z^#-f2#pQ2lC%m~9m9JWJZL3|T8FRX>A78<>cU{Z6XGFG336=OLGkw25|29|RG_k%U
z>j|M}Ih2i#-w+9_<r3R)WpUAchOn-RHL<s=Lf(b2eK(Z&HDT8se{PS{FHG*n=5x<`
zd$sD!x#j1d%}jr;ba|5Bzo#BK+dePv4*lLObfom}ocrsiUF3UWxzDDpWTH<0^^(i?
z<2yBtJoul=&zt%^h38kc6EJj~OI#yLg7ec#$`gxH85}f}H9d0^(^GvD(=(H^6- at Mu
z^ehxCE%gm7^bHIZl8Z8nODY|5D~n4qll4-I^- at X;^7BgclJj#X?o!+VG)@a_TxNP+
zVo52`JmaJka})Dq(^L~v3j=cl1G6LpOM{dY<5crRBeP@^6Oqf4e*tyzfpi4`tz|GW
asW39pHqbUGism!~MWUyxpUXO at geCyC{J9YT
diff --git a/README_FILES/images/prev.png b/README_FILES/images/prev.png
new file mode 100644
index 0000000000000000000000000000000000000000..cf24654f8a9d6826bf5ee3f6b640d0b34f44d2ed
GIT binary patch
literal 1132
zc%17D at N?(olHy`uVBq!ia0vp^%0SG|!3-oLGuzY{7?>FXd_r6qS{MX+1QdG|HK%AA
z&M|bF<K(r}D`;&{_SWp;y~QniTY8T5OgT4Y>A9tAudO|H at 7THL=dQiK_Wb?x_ut?D
z9|faeh=V<Ej;sP^3C1LEcNc~ZR#^`qr`prSF{I+w)`<uC4jJ$`?`AnA6{c$R?|;5(
z>Z1ckwsBibkEq%tviw8L)9pMqrPD57nf+kKM(veu=M#_EDSL|q&ljJq*c-Sp at yIG(
zZVk(g;bk=^BiD9EUJ~xv&68sL=vnl#m#dp5{haQ1T>9^ubQ$5rDtCDM at BIJT?QF}q
z^^0m35BJV)pfj9HTq8<?^V3So6N^$A95j?QJ#!P&Q+*TDGn2CwO!SQOEEFs)^$jfa
z4Ga~Mi!zH#Djjkwi%T+-^-_!VQc4T*^GfuR^K&QeQrrPFP77>YW_n&?Nh#1ggER~C
z#Kbg%Bm<LFBa@^=Lo?GfqePRWv_!L{<U~{R><crd0(J3$bOivdWiU3aFf`RR&^8dN
SbNd98VeoYIb6Mw<&;$VU^kS_5
diff --git a/README_FILES/images/tip.png b/README_FILES/images/tip.png
new file mode 100644
index 0000000000000000000000000000000000000000..5c4aab3bb3543191c360387c4af9a3cbaa051345
GIT binary patch
literal 449
zc%17D at N?(olHy`uVBq!ia0vp^l0YoM0VEi-?r};1DW)WEcNYeRRlUkaK;CUn7srr_
zTcwkB3m#VBagbEhuIO1a$!w|j^feEHv@=+|woj32oF*6|l<3XH$!T}Lao)e6jJhoQ
z_vr?OZVprC|7iZ8xaicUR0b&zR-JdP408=+?zC0Rn8IN&QLTWhMIcU0X8P7uUN4WB
zR_?a_;&p>ll{>(OzfJ4N{OX={Q&zNgO}j31DC;|ya^r9Fjd348&p+!^cT+^sws+=&
zCu*T*mRNK3PE;$BNx$Fa_9(Y=&DoXLMFRd#U31O)<`W^F&-o=xbIqK-c?mr!bmn>!
zky6TW;ML~4nXw$T{yra=OteL6G!=B;0=?;6;u=vBoS#-wo>-L1;Gm(b>6x3Dp6Z*J
zo|&AjV4`QFXQ5zesc&GRZ(yj9T$EW{Qt6OeSzMBtte0A>mr`1gpI4%noS!>!m*Ngk
zP=bxiOwUU!DFvFBnrfM1VQy}0X_lO7WM-b2nq**TZkb|inr2~`WR#dJWc*7VsEZGz
hD*$LMgOO>4k%_i}w!yxSoa;ex<LT<>vd$@?2>`ZxpwIvS
diff --git a/README_FILES/images/toc-blank.png b/README_FILES/images/toc-blank.png
new file mode 100644
index 0000000000000000000000000000000000000000..6ffad17a0c7a78deaae58716e8071cc40cb0b8e0
GIT binary patch
literal 318
zc%17D at N?(olHy`uVBq!ia0vp^{6Ngf!VDzk7iOmbDT4r?5LY1G0LBeqssYS8aNs~g
zL&JZdz<-ASVCp}Z1qA-WYMwv=#w2fd7lsa2Sq~sb&eO#)q~ccckN^As+Y8hr{NQI)
zU=tG*$WU0oc7}mr$_1xO8FCvR0987dxJHx&=ckpFCl;kLIA|zqdgdmkr}`$QXC`MW
znCKblStwXq>Kj<-8yG4i7iAWgR668V7MElu>!lX!rIZ%r=auLs=jTq`rMLrVoEF%)
z%=Em(l2V{~Y38ZM78WUqNhv0&X$A(C=H`YbNv38d#%U=j#>r_(No{j3fdT}iD*$LM
bgQ01KrM7{#fqB^nkZ}y2u6{1-oD!M<OHX0g
diff --git a/README_FILES/images/toc-minus.png b/README_FILES/images/toc-minus.png
new file mode 100644
index 0000000000000000000000000000000000000000..abbb020c8e2d6705ebc2f0fc17deed30f2977a46
GIT binary patch
literal 259
zc%17D at N?(olHy`uVBq!ia0vp^{6Ngf0VEhsJkjh1QcOwS?k)@rt9q4<fIK-*7srr_
zTge%XjsFu4Ff9Eqd_J9lQ9+)^^Ll at lJwvT-!Mdo=yIg at Pol9IJN`mv#O3D+9QW+dH
zlr=qb6Vp?D6Vo%3vlUGAjPxuNEG_j7Ec6Ww6_SfGi%TjUax05VGL!XEi}g}U3-a?y
z^pf*)C+<?*0W?kvY+Pn~USdfp(7e>-G!sMP)HD-wQzH`-1CumMgJctv6pLi at 6hos#
pqtv?{|7L-F1=1A&w3flpw8B!`K-<8)>;uR+22WQ%mvv4FO#uFaO+^3z
diff --git a/README_FILES/images/toc-plus.png b/README_FILES/images/toc-plus.png
new file mode 100644
index 0000000000000000000000000000000000000000..941312ce0dab168e0efcc5b572e387259880e541
GIT binary patch
literal 264
zc%17D at N?(olHy`uVBq!ia0vp^{6Ngf0VEhsJkjh1QcOwS?k)@rt9q4<fIMYS7srr_
zTge%XjsFu4Ff9GA{8{qDCrJl}6YS1PR~FCg&$4F-^2*tiGSkWgsNA{4HKHUqKdq!Z
zu_%?nK|@*7GdD3k)i*IcGdWwqM9)aiLc!8f- at roOz)&H%D6_bv(jm99xFj=KFSS at N
zrL-VFuS737KX>9T#T`K7w7|w?rspM=lmg95OfodLFfd9rOi4*hH8wIdOfpPPHA_l1
vPBO4aOiebg{<Ta4<Ta440HC!DhNcyk+6LMN=4Bs1#xZ!h`njxgN at xNAvM^9T
diff --git a/README_FILES/images/up.png b/README_FILES/images/up.png
new file mode 100644
index 0000000000000000000000000000000000000000..07634de26b325b09b6686543e3743ec58426e64b
GIT binary patch
literal 1111
zc%17D at N?(olHy`uVBq!ia0vp^%0SG|!3-oLGuzY{7?>FXd_r6qS{MX+1QdG|HK%AA
z&M|bF<CMHMIeTk%%iflrV?9&OP1$;H>#=*s&OJYO?ftdq at 1MW_{{H_c7zM)xY}!4u
z4wxkvlf2zs7&=&GJ%F4vPZ!6Kid$1Bc=H`l;9$*t&CcrhuKwn=xt`if@(;G1bchSE
zf2gkP5o`T$gU!z%*@<VPCp-&Cefh{>j*(i44TIM5^J}`zgLC_3mRylj^UB=K=gvB7
zAJ6I2_ifMD%d~kX_8V5;d%~8!MPZrPu??Uc>0II(Q4*Y=R#Ki=l*-_sp{(hdo0y*J
zo0y)NoULG at XQXGLU}>puV4-hdsE}NgSzJ=-kXu<?l9{ZRTCA5+T9BVtqL-YXJ8_rd
z4xn*bVB<2=^Abx+f#w;Rn;RKh7#mqwq*^8?ry3e2ni?9V8YNmJrlc90CmCe(YyoOw
j-~;Ij09wmnY+7Mxs%@ZcAXMk}2_)s|>gTe~DWM4f3>H<j
diff --git a/README_FILES/images/warning.png b/README_FILES/images/warning.png
new file mode 100644
index 0000000000000000000000000000000000000000..1c33db8f34a8b42b373179b46a2d8d8a10e061a9
GIT binary patch
literal 1241
zc%1E2YfF;>6n<h^OL8m;8sVlhgw|kgYs(F#dD(E&v}N9=Q0=CTvCXzu(uQATg at Q=t
zBC2IczKS2#2jPXX%#4^KwZsn#%C;bKn$;4Fffww}-_S=N&V|F_c at 75-r(0iHnwEMr
z6+oI!tFEFm=btkfbo?;-ZWdrbf2LZ)1lT)0J|F-jKrv7T(2NGa4tM}B-~)O9KW$!b
zpsFg+&=Bx?Nq|%;$+9xi=Og`ok|cvbP$CKX{S-h61XNEC6$nryNm0}UFcAz+P*eyA
zvDqOGClm^WfiRmL=5WFSK?I1<F%pRa(P%Ud#QFSqJe~j&OlBgHNCL at ZavRv*#V-C6
zTJpcrSB*WU@=`UrQn*^WSkkGBs at py~Zgsdi#q3bRU$L<GW_g$)XC1ko^Wt`nBQ3V=
z3I4dI at 4vA9s6aIkpXbe;>%LdshqalyX<zMF9ZRbkTAb}#EBuyt?3}4J+?;$i&VTql
zb7*tRXz1lFxx=dciaRBlY+=}ty)S2^qv$~kQ7|<f`eohwr*F!dI!L_wa+&wru#xfe
z;6>{FV97!ocioo0!HsK{6RYXtEXe?4+H*KxKl1iv#<0R}YYe6kEW>7fhj9H<&Bn^-
z;7Uw8SKwUcj%M at xtCn7;`^Me-r^m<|_3N#2XUyK1KmS}MPI2-l1B@&5&LxDVo;dGn
zBCZ<URumpTCeS&I&DIKIv(0SJl8K}uMV3;TD^}!+C0S;-%|o;oI@&#i%`CEdM3&YT
zmy-~gU5>tS{!?1+#EzV;*=ZzNX+5({YRa=nE%H+`qtPT+C=^PGGEbQ&KG`Ienq=~`
lhhqISi}jDCr^iAnYm>@^VxidGc!2&%fKF4Xeo}m?^DkXAMacjF
diff --git a/README_FILES/screen.css b/README_FILES/screen.css
new file mode 100644
--- /dev/null
+++ b/README_FILES/screen.css
@@ -0,0 +1,295 @@
+/*
+$Id: screen.css 11 2005-12-27 17:00:04Z patrick $
+*/
+
+body {
+ font-family: Verdana,Helvetica,sans-serif;
+ font-size: 100.01%;
+ color: #000000;
+ background-color: #FFFFFF;
+ margin:0;
+ padding:0.5em;
+}
+
+a:link {
+ color: blue;
+ text-decoration: underline;
+}
+
+a:active {
+ color: red;
+ text-decoration: underline;
+}
+
+a:visited {
+ color: darkblue;
+ text-decoration: underline;
+}
+
+a:hover {
+ color: red;
+ text-decoration: underline;
+}
+
+div.author,
+div.chapter {
+ font-size:0.8em;
+ }
+
+table {
+ font-size:1em;
+ }
+
+.title {
+ font-size:1em;
+ }
+
+h2, h3, h4, h5 {
+ margin:2em 0em 0em 0em;
+ }
+
+pre {
+ font-size:1.1em;
+ }
+
+.programlisting {
+ font-family: "Courier New", monotype;
+ padding:0.5em;
+ border-left:1px dashed #000000;
+ background-color:#EBEBEB;
+ }
+
+.screen {
+ font-family: "Courier New", monotype;
+ padding:0.5em;
+ background-color:#F0F8FF;
+ }
+
+.important {
+ -moz-border-radius:15;
+ border:2px solid #FFFF00;
+ padding:0.5em;
+ }
+
+.note,
+.tip {
+ -moz-border-radius:15;
+ border:2px solid #949494;
+ padding:0.5em;
+ }
+
+code {
+ font-size:1.2em;
+ }
+
+em.parameter {
+ font-family: "Courier New", monotype;
+ font-style:normal;
+ }
+
+ol li p {
+ margin:0;
+ }
+
+ul li p {
+ margin:0;
+ }
+
+p {
+ line-height:1.3em;
+ }
+
+
+dd p {
+ margin:0em 0em 0.5em 0em;
+ padding:0em 0em 0em 0em;
+ }
+
+dt {
+ margin:1em 0em 0em 0em;
+ padding:0em 0em 0em 0em;
+ }
+
+
+div.toc a {
+text-decoration:none;
+}
+
+div.toc {
+ margin:2em 0em 2em 0em;
+ }
+
+div.toc dt {
+ margin:0.2em 0em 0em 0em;
+ padding:0em 0em 0em 0em;
+ }
+
+div.calloutlist a {
+ color:#000000;
+ text-decoration:none;
+ }
+
+div.revhistory table,
+div.revhistory table td,
+div.revhistory table th {
+ border:none;
+ }
+
+
+/*
+h1 {
+ font-size: 0.7em;
+ margin: 0em 0em 0em 0em;
+}
+
+h2 {
+ font-size: 0.7em;
+ margin: 0.5em 0em 0em 0em;
+}
+
+h3 {
+ font-size: 0.7em;
+ margin: 0.5em 0em 0em 0em;
+}
+
+h4 {
+ font-size: 0.7em;
+ margin: 0em 0em 0em 0em;
+}
+
+h5 {
+ font-size: 0.7em;
+ margin: 0em 0em 0em 0em;
+}
+
+p {
+ font-size: 0.7em;
+ margin: 0em 0em 0.5em 0em;
+}
+
+ol {
+ font-size: 0.7em;
+ }
+
+tt {
+ font-family: monotype;
+ font-weight: bold;
+ font-size:100%;
+}
+
+em {
+ font-size:120%;
+}
+
+
+code {
+
+}
+
+p code {
+ font-size:0.7em;
+}
+
+pre {
+ font-size:0.7em;
+}
+
+hr {
+ display:none;
+}
+
+.authorgroup {
+ margin:1em 0em;
+}
+
+.revhistory table {
+ border: none;
+ padding: 0px 0px 0px 0px;
+ margin: 1em 0em 1em 0em;
+}
+
+.revhistory th {
+ border: none;
+ padding: 0px 0px 0px 0px;
+}
+
+.revhistory td {
+ border: none;
+ padding: 0px 0px 0px 0px;
+}
+
+.note {
+ border: 1px solid #CCCC99;
+ background-color: #F5F5E7;
+ padding: 0.5em 0.5em 0.5em 0.5em;
+ margin: 1em 0em 1em 0em;
+}
+
+.note table {
+ font-size:0.7em;
+}
+
+.caution {
+ border: 1px solid #F6EA00;
+ background-color: #FFFFC9;
+ padding: 0.5em 0.5em 0.5em 0.5em;
+ margin: 1em 0em 1em 0em;
+}
+
+.tipp {
+ background-color: #F5F5DC;
+}
+
+.important {
+ font-family: "Courier New";
+ border: 1px solid #F95E00;
+ background-color: #F9CDB3;
+ padding: 0.5em 0.5em 0.5em 0.5em;
+ margin: 2em 0em 2em 0em;
+}
+
+.screen {
+ font-size:0.9em;
+ font-family: monospace;
+ border: 1px solid #999999;
+ background-color: #EBEBEB;
+ padding: 0.5em 0.5em 0.5em 0.5em;
+ margin: 1.5em 0em;
+}
+
+.programlisting {
+ font-size:1em;
+ font-family: monospace;
+ border: 1px solid #B2DBFF;
+ background-color: #F0F8FF;
+ padding: 0.5em 0.5em 0.5em 0.5em;
+ margin: 1.5em 0em;
+}
+
+.important .programlisting {
+ font-family: "Courier New";
+ font-size: 0.7em;
+ border: 1px solid #B2DBFF;
+ background-color: #F0F8FF;
+ padding: 0.5em 0.5em 0.5em 0.5em;
+ margin: 2em 0em 2em 0em;
+}
+
+.important .screen {
+ font-family: "Courier New";
+ font-size: 0.7em;
+ border: 1px solid #999999;
+ background-color: #EBEBEB;
+ padding: 0.5em 0.5em 0.5em 0.5em;
+ margin: 2em 0em 2em 0em;
+}
+
+.informalexample {
+ font-family: "Courier New";
+ font-size: 0.7em;
+ border: 1px solid #B2DBFF;
+ background-color: #F0F8FF;
+ padding: 0.5em 0.5em 0.5em 0.5em;
+ margin: 2em 0em 2em 0em;
+}
+*/
diff --git a/RELEASE_NOTES b/RELEASE_NOTES
--- a/RELEASE_NOTES
+++ b/RELEASE_NOTES
@@ -1,3 +1,1653 @@
+---------------------------------------------------------------------------
+ June 27, 2007
+amavisd-new-2.5.2 release notes
+
+
+BUG FIXES
+
+- in a milter setup log_id was left undefined, which resulted in log lines
+ without id, and a SQL constraint violation "Column 'am_id' cannot be null"
+ when logging to SQL was enabled. The bug was introduced in 2.5.1;
+ problem reported by Martin Svensson;
+
+- suppress a quarantining attempt if the message also needs to be archived
+ to the same location (same sql key or same local filename);
+ reported by Wazir Shpoon;
+
+- adjust $socketname in amavisd-release to match its default counterpart
+ in amavisd (i.e. /var/amavis/amavisd.sock); reported by Stanley Appel;
+
+
+NEW FEATURES
+
+- add snmp-like counters for PenPalsSavedFromKill, PenPalsSavedFromTag3
+ and PenPalsSavedFromTag2, which correspond to the number of messages
+ where spam level would have exceeded a corresponding level had there
+ not been for (negative) score points contributed by pen pals lookups.
+ Note that for any message only one of the three counters could
+ increment, the one corresponding to the highest level crossed.
+ To find more information about rescued mail messages, search the log
+ for a string 'PenPalsSavedFrom' (available at log level 2 or higher).
+ Practical value: mail saved by pen pals from being blocked usually
+ indicate false positives by SpamAssassin; examining rules which
+ contributed most to the score may indicate these rules need adjustment;
+
+- when preparing a SQL SELECT clause in lookup_sql, provide an additional
+ placeholder %a in a clause template, which is much like the existing %k,
+ but evaluates to an exact mail address (i.e. the same as the first entry
+ in the %k list), which makes it suitable for SQL pattern matching;
+ suggested by Daniel Duerr;
+
+- macro supplementary_info can supply information on two additional
+ SpamAssassin tags: AUTOLEARNSCORE and LANGUAGES if corresponding plugins
+ are enabled in SpamAssassin; see README.customize for the complete list;
+
+- provide two new subroutines available for calling from config files:
+ include_config_files() and include_optional_config_files(), each take
+ a list of filenames as arguments, and reads & evaluates them just like
+ normal configuration files specified on a command line (option -c
+ or a default amavisd.conf). This provides a simplified and uniform
+ mechanism for 'including' additional configuration files, which formerly
+ could be invoked through a perl do() function. The only difference
+ between include_config_files and include_optional_config_files is that
+ the former aborts if some specified file does not exist, while the later
+ silently ignores specified but missing files. Both/each subroutine
+ may be called multiple times, recursion is allowed (but some sanity
+ limit to recursion is provided); based on a suggestion by Gary V.
+
+ Example line in amavisd.conf:
+ include_config_files('/etc/amavisd-custom.conf');
+
+
+OTHER
+
+- provide a workaround for a crashing altermime by removing its leftover
+ temporary file which would otherwise cause a temporary failure:
+ TempDir::check: Unexpected file
+ problem reported by Dennis A. Kanevsky;
+
+- add a mapping to 'doc' for a result 'Microsoft Installer' from a file(1)
+ utility; it seems like versions 4.20 and 4.21 of file(1) (possibly earlier
+ versions too) misclassify all MS Word, Excel, and PowerPoint documents
+ as 'Microsoft Installer'; problem investigated and a workaround
+ suggested by Noel Jones, Mike Cappella and Michael Scheidell;
+
+- add a mapping to 'asc' for a result 'COM executable for DOS' from a file(1)
+ utility; it seems like later versions of file(1) can misclassify a text
+ in a GB2312 character set as a COM file; reported by Daniel J McDonald;
+
+- updated AV entry for ESET NOD32 Linux Mail Server again - command line
+ interface (nod32cli): added a status 3 (e.g. corrupted archive) back to
+ the list of clean statuses; the 3 was removed in 2.5.1 as the entry
+ was substituted with the one from a NOD32 documentation; reported by
+ Tamás Gregorics;
+
+- updated AV entry for 'F-Secure Antivirus for Linux servers' to cope
+ with version 5.5 and later; a new entry provided by Peter Bieringer;
+
+- when a command line option -g requests changing of group ID, do so
+ by calling POSIX::setgid, after also attempting to assign to perl
+ variables $( and $), which may not work correctly on systems where
+ group ID can be negative (like group 'nobody' being -2 on Mac OS X);
+ follows a SpamAssassin problem report 3994, investigated
+ by Sidney Markowitz;
+
+- when an AUTH command parameter (RFC 2554) is supplied on a MAIL FROM
+ SMTP command but AUTH support has not been previously offered
+ (like when authentication is disabled by an empty @auth_mech_avail),
+ no longer treat the situation as a fatal error:
+ 503 5.7.4 Error: authentication disabled
+ but mercifully ignore the parameter and just log an informational
+ message. This is a deviation from RFC 2554, but makes it friendlier
+ for those insisting on running amavisd as a Postfix pre-queue smtp
+ proxy; suggested by Alexander 'Leo' Bergolth;
+
+- adjust the list of pre-loaded perl modules required by SpamAssassin;
+
+- internal: pass a mail message to SpamAssassin as a GLOB instead of an
+ array reference, saving one in-memory copy of a message during a SA call;
+
+- internal: make it slightly easier to switch message digest from MD5 to
+ a Digest::SHA family by turning a hard-wired key length into a parameter
+ (admittedly it is still ugly, requiring a change in three places for
+ switching); also pave a transition from Digest::SHA1 to Digest::SHA;
+
+- documentation: updated files README.postfix and README.postfix.html
+ now include a section 'Advanced Postfix and amavisd-new configuration'
+ explaining a multiple cleanup service architecture; thanks to
+ Patrick Ben Koetter; retired file: README.postfix.old
+
+- documentation: updated README.sql-pg to include a faster alternative to
+ purging a SQL logging database: the alternative 'DELETE FROM maddr' on
+ PostgreSQL runs faster by a factor of 1.5 to 2 from the one previously
+ suggested;
+
+- suggestion: when using SpamAssassin plugin Rule2XSBody
+ (available in more recent versions of SA), adding an entry like:
+ Mail::SpamAssassin::CompiledRegexps::body_0
+ to the @additional_perl_modules list allows preloading of compiled rules.
+
+ Adding the following two lines to amavisd.conf adds the directory name
+ containing modules with compiled rules to Perl modules search path and
+ allows Perl to find the listed module(s):
+ my($sa_instdir) = '/var/db/spamassassin/compiled/3.002001';
+ unshift(@INC, $sa_instdir, $sa_instdir.'/auto');
+
+
+---------------------------------------------------------------------------
+ May 31, 2007
+amavisd-new-2.5.1 release notes
+
+
+COMPATIBILITY WITH 2.5.0
+
+- setting $bypass_decode_parts to true now also disables MIME decoding
+ (see below);
+
+
+SECURITY
+
+- provides checking the number of archive members against $MAXFILES quota
+ even when just listing an archive directory, providing some additional
+ protection (besides a time limit) against runaway dearchivers
+ (such as a recent Zoo archiver DoS);
+
+- please use the most recent versions of file(1) utility (currently 4.21)
+ and recent versions of external dearchivers/decoders to avoid known
+ security vulnerabilities in them;
+
+
+NEW FEATURES
+
+- introduced a variation of a message release from a quarantine, allowing
+ a releaser to choose between forwarding a message to the back-end MTA
+ port as usual (avoiding re-checking of a message), or to send it to MTA
+ on its incoming port (normally 25) and let the message be rescanned,
+ which might be useful after adjusting spam rules or antivirus database.
+
+ It is implemented by:
+
+ * adding a configuration variable $requeue_method (also a member
+ of policy banks), with a default value: 'smtp:[127.0.0.1]:25'
+
+ * extending the AM.PDP protocol with a 'request=requeue' attribute
+ which can be used in place of a 'request=release',
+
+ * enhancing the 'amavisd-release' utility program to choose between
+ sending 'request=release' and 'request=requeue' based on its
+ program name, i.e. by making a soft or hard link to amavisd-release
+ (or its copy) named 'amavisd-requeue', the utility will send
+ a 'request=requeue' in place of the usual 'request=release', e.g.:
+ # ln -s amavisd-release amavisd-requeue
+ $ amavisd-requeue spam/k/kg2P0rP9Lpu3.gz
+
+ * enhancing amavisd daemon to choose between forwarding a released
+ message either to $release_method or to $requeue_method destination
+ based on a 'request' attribute value in an AM.PDP request;
+
+- new AV entry: ArcaVir for Linux and Unix, see below for links;
+
+- a new macro 'supplementary_info' gives access to some additional information
+ provided by content scanners, such as a provided by SpamAssassin API
+ routine get_tag. The macro takes two arguments, the first is a tag name
+ (a name of some attribute which is expected to provide an associated
+ value), the second argument is a sprintf format string and is optional,
+ if missing a %s is assumed. Currently the only available attributes are
+ AUTOLEARN, SC, SCRULE, SCTYPE, and RELAYCOUNTRY. These are nonempty only
+ when an associated SpamAssassin plugin or function is enabled.
+
+
+BUG FIXES
+
+- fixed quarantining to a SQL database of messages with a null envelope
+ sender address (broken in 2.5.0, causing such messages to tempfail);
+ reported by Markus Edholm, Vahur Jõesalu and Michael Scheidell;
+
+- fixed parsing of certain broken 'From' header fields, which would
+ result in a temporary failure and the following logged error:
+ check_init2 FAILED: parse_address_list PANIC1 53
+ at /usr/local/sbin/amavisd line 3292
+ reported by Michael Scheidell;
+
+- avoid encoding nonprintable characters in X-Envelope-From and X-Envelope-To
+ header fields in a quarantined message even if envelope mail addresses
+ contain such invalid characters, so that a quarantine release is possible;
+ (RFC 2047 allows encoding of a 'phrase' in From, To, and similar headers,
+ as well as in comments, but not in the address specification);
+
+- avoid unnecessarily RFC 2047 -encoding of 8-bit characters in those
+ lines of inserted X-Spam-Report (and similar) multiline header fields
+ which only contain ASCII characters; also avoid encoding of newlines;
+ reported by Anant Nitya;
+
+- properly recognize PostgreSQL error code 'S8006' and reconnect to
+ a disconnected server right away; thanks to Brian Wong;
+
+- call $mail_obj->finish after a SA call to allow for garbage collection
+ and removal of SA temporary files; see:
+ http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5444
+
+- avoid nonstandard SMTP status code 254 on discarded malware;
+ on discarding turn status 554 into a 250 instead; violation
+ of a SHOULD in RFC 2822 pointed out by Alexander Bergolth;
+
+- an informational log message was reported inappropriately:
+ INFO: truncated ... header line(s) longer than 998 characters
+ it didn't reflect reality, it was always reported together with the:
+ INFO: unfolded 1 illegal all-whitespace continuation lines
+
+- when a SMTP option BODY=8BITMIME (RFC 1652) is not given on mail
+ reception, avoid turning it on while forwarding even if mail body
+ contains 8-bit characters; following a garbage-in-garbage-out
+ principle, this doesn't break anything that isn't already broken,
+ but might prevent later conversion to 7-bit quoted-printable MIME
+ by some downstream MTA, possibly preventing signature invalidations
+ (DKIM, S/MIME, PGP, ...) - at a risk that some overzealous firewall
+ might block a mail transfer;
+
+- fixed a couple of documentation typos/bugs in README.customize,
+ thanks to Mike Cappella;
+
+
+OTHER
+
+- modified code for checking each eval {} status: it turns out that eval
+ is able to capture certain error conditions (e.g. certain I/O errors)
+ but without setting the $@ variable, leaving it empty; use new idiom
+ throughout for proper error handling and more informative reporting,
+ showing errno in such cases;
+
+- setting $bypass_decode_parts to true now also disables MIME decoding,
+ not just decoders/dearchivers listed in a @decoders list, and also
+ implicitly retains full original message for virus checking, equivalent
+ to having a regular expression /^MAIL$/ in a @keep_decoded_original_maps
+ list; prompted by Bill Landry;
+
+- new AV entry: ArcaVir for Linux and Unix, see:
+ http://www.arcabit.pl/
+ http://www.arcabit.com/download_product.html?product=ArcaVirLinux2007
+ http://www.arcabit.com/products_arcavir_for_unix_2006.html
+ the entry was kindly provided by Michal Seremak;
+
+- updated AV entry for ESET NOD32 Linux Mail Server - command line
+ interface (nod32cli), version 2.7, thanks to Simon;
+
+- updated AV entry for Sophos sweep, adding options -mime and -oe ;
+
+- avoid repeatedly reporting the same set of modules by a log entry
+ 'extra modules loaded:', only report it on changes to the list;
+ repeated reports could be misinterpreted that modules were loaded
+ with each mail task, where actually missing modules were only loaded
+ once within each child process;
+
+- avoid reporting 'BOUNCE' in a SMTP response text when a bounce (i.e.
+ a nondelivery status notification) was actually suppressed, such as
+ is usually the case with infected mail or when spam score exceeds
+ spam_dsn_cutoff_level. Previously the SMTP response text only reflected
+ the setting of a final_*_destiny, which could mislead mail administrators
+ into believing that excessive unconditional backscatter was being
+ generated. The new text looks like:
+ 250 2.5.0 Ok, id=67685-15, DISCARD(bounce.suppressed)
+ instead of previous:
+ 250 2.5.0 Ok, id=67685-15, BOUNCE
+
+ A general note worth reiterating: to reduce backscatter pollution
+ (sending of bounces to innocent sender addresses), please either:
+
+ * set $final_virus_destiny and $final_spam_destiny to D_DISCARD
+ or to D_PASS (_not_ to D_REJECT or D_BOUNCE),
+
+ or:
+ * carefully configure virus and spam bounce suppression by:
+ . configuring @viruses_that_fake_sender_maps correctly (the default
+ is fine, it suppresses all bounces to infected mail), then one may
+ safely set $final_virus_destiny to D_BOUNCE, it is equivalent to
+ D_DISCARD for all infected mail containing malware matching the
+ @viruses_that_fake_sender_maps;
+
+ . and: configuring @spam_dsn_cutoff_level_maps and
+ @spam_dsn_cutoff_level_bysender_maps, keeping levels just slightly
+ over a kill level, have a well maintained SpamAssassin with
+ network tests enabled and updated rules - then one may set
+ $final_spam_destiny to D_BOUNCE, which will produce bounces for
+ mail with spam score between kill level and cutoff level, and
+ suppress bounces above suppress level; some domains may still
+ consider such practice abusive, so consider decisions twice;
+
+ . to monitor bounces generated by amavisd, one may assign some
+ dedicated monitoring e-mail address to $dsn_bcc, which will then
+ receive a copy of all delivery status notifications sent out
+ by amavisd;
+
+- dspam options changed with version 3.8.0, replacing option --feature
+ with --tokenizer; reported by Jim Knuth;
+
+- modified syslog writing to check errno after calling Unix::Syslog::syslog,
+ and to informatively attempt to log status when unsuccessful;
+ an unsuccessful status is just informational, as syslog(3) routine does
+ its own retries and leaves an unsuccessful status of a previous attempt
+ in errno even if a subsequent logging attempt did succeed; unfortunately
+ the system routine syslog(3) returns no value according to documentation
+ (and according to its source code), so its completion status can not be
+ tested; problem of logging loss on syslogd restart on OS X reported by
+ Paul Walker, but unfortunately can not be solved on the application side;
+
+- uncomment some debugging printouts in p0f-analyzer.pl and land them
+ under control of a $debug variable;
+
+
+---------------------------------------------------------------------------
+ April 23, 2007
+amavisd-new-2.5.0 release notes
+
+
+COMPATIBILITY WITH 2.4.5
+
+The 2.5.0 is upwards compatible with 2.4.* versions, except
+for the following:
+
+Default notification and logging templates are enhanced to take advantage
+of new macros and new concepts, so it is prudent to update templates
+if defaults are overridden, e.g. $log_templ, $notify_*_admin_templ, ...
+
+A client-side AUTH (rfc2554: SMTP Service Extension for Authentication)
+is currently not available. The last version with this feature working
+is amavisd-new-2.5.0-pre4. The feature may be restored in a future version
+if interest is demonstrated.
+
+A workaround for a qmail bug (which complains when CR and LF are split
+across a TCP segment boundary) is no longer available, as the program
+has no control over IP packet splitting done by the TCP/IP stack.
+
+
+NEW FEATURES AT A GLANCE
+
+- new concept: blocking contents category;
+
+- true per-recipient defanging/sanitation of a mail body (previously
+ a true per-recipient handling was available for mail header edits,
+ but not for mail body modifications);
+
+- added interface code to invoke Anomy Sanitizer or the 'altermime' program
+ allows defanging or adding disclaimers by external utilities on a
+ per-recipient basis;
+
+- rewritten SMTP client code: get rid of the troublesome module Net::SMTP;
+ new code now supports pipelining, client-side LMTP, IPv6, Unix sockets,
+ more reliable error detection and handling, passes on ENVID parameter
+ unchanged, is bare-CR-clean, tidier code (no workarounds for rough corners
+ in Net::SMTP), fewer context switches (handshake handovers) due to
+ pipelining if pipelining is offered by MTA (which usually is);
+
+- makes available pedantically parsed addresses from a mail header:
+ From, Sender, To, Cc. Addresses from mail header may be needed for
+ deciding on inserting disclaimers, signing mail (DKIM), custom hooks
+ (like 'vacation'-type applications), and other future applications.
+ Get rid of inexact parsing by module Mail::Address, provide own parser;
+
+- phishing fraud as returned by ClamAV is now treated as spam, no longer
+ as a virus;
+
+- compatible with SpamAssassin 3.2.0;
+
+- enhancements to amavisd-nanny: shows more detailed states of processes;
+
+- enhancements to amavisd-agent: shows average processing times per message;
+
+- extended AM.PDP protocol with an attribute 'policy_bank' which may be used
+ in a client's request to require loading additional policy banks;
+
+- add support for 7-Zip archives if external utility 7z is available;
+
+- custom hooks allow custom code to be called at few strategic places;
+
+- penpals can now also match replies which reference previous outgoing mail
+ by its MessageID (taking into account References or In-Reply-To header
+ field);
+
+- new key 'originating' in policy banks generalizes a MYNETS policy bank;
+
+- a documentation rewrite for setting up amavisd-new with Postfix
+ by Patrick Ben Koetter (one of the two authors of The Book of Postfix).
+ Previous documentation has been renamed to README.postfix.old and will be
+ removed in the next version; the new documentation is README.postfix.html,
+ and its automatically converted plain text version is README.postfix.
+ A big thanks to Patrick for his efforts!
+
+
+BUG FIXES
+
+- if a sender is both white- and black-listed at the same time, then
+ inserted X-Spam-* header fields were inconsistent, e.g. X-Spam-Level,
+ X-Spam-Flag and X-Spam-Status reflected a whitelisted status (no asterisks,
+ not a spam), while X-Spam-Score showed 64 points; now whitelisting prevails
+ in all X-Spam-* header fields;
+
+- relax argument parsing in amavisd-release to allow releasing of
+ quarantine id containing a body hash in a name (%b in template);
+ reported by Ron Rademaker;
+
+- skip a SQL-logging database operation if an associated clause in %sql_clause
+ is disabled, e.g. set to undef or ''; this allows for example to selectively
+ disable SQL logging based on a policy bank; thanks to Riaan Kok;
+
+- let LHA decoder (do_lha) recognize also other listing formats, e.g. MS-DOS,
+ symlinks, not just plain Unix archives; problem reported by Ryuhei Funatsu;
+
+
+OTHER
+
+- catch and log uncaught '__DIE__' and '__WARN__' Perl pseudo-signals
+ which would otherwise go to stderr and not be noticed with a daemonized
+ process; patch by Alexander 'Leo' Bergolth <leo at strike.wu-wien.ac.at>;
+
+- insert 'X-Spam-Flag: NO' if spam level is above tag level but below tag2
+ level, previously it was not inserted at all (it is still redundant, but
+ some may appreciate an explicit statement nevertheless);
+
+- drop support for Archive::Tar; main drawback of this module is: it either
+ loads an entire tar into memory (horrors!), or when using extract_archive()
+ it does not relativize absolute paths (which makes it possible to store
+ members in any directory writable by uid), and does not provide a way to
+ capture contents of members with the same name. Use pax program instead!
+
+- abandon the use of libnet (modules Net::SMTP and Net::Cmd), replaced by
+ own code to implement client-side SMTP and LMTP protocol support, with a
+ full support for pipelining and IPv6. Some of my issues with Net::SMTP go
+ back to year 2002, but the one that broke camel's back is a 3+ months
+ status quo in not fixing a serious misfeature introduced in 1.20, which
+ mangles 8-bit characters in mail, causing a series of support questions,
+ and even affecting some larger service providers without them realizing
+ there is a problem. Here are some relevant bug reports:
+ http://rt.cpan.org/Public/Bug/Display.html?id=24835
+ http://rt.cpan.org/Public/Bug/Display.html?id=2608
+ http://rt.cpan.org/Public/Bug/Display.html?id=2607
+ http://rt.cpan.org/Public/Bug/Display.html?id=14875
+ http://rt.cpan.org/Public/Bug/Display.html?id=9394
+
+ P.S. libnet-1.21 eventually fixed the UTF8 encoding problem and
+ added support for ENVID and AUTH options; other issues are still open
+ (e.g. split code/text in smtp status, no pipelining support, no LMTP);
+
+- represent score in X-Spam-Status header field as a single numeric field
+ instead of the previous explicit sum of a SA score and a score boost,
+ which some MUAs don't know how to interpret and label mail spaminess
+ incorrectly;
+
+- recipient notifications (in their default template) keep the original
+ To and Cc header fields, instead of placing a recipient envelope address
+ into a To header field; suggested by Jorgen Lundman;
+
+- no longer removes an X-Amavis-Alert header field (as inserted by some
+ foreign MTA), it does not hurt to keep it;
+
+- kavscanner AV entry (Kaspersky Antivirus): added a new entry to the
+ search list of paths to a binary, kavscanner changed its default
+ installation location again; provided by Gary V;
+
+- added a rule into a $map_full_type_to_short_type_re list, mapping a
+ file(1) type 'RIFF...animated cursor' into '.ani'; a patch by Eric;
+
+- example rules (commented-out) to block animated cursors (ANI) and icons
+ are added to configuration files amavisd.conf and amavisd.conf-sample;
+
+- suppress bounces not only for Precedence: (bulk|list|junk) or null return
+ path (as before), but also for mail with a 2822.From header field matching
+ one of: *-request | *-owner | *-relay | *-bounces | owner-* | postmaster |
+ mailer-daemon | mailer | uucp, and for mail containing a header field
+ List-Id (RFC 2919). Note that this new rule practically never occurs in
+ practice, true bounces and common mailing list traffic is already covered
+ by previous checks;
+
+- make available a list of detected virus names in
+ an Amavis::In::Message object (virusnames), suggested by Tom Sommer;
+
+- split SQL documentation into three files:
+ README.sql general SQL considerations and some examples
+ README.sql-mysql MySQL-specific notes and schema
+ README.sql-pg PostgreSQL-specific notes and schema (also SQLite)
+
+- README.sql-pg now adds CHECK (x >= 0) for fields that are supposed
+ to contain unsigned integers; suggested by Hanne Moa;
+
+- repurpose/rename a contents category CC_TEMPFAIL to CC_MTA; it is turned
+ on when MTA rejects or tempfails a mail when amavisd attempts to pass it
+ back after checking it; (this should not normally happen, MTA should
+ be doing most of its checks (e.g. recipient validation) before passing
+ mail to a content filter, not after);
+
+
+NEW FEATURES
+
+- custom hooks allow custom code to be called at few strategic places:
+ * during child process initialization: allows initialization of custom
+ code, including establishing an additional SQL session or similar;
+ * after built-in checks: allows custom code to inspect and/or modify
+ results of checks;
+ * before sending, allows for additional quarantining and for sending
+ additional notifications;
+ * at the end of message processing: allows inspecting results of checks
+ and status of mail forwarding (e.g. for statistics or logging purposes).
+
+ Mail processing sequence:
+
+ child process initialization
+ *custom hook: new()
+ loop for each mail:
+ receive mail
+ mail checking and collecting results
+ *custom hook: checks() - may inspect or modify checking results
+ deciding mail fate (lookup on *_lovers, thresholds, ...)
+ quarantining
+ sending notifications (to admin and recip)
+ *custom hook: before_send() - may send additional notifications,
+ additional quarantining, may modify mail
+ forwarding (unless blocked)
+ sending delivery status notification (if needed)
+ issue main log entry, manage statistics (timing, counters, nanny)
+ *custom hook: mail_done() - may inspect results
+ endloop after $max_requests or earlier
+
+ If Amavis::Custom::new returns undef then no further custom calls are made.
+ Otherwise, this method is supposed to return an object, thus enabling
+ other custom hooks, which receive this value as their first argument.
+
+ See amavisd-custom.conf for an example, and the example invocation
+ of amavisd-custom.conf at the end of file amavisd.conf-sample;
+ thanks to Kasscie (Yohanna Monsalvez);
+
+
+- formerly penpals could only match replies to previous outgoing mail
+ where envelope sender and recipient addresses are exactly reversed.
+ Now, in addition to this, penpals can also match replies which reference
+ previous outgoing mail by its 'Message-ID' (taking into account the
+ 'References' or 'In-Reply-To' header fields), even if the envelope
+ sender address of the reply is null or does not match a recipient address
+ of a previous outgoing mail. This covers for incoming replies to mailing
+ list postings, incoming message disposition notifications (MDN, RFC 3798)
+ and incoming replies from alias or role addresses. A query on a
+ message-id is fast compared to matching on recipient id, and if it
+ succeeds, the later one is skipped. Based on a suggestion and a patch
+ by Alexander 'Leo' Bergolth;
+
+ The %sql_clause now contains one additional SQL SELECT clause under
+ 'sel_penpals_msgid' key, which is used in place of 'sel_penpals' when
+ a list of references in a reply is nonempty. It is worth creating
+ an index on a msgs.message_id field to speed up SQL lookups:
+
+ CREATE INDEX msgs_idx_mess_id ON msgs (message_id);
+
+ Note that a spammer can gain a (small) advantage by including a reference
+ to a recent outgoing message in his message. With private correspondence
+ this information is hard to come by efficiently an in a timely manner,
+ and can effectively be exploited only if generated Message-IDs of outgoing
+ mail is very easy to guess (e.g. generated by a very poor random number
+ generator). With postings to public mailing lists this information is more
+ readily available. Setting $sql_clause{'sel_penpals_msgid'} to undef or
+ to empty disables matching on message-id, if it turns out the mechanism
+ is being actively exploited.
+
+
+- added penpals snmp-like counters, which are displayed by amavisd-agent
+ like the following:
+
+ PenPalsAttempts 10222 438/h 18.3 % (InMsgsRecipsInboundOrInt)
+ PenPalsHits 2957 127/h 48.9 % (ContentCleanMsgsInboundOrInt)
+ PenPalsHitsMid 88 4/h 3.0 % (PenPalsHits)
+ PenPalsHitsMidNullRPath 26 1/h 0.9 % (PenPalsHits)
+ PenPalsHitsMidRid 917 39/h 31.0 % (PenPalsHits)
+ PenPalsHitsRid 1926 82/h 65.1 % (PenPalsHits)
+
+ Some comments on the above figures:
+ - PenPalsAttempts shows that SQL lookups are performed only on a fraction
+ of all incoming and internal mail (and not at all on outgoing mail)
+ because a lookup is not performed on a high score spam (score above
+ $penpals_threshold_high, which was 8 in this example, typically
+ the same as kill level);
+ - PenPalsHits shows that more than half of incoming clean messages can be
+ associated with previous outgoing mail and can benefit from pen pals
+ soft-whitelisting, such message is either a reply directly referencing
+ a previous mail, or a new conversation between a sender/recipient pair
+ which had previous conversation on a different topic;
+ - PenPalsHitsMidRid shows that 1/3 of incoming matching replies match
+ a previous outgoing mail by both the Message-ID as well as the exact
+ recipient address; these are true replies to previous outgoing messages;
+ - PenPalsHitsRid shows that 2/3 of incoming matching replies match a
+ a previous outgoing mail by recipient address, but not by a Message-ID;
+ these messages usually correspond to new topics among previous
+ correspondents;
+ - PenPalsHitsMid match only in reference to a previous Message-ID, but not
+ by sender/recipient addresses; these are usually mailing list replies
+ to a previous posting by a local user;
+ - PenPalsHitsMidNullRPath are messages with null return path which are
+ matching some previous outgoing message by Message-ID; these are
+ usually incoming message disposition notifications (MDN, RFC 3798), or
+ certain bounces which specify In-Reply-To or References in their header;
+
+
+- configuration variable %defang_by_ccat is renamed to %defang_maps_by_ccat
+ and may now contain a list of by-recipient lookup tables (or a boolean
+ as before for compatibility); this allows defanging/mangling to be selected
+ on per-recipient basis; compatibility is retained by making the old
+ variable %defang_by_ccat an alias for %defang_maps_by_ccat;
+
+
+- provided interface code to allow mangling/defanging/sanitation
+ to be performed by an external utility, either by directly calling
+ a Perl module Anomy Sanitizer (within the same process, avoiding
+ startup cost), or by invoking a program 'altermime' (or by internal
+ defanging code as before); mail body mangling is only allowed
+ for recipients matching @local_domains_maps, as before;
+
+ If there is more than one mangling code option available, the result
+ of a %defang_maps_by_ccat can choose between them by returning one of
+ the following strings, the selection can depend on mail content type
+ and on by-recipient lookups if needed:
+ 'anomy' chooses Anomy Sanitizer (if $enable_anomy_sanitizer is true);
+ 'altermime' chooses a program whose path is $altermime (if found);
+ 'attach' chooses the traditional amavisd-new defanging method
+ which pushes an original mail message to an attachment;
+ 'null' for testing purposes - doesn't modify mail body, but
+ pretends it does (in logging and mail header);
+ other non-empty and non-zero value automatically choose one
+ of the above options depending on what is available;
+ at least the 'attach' is always available;
+ an empty, zero or undef value disables mail body modifications;
+
+ Controls: $enable_anomy_sanitizer, @anomy_sanitizer_args,
+ and: $altermime, @altermime_args_defang;
+
+ Typical use:
+
+ # with altermime:
+ $altermime = '/usr/local/bin/altermime';
+ @altermime_args_defang = qw(--verbose --removeall);
+
+ # with Anomy Sanitizer:
+ $enable_anomy_sanitizer = 1;
+ @anomy_sanitizer_args = qw( /usr/local/etc/sanitizer.cfg );
+
+ $defang_spam = 1; # old style, applies the first available mangler
+ # to all spam-loving local recipients
+
+ # unnecessarily complicated example of selective choices:
+ $defang_maps_by_ccat{+CC_BANNED} = [
+ 'altermime', # user altermime for everybody (a 'constant' lookup table)
+ ];
+ $defang_maps_by_ccat{+CC_SPAM} = [
+ { # a per-recipient hash lookup table
+ 'user at example.com' => 1, # old style, auto-selects one mangler
+ 'user-a at example.com' => 'anomy',
+ 'user-m at example.com' => 'altermime',
+ 'user-t at example.com' => 'attach',
+ '.example.net' => 0, # no mangling
+ },
+ $defang_spam, # fallback to old style setting if no match above
+ ];
+
+
+- a special case of mangling is adding a disclaimer, by invoking an external
+ program 'altermime' (if available and enabled). This differs from mangling
+ inbound mail in two details:
+ * uses a separately configurable list of arguments to altermime:
+ @altermime_args_disclaimer; and
+ * it applies only to mail submitted from internal networks or roaming users
+ (as recognized through a policy bank which sets: allow_disclaimers => 1),
+ and where any of the following addresses matches local domains:
+ author (2822.From) or sender (2822.Sender) or return path (2821.mail_from);
+
+ In addition to strings that may be returned by %defang_maps_by_ccat
+ as described above, there are two more, only taken into account
+ when $allow_disclaimers is true:
+ 'disclaimer' invokes $altermime program for outgoing mail with
+ arguments as given in @altermime_args_disclaimer;
+ 'nulldisclaimer' for testing purposes - doesn't modify mail body,
+ but pretends it does (in logging and mail header);
+
+ Typical use:
+
+ $altermime = '/usr/local/bin/altermime';
+ @altermime_args_disclaimer =
+ qw(--verbose --disclaimer=/etc/altermime-disclaimer.txt);
+ $defang_maps_by_ccat{+CC_CATCHALL} = [ 'disclaimer' ];
+ @mynetworks = qw( ... );
+ $policy_bank{'MYNETS'} = { # mail originating from our networks
+ allow_disclaimers => 1,
+ }
+
+ For the moment there is one limitation: there can only be one mangler
+ in effect at a time, it is not currently possible to both defang and
+ to append a disclaimer, e.g. for internal-to-internal mail inserting
+ a disclaimer takes precedence.
+
+ To make it possible to provide different disclaimer texts when hosting
+ multiple domains, there is an experimental additional configuration
+ variable available: the @disclaimer_options_bysender_maps.
+ It is a list of lookup tables, looked up by a sender address.
+ The sender address is chosen from the following list, first match wins:
+ * 'Sender:' header field, if its domain matches @local_domains_maps;
+ * 'From:' header field, if its domain matches @local_domains_maps;
+ * envelope sender address, if its domain matches @local_domains_maps;
+ We already know that at least one of the above will match, otherwise
+ adding disclaimers would be skipped at an earlier stage. The result of
+ lookups should be one simple string, which replaces a string '_OPTION_'
+ anywhere in @altermime_args_disclaimer elements.
+
+ Typical use:
+
+ @altermime_args_disclaimer = qw(--disclaimer=/etc/_OPTION_.txt);
+
+ @disclaimer_options_bysender_maps = (
+ { 'host1.example.com' => 'altermime-disclaimer-host1',
+ 'boss at example.net' => 'altermime-disclaimer-boss',
+ '.example.net' => 'altermime-disclaimer-net',
+ '.' => 'altermime-disclaimer-default' },
+ );
+
+ It is currently not possible to disable adding disclaimers through
+ @disclaimer_options_bysender_maps results. This needs to be improved.
+ The exact interpretation of the @disclaimer_options_bysender_maps lookup
+ result may change in the future (which is why I call it 'experimental').
+
+ Note that disclaimers are pretty much useless legally.
+ If you can help it at all, please avoid the pollution. See:
+ http://www.goldmark.org/jeff/stupid-disclaimers/
+
+
+- as mentioned above, the new SMTP/LMTP client code now supports a LMTP
+ protocol too. This allows amavisd-new to act as a LMTP-to-LMTP content
+ filter, possibly being inserted between MTA and a LMTP-based mail delivery
+ agent such as Cyrus (if checking of outgoing mail is not needed). LMTP is
+ selected when the first field of a $*_method (such as $forward_method,
+ $notify_method, $resend_method, $release_method, $*_quarantine_method)
+ is a 'lmtp:'.
+
+ Possible uses:
+ $forward_method = 'lmtp:/var/imap/socket/lmtp'; # over a Unix socket
+ or:
+ $forward_method = 'lmtp:[127.0.0.1]:24'; # over IPv4
+ or:
+ $forward_method = 'lmtp:[::1]:24'; # over IPv6
+
+ If a Postfix lmtp service is used to feed amavisd (instead of the more
+ usual content filter feed through service named 'amavisfeed' or
+ 'smtp-amavis'), make sure not to forget to limit the number of concurrent
+ feeds to amavisd (e.g. lmtp_destination_concurrency_limit=15) to a value
+ same (or less) than $max_servers, or limit the maxproc field in master.cf
+ such as: 'lmtp unix - - n - 15 lmtp' .
+
+ Note that placing amavisd as a LMTP delivery agent has a disadvantage
+ that outgoing mail is not being checked, so infected internal hosts are
+ able to pollute the world. Also the pen pals feature is no longer useful,
+ as it requires the information on previous outgoing mail to be present
+ in a SQL database.
+
+
+- a new command line option -i, it takes one argument which can be any
+ string (an instance/personality name), which is then made available to
+ amavisd.conf in a variable $instance_name (intended to be read-only);
+ code in amavisd does not assign any semantics to this argument and does
+ not use it for any purpose, it is purely intended for administrator's
+ use in amavisd.conf if desired; this simple mechanism may facilitate
+ running multiple instances of amavisd using a single configuration file,
+ or to choose at startup time between amavisd personalities using the
+ same config file;
+
+
+- policy banks now contain a new key 'originating', which generalizes
+ a previously hard-wired policy bank MYNETS. It is a boolean variable,
+ turned on automatically in the currently loaded policy bank when a
+ smtp client's IP address matches @mynetworks_maps, to retain full
+ compatibility with existing setups. When a new policy bank is loaded
+ over a current one, the new policy bank may also modify the 'originating'
+ key - a typical use is to turn it on by a policy bank activated by mail
+ submission from authenticated roaming users (SASL/AUTH), so that such
+ users are treated as locals (originating mail) even though their IP address
+ does not match a @mynetworks_maps list of lookup tables.
+
+ The current value of variable 'originating' is now the only control to
+ some macros or decisions, which were previously controlled implicitly
+ by a @mynetworks_maps match. These are:
+ * macro %l now directly corresponds to the current value of the
+ 'originating' variable (returning a '1' or an empty string);
+ * some statistics counters differentiate between 'Inbound' and 'Internal'
+ mail directly based on the value of the 'originating' variable
+ (applies to mail with local recipients, otherwise it is 'Outbound');
+ * pen pals lookups are performed only when 'originating' is false
+ (i.e. for all inbound or internal mail);
+ * there may be other uses in the future (e.g. DKIM signing perhaps),
+ so it is prudent to keep @mynetworks_maps and @local_domains_maps
+ configured correctly, and if necessary turn on the 'originating' flag
+ for mail that is supposed to be treated as originating from internal
+ or authorized roaming users;
+
+ Example use:
+
+ $interface_policy{'10026'} = 'ORIG';
+
+ $policy_bank{'ORIG'} = { # mail originating from our users
+ originating => 1, # declare that mail was submitted by our smtp client
+ allow_disclaimers => 1, # enables disclaimer insertion if available
+ virus_admin_maps => ["virusalert\@$mydomain"],
+ spam_admin_maps => ["virusalert\@$mydomain"],
+ warnbadhsender => 1,
+ # forward to a smtpd service providing DKIM signing service
+ forward_method => 'smtp:[127.0.0.1]:10027',
+ # force MTA conversion to 7-bit (e.g. before DKIM signing)
+ smtpd_discard_ehlo_keywords => ['8BITMIME'],
+ bypass_banned_checks_maps => [1], # allow sending any file names & types
+ terminate_dsn_on_notify_success => 0, # don't remove NOTIFY=SUCCESS opt.
+ };
+
+
+- make it possible for a virus scanner to derate an infection report
+ to a spam report, contributing to spam score and to spam report/status.
+ A new configuration variable @virus_name_to_spam_score_maps
+ (also member of policy banks) can turn a reported virus name
+ into a spam score. Its default setting is:
+
+ @virus_name_to_spam_score_maps =
+ (new_RE( [ qr'^(Email|HTML)\.(Phishing|Spam|Scam[a-z0-9]?)\.'i => 0.1 ],
+ [ qr'^(Email|Html)\.Malware\.Sanesecurity\.' => undef ],
+ [ qr'^(Email|Html)(\.[^., ]*)*\.Sanesecurity\.' => 0.1 ],
+ # [ qr'^(Email|Html)\.(Hdr|Img|ImgO|Bou|Stk|Loan|Cred|Job|Dipl|Doc)
+ # (\.[^., ]*)* \.Sanesecurity\.'x => 0.1 ],
+ [ qr'^(MSRBL-Images/|MSRBL-SPAM\.)' => 0.1 ],
+ ));
+
+ and can be replaced in amavisd.conf.
+ To disable the feature assign an empty list to the configuration variable:
+
+ @virus_name_to_spam_score_maps = ();
+
+ When a virus scanner returns names of viruses, and all provided names are
+ matched by the @virus_name_to_spam_score_maps, and no other virus scanner
+ has anything more sinister to report, then a message is _not_ flagged
+ as a virus, but a corresponding spam score is contributed to other
+ spam results as returned by a normal spam scan by SA. All the usual
+ spam rules are then followed. Phishing fraud as indicated by ClamAV
+ is now by default treated as spam, and no longer as a virus.
+
+ The log can now show entries like:
+
+ amavis[26733]: (26733-03-2) Turning AV infection into a spam report:
+ score=0.1, AV:HTML.Phishing.Auction-289=0.1
+
+ amavis[26733]: (26733-03-2) adding SA score 38.628 to existing 0.1
+ from an earlier spam check
+
+ amavis[26733]: (26733-03-2) Blocked SPAM, ... Hits: 34.728, ...
+ Tests: [AV:HTML.Phishing.Auction-289=0.1, ... L_AV_Phish=14, ...]
+
+ The information is also available to SA rules in a form of a synthetic
+ header field X-Amavis-AV-Status which will be seen by SA only (not inserted
+ into passed or quarantined mail). One has a choice to adjust scores either
+ in the @virus_name_to_spam_score_maps in amavisd.conf, or by providing rules
+ to match on the provided header field. Doing it by SA rules has an advantage
+ of letting other SA rules contribute their score points, possibly preventing
+ a false positive of a ClamAV rule, or pushing score even higher for a clean
+ bounce suppression. It also makes more sense when checks are cached and
+ result reused later for some other message with the same contents in body.
+
+ Here is one example of such SA rules (some long lines are wrapped,
+ these should be unwrapped before placing them into local.cf):
+
+ header L_AV_Phish X-Amavis-AV-Status =~
+ m{\b(Email|HTML)\.Phishing\.}i
+ header L_AV_SS_Phish X-Amavis-AV-Status =~
+ m{\b(Email|Html)\.Phishing(\.[^., ]*)*\.Sanesecurity\.}
+ header L_AV_SS_Scam X-Amavis-AV-Status =~
+ m{\b(Email|Html)\.(Scam[A-Za-z0-9]?)(\.[^., ]*)*\.Sanesecurity\.}
+ header L_AV_SS_Spam X-Amavis-AV-Status =~
+ m{\b(Email|Html)\.(Spam|Bou|Stk|Loan|Cred|Job|Dipl|Doc)
+ (\.[^., ]*)*\.Sanesecurity\.}
+ header L_AV_SS_Hdr X-Amavis-AV-Status =~
+ m{\b(Email|Html)\.Hdr(\.[^., ]*)*\.Sanesecurity\.}
+ header L_AV_SS_Img X-Amavis-AV-Status =~
+ m{\b(Email|Html)\.(Img|ImgO)(\.[^., ]*)*\.Sanesecurity\.}
+ header L_AV_MSRBL_Img X-Amavis-AV-Status =~ m{\bMSRBL-Images/}
+ header L_AV_MSRBL_Spam X-Amavis-AV-Status =~ m{\bMSRBL-SPAM\.}
+
+ score L_AV_Phish 14
+ score L_AV_SS_Phish -3
+ score L_AV_SS_Scam 8
+ score L_AV_SS_Spam 8
+ score L_AV_SS_Hdr 6
+ score L_AV_SS_Img 3.5
+ score L_AV_MSRBL_Img 3.5
+ score L_AV_MSRBL_Spam 6
+
+
+- added a new concept of a 'blocking contents category', which in most cases
+ corresponds to a familiar 'main contents category' (the highest ranking
+ category of contents pertaining to a message, e.g. virus, blocked, spam,
+ spammy, bad header ...). The difference between the two arises when
+ recipients are declared to be 'lovers' of some higher-ranking contents,
+ or when a higher ranking contents category has its *_destiny set to D_PASS.
+
+ For example: a message contains a banned part, but is also spam
+ and may even have a bad header. Its contents categories are (simplified):
+ CC_BANNED, CC_SPAM and CC_BADH, in this order. The main contents
+ category of a message is CC_BANNED, which usually is also a reason
+ for blocking a message, yielding a blocking ccat to also be CC_BANNED.
+
+ But if some recipient is banned_files_lover (or if $final_banned_destiny
+ is set to D_PASS), then the main ccat remains to be CC_BANNED, but the
+ blocking ccat is CC_SPAM, i.e. the next in the list which is responsible
+ for actually blocking the mail. If recipient would also be a spam lover,
+ the blocking ccat might be CC_BADH (if $final_bad_header_destiny were
+ not D_PASS);
+
+ If a message is not being blocked, the 'blocking contents category'
+ (i.e. a blocking_ccat attribute of a per-message or a per-recipient object)
+ remains empty (undefined). For convenience some internal routines
+ and some new macros fall back to showing the main contents category
+ in this case.
+
+ Almost all processing decisions, DSN, notification assembling, quarantining,
+ logging etc. is now based on 'blocking contents category' when a message
+ is being blocked, and on 'main contents category' (as before) when a
+ message is not being blocked.
+
+ There is a new macro 'ccat' which is useful in notification and logging
+ templates, which can query the blocking contents category, as well
+ as a main contents category. It provides access to information that
+ was formerly available through macros ccat_maj, ccat_min, ccat_name,
+ plus access to additional information. Macros ccat_maj, ccat_min and
+ ccat_name are still available, but their use is deprecated, as their
+ functionality has been incorporated into the new macro 'ccat'.
+
+ Macro 'ccat' takes two optional fixed-string arguments, which are
+ interpreted case-insensitively. In their absence it expands to a
+ string "(maj,min)" which shows a major and a minor contents category
+ number of a blocking ccat for a blocked message, and of a main contents
+ category for a passed message.
+
+ The first argument specifies which attribute of a ccat is to be provided,
+ the second argument specifies whether a main or a blocking contents
+ category is to be consulted:
+
+ The first argument may be any of the following strings:
+ name ... provide a human-readable name of a ccat (%ccat_display_names)
+ major ... provide a number: a major contents category,
+ values correspond to CC_* constants in the program
+ minor ... provide a number: a minor contents category, often a 0
+ <empty>... empty argument (also a default) results in a string "(maj,min)"
+ is_blocking ... '1' if blocking_ccat is true (message is being blocked),
+ or an empty string when a message is being passed;
+ is_nonblocking .. the opposite: '1' if blocking_ccat false, '' otherwise
+ is_blocked_by_nonmain .. '1' if blocking_ccat is true
+ _and_ is different from a main contents category;
+
+ The second argument may be any of the following strings:
+ main ... provide information on main contents category
+ when asked for name/major/minor/<empty>
+ blocking.. provide information on blocking contents category if it exists,
+ otherwise it falls back to providing info on main ccat;
+ this is also a default in the absence of this argument;
+
+ For illustration, instead of a former call [:ccat_maj] use [:ccat|major] ,
+ instead of [:ccat_min] use [:ccat|minor], and instead of [:ccat_name]
+ please use [:ccat|name] . For more examples please consult the default
+ templates, glued to the end of file 'amavisd'.
+
+
+- when amavisd-nanny is given an invalid command-line argument
+ it now shows 'Usage: ...' as well as a legend for process states;
+
+
+- amavisd-nanny enhanced and new process-state instrumentation added to
+ amavisd daemon; previously only busy/idle states of child processes were
+ shown in amavisd-nanny output, now a more detailed process state can be
+ shown by setting a new verbosity control configuration variable
+ $nanny_details_level to a higher than a default value of 1, e.g. to 2;
+
+ The following characters in amavisd-nanny bars represent amavisd child
+ process states as follows, in the shown order of events:
+
+ A accepted a connection
+ b begin with a protocol for accepting a request
+ m 'MAIL FROM' smtp command started a new transaction in the same session
+ d data transfer from MTA to amavisd
+ = content checking just started
+ D decoding of mail parts
+ V virus scanning
+ S spam scanning
+ P pen pals database lookup and updates
+ r preparing results
+ Q quarantining and preparing/sending notifications
+ F forwarding mail to MTA
+ . content checking just finished
+
+ A nanny bdb database has changed in an incompatible way, so older
+ versions of amavisd-nanny would complain about contents of a new
+ database. Backward compatibility is retained: new version of
+ amavisd-nanny is able to deal with a database from older versions
+ of amavisd-new. There is no need for conversion, a new database
+ is created on each amavisd restart.
+
+ Note that a history of process states is _not_ maintained in a nanny
+ database, but only in a running amavisd-nanny, which is why a just-started
+ amavisd-nanny can not show previous states of processes from time before
+ amavisd-nanny was started - a '=' is shown instead. A display eventually
+ catches up and all newly-entered states are shown correctly.
+
+
+- snmp-like database can now also store 64-bit counters data type,
+ amavisd-agent utility modified accordingly;
+
+
+- amavisd-agent utility and amavisd daemon enhanced to provide and to display
+ cumulative elapsed time by sections; currently only some of the more
+ important sections have been instrumented, e.g.:
+
+ TimeElapsedReceiving 10631 s 0.608 s/msg (InMsgs)
+ TimeElapsedDecoding 970 s 0.056 s/msg (InMsgs)
+ TimeElapsedVirusCheck 629 s 0.036 s/msg (InMsgs)
+ TimeElapsedSpamCheck 75866 s 4.341 s/msg (InMsgs)
+ TimeElapsedPenPals 2150 s 0.123 s/msg (InMsgs)
+ TimeElapsedSending 2231 s 0.128 s/msg (InMsgs)
+ TimeElapsedTotal 94709 s 5.419 s/msg (InMsgs)
+
+ Don't be surprised if the total elapsed time exceeds amavisd uptime,
+ 10 processes progressing slowly for 5 seconds each will accumulate
+ 50 seconds of reported elapsed time. The average seconds-per-message
+ figure as reported in the last column makes more sense;
+
+
+- amavisd-agent utility and amavisd daemon enhanced to provide and to
+ display cumulative mail sizes in bytes, and additional message counters
+ based on outbound/inbound/internal mail direction:
+ message counts:
+ InMsgs all mail received by amavisd (as in previous versions);
+ InMsgsOutbound at least one recipient NOT in @local_domains_map;
+ InMsgsInternal at least one recipient in @local_domains_map,
+ and client IP address (submitter) in @mynetworks_maps;
+ InMsgsInbound at least one recipient in @local_domains_map,
+ and client IP address NOT in @mynetworks_maps;
+ message sizes:
+ InMsgsSize total mail size in bytes as received by amavisd;
+ InMsgsSizeOutbound as above, but with at least one recipient
+ NOT in @local_domains_map;
+ InMsgsSizeInternal at least one recipient in @local_domains_map,
+ and client IP address in @mynetworks_maps;
+ InMsgsSizeInbound at least one recipient in @local_domains_map,
+ and client IP address NOT in @mynetworks_maps;
+ Note that a mail with multiple recipients can be both internal and outbound.
+
+ Example output:
+
+ InMsgsSize 4332MB 109MB/h 100.0 % (InMsgsSize)
+ InMsgsSizeInbound 3102MB 78MB/h 71.6 % (InMsgsSize)
+ InMsgsSizeInternal 554MB 14MB/h 12.8 % (InMsgsSize)
+ InMsgsSizeOutbound 816MB 21MB/h 18.8 % (InMsgsSize)
+
+
+- new configuration variables $always_bcc and %always_bcc_by_ccat, also
+ members of policy banks, allow adding one extra envelope recipient to
+ each message, either regardless of contents ($always_bcc), or selectively
+ based on contents category. For example:
+
+ $always_bcc = 'archiver+clean at example.com';
+
+ or selectively based on contents category:
+
+ $always_bcc_by_ccat{+CC_CLEAN} = 'archiver+clean at example.com';
+ $always_bcc_by_ccat{+CC_VIRUS} = 'archiver+virus at example.com';
+
+ or as a member of policy banks:
+
+ $policy_bank{'MYNETS'} = {
+ always_bcc_by_ccat => {
+ CC_BADH, 'archiver at example.com',
+ CC_CLEAN, 'archiver at example.com',
+ CC_CATCHALL, undef,
+ },
+ };
+
+- amavisd-nanny and amavisd-agent utilities now recognize an optional
+ command-line option: -w <wait> , where the specified value is time
+ in seconds between re-displays. The default interval is 2 seconds
+ for amavisd-nanny, and 10 seconds for amavisd-agent as before.
+ The specified interval time may be fractional;
+
+- macro 'useragent' can accept an optional argument: a string 'name' or 'body',
+ restricting the information to be returned as follows: macro 'useragent'
+ returns 'User-Agent: ...' or 'X-Mailer: ...' header field from a message
+ (whichever is present, or empty); an optional argument specifies whether:
+ an entire field is to be returned (empty or unrecognized argument),
+ or just a field name (argument: 'name'), e.g. 'X-Mailer';
+ or just a field body (argument 'body'), e.g. 'Thunderbird_1.5.0.9';
+
+- interfacing to Mail::ClamAV (a perl module to a clamav library) now
+ performs processing in a subprocess to prevent bugs in external library
+ from bringing down amavisd process, and to prevent virtual memory of
+ an amavisd child process from expanding uncontrollably - at the expense
+ of additional 20..30 ms for a fork;
+
+- extended AM.PDP protocol with an attribute 'policy_bank' which may be used
+ in a client's request to require loading additional policy banks, e.g.:
+ policy_bank=TLS,ORIGINATING,MYNETS
+ Its value is a comma-separated list of policy bank names. Names of
+ nonexistent banks are silently ignored, so are leading and trailing spaces
+ and TABs around each name. The order of policy bank loading generally
+ follows the order in which information about a message were obtained:
+ - interface- or socket-based policy banks (when MTA connects to amavisd);
+ - MYNETS (when client's IP address becomes known);
+ - the list of policy bank names as specified in a
+ 'policy_bank' attribute of AM.PDP protocol, comma-separated;
+ - MYUSERS (when sender's e-mail address becomes known);
+
+- added a field 'Final-Log-ID' to a DSN report (RFC 3464), which will
+ provide information on log_id and mail_id, e.g. '77790-10-3/uez9wtcVNTO5'
+ in a standard way, much like the 'Our internal reference code for your
+ message is: ...' in a DSN plain text part;
+
+- added mapping from 'RIFF...animated cursor' to ['movie','ani']
+ in $map_full_type_to_short_type_re, to facilitate blocking animated
+ cursors (Microsoft Windows ANI header stack buffer overflow is being
+ actively exploited); by Henrik Krohns;
+
+- add support for 7-Zip archives if external utility program 7z
+ is available (under names 7zr, 7za or 7z); requested by Bob Marcan;
+ see: http://www.7-zip.org/
+
+- add configurable global settings $min_servers, $min_spare_servers, and
+ $max_spare_servers (all undefined by default, see Net::Server::PreFork
+ documentation for their semantics), pass them to Net::Server at startup
+ time (complementing the usual $max_servers setting) and allow to choose
+ between Net::Server personalities Net::Server::PreForkSimple and
+ Net::Server::PreFork - if $min_servers is defined the PreFork is chosen,
+ otherwise the more usual PreForkSimple. The feature is mostly intended for
+ use of amavisd as a pre-queue content filter, which is unsupported anyway.
+ For normal post-queue use the PreForkSimple already does a good job.
+ Based on a patch by Alexander 'Leo' Bergolth;
+
+- internal: incompatibly changed order and indirection level of arguments
+ to routines dealing with contents categories, and some of their names;
+
+- new macro 'join', behaves like a Perl join function: the first argument
+ is a separator string, remaining arguments are strings to be concatenated,
+ with a separator string inserted at every concatenation point;
+
+- macro 'dquote' (as used in a default log template to protect Subject)
+ previously escaped double quotes with \, but missed to escape \ itself,
+ making log parsing tricky; also, as logging layer escapes \ by itself,
+ the result was ugly and inconsistently parsable; new behaviour
+ is to protect a double quote within a string by doubling it, so
+ a [dquote|one"oops"two] now yields "one""oops""two", instead of
+ "one\"oops\"two", which when logged showed as "one\\"oops\\"two";
+
+- convenience: do not drop privileges early despite a command line option -u
+ when an option -R is also specified with a non-empty (and non-slash) value,
+ otherwise the requested chroot operation is not possible (root privileges
+ are required for chrooting);
+
+- version 2.4.3 introduced some substitutions of subject tag template strings:
+ SCORE, REQD, YESNO and YESNOCAPS; this list is now extended with few more,
+ to facilitate cross-host troubleshooting; the full list now consists of:
+
+ _SCORE_ spam score (hits), same as macro %c
+ _REQD_ tag2_level
+ _YESNO_ score above tag2_level? 'Yes' or 'No'
+ _YESNOCAPS_ same, but yields: 'YES' or 'NO'
+ _HOSTNAME_ fqdn of this host ($myhostname), same as macro %h
+ _DATE_ rfc2822 timestamp of mail entering this amavisd, as macro %d
+ _U_ iso8601 UTC timestamp of mail entering this amavisd, as %U
+ _LOGID_ log id (am_id) as shown in the log, e.g. 58725-05-2, as %n
+ _MAILID_ mail_id as used in quarantine names, e.g. jaUETfyBMJHG, as %i
+
+ See also README.customize for explanation of macros.
+
+
+---------------------------------------------------------------------------
+ January 30, 2007
+amavisd-new-2.4.5 release notes
+
+SECURITY
+
+- Recommended version of Convert::UUlib is 1.08 or higher
+ to avoid processing of uninitialized data containing 'random' garbage.
+
+ Note that a security hole in uulib which comes with Convert::UUlib 1.04
+ and older is now (as of 2006-12-05) known to be exploitable:
+ http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2005-1349
+ credits to Jean-Sébastien Guay-Leroux;
+
+- p0f-analyzer.pl will no longer reply to queries coming from low-numbered
+ UDP ports below 1024 or from nfsd port 2049, and will ignore queries
+ with nonce longer than 1024 character or containing characters outside
+ of \040-\177 range to limit its usefulness as a potential reflector
+ for an attacker from internal networks.
+
+
+INCOMPATIBLE CHANGE WITH 2.4.4
+
+- p0f-analyzer.pl now only binds to a loopback interface by default, instead
+ of to all interfaces; change $bind_addr in p0f-analyzer.pl to '0.0.0.0'
+ if p0f-analyzer.pl is running on a different host from amavisd or from
+ other querying clients; suggested by Shaun T. Erickson and Mario Liehr;
+
+
+BUG FIXES
+
+- let p0f-analyzer.pl exit when a pipe on stdin is closed (e.g. when p0f
+ is killed or crashes), instead of entering a tight loop; reported by
+ Justin Piszcz and Henrik Krohns;
+
+- hard-blacklisting no longer skips quarantining when
+ $spam_quarantine_cutoff_level is undefined (or is an empty string);
+
+- restart timer after Sophie times out; previously the next attempt
+ would run with no time limit; reported by Nick Leverton and
+ Nicklas Bondesson;
+
+- fixed AM.PDP code to always provide smtp-quoted form in angle brackets
+ in delrcpt and addrcpt attributes of a response, i.e. in the same form
+ as was received in sender and recipient attributes;
+
+- fix error reporting in open_on_specific_fd when POSIX::dup2 fails;
+ thanks to Chris (decoder);
+
+- fix signal handling in read_snmp_variables() and register_proc(),
+ a signal could previously get lost (not re-signaled) if it occurred
+ within these subroutines;
+
+- fixed get_body_digest which incorrectly determined 7- or 8-bitness
+ of mail header and body, setting body_type incorrectly (with only
+ cosmetic ill-effects);
+
+- AM.PDP protocol: ensure proper address form is used in server response
+ attributes 'delrcpt' and 'addrcpt': the same form should be used as
+ in 'sender' and 'recipient' attributes. The attribute value syntax is
+ specified in RFC 2821 as 'Reverse-path' (i.e. smtp-quoted form, enclosed
+ in <>); previously enclosing angle brackets were missing in a server reply;
+
+- documentation - amavisd.conf-default incorrectly stated that a default
+ value for $prepend_header_fields_hdridx is 1; actually the default is 0
+ as correctly indicated in release notes; reported by Jo Rhett;
+
+
+OTHER
+
+- qmail interfacing notice:
+ MTA timeout for waiting on results from amavisd should be longer than
+ $child_timeout (8 minutes by default) with some margin, setting MTA timeout
+ to 15 or 20 minutes is usual. With qmail however the QMQP code in qmail
+ has hard-coded timeouts set, 10 seconds for connect and 60 seconds for
+ read/write. If amavisd processing takes longer than 60 seconds, the MTA
+ drops connection and retries later, yet amavisd continues processing
+ and eventually delivers a mail (with each MTA retry), causing repeated
+ deliveries of the same message. The following patch by Eric Huss on
+ the www.qmail.org page: http://www.ehuss.org/qmail/qmqpc-timeout.tar.gz
+ should be applied to qmail when interfacing it to a post-queue content
+ filter. Problem researched by Nicklas Bondesson;
+
+- better timeout handling in interface code to daemonized virus scanners
+ like clamd, Sophie, Trophie: allow short time (10 s) for connect and
+ for sending a request, then allow normal (long) time to collect results;
+ keep evidence of the initial deadline on retries;
+
+- prefer '7bit' as Content-Transfer-Encoding when attaching original message
+ or its headers (message/rfc822 or text/rfc822-headers) to DSN or to a
+ defanged mail, and only specify '8bit' when necessary;
+
+- remove protecting the $ and @ characters in second argument
+ of a regexp selector macro, it is unnecessary and confusing;
+
+- macros %m, %r and header_field now return parsed and sanitized
+ message IDs in header fields Message-ID, Resent-Message-ID. In-Reply-To,
+ or References, void of CFWS (comments and FWS as specified by RFC 2822),
+ through the use of new subroutine parse_message_id();
+
+- when logging to SQL, the field msgs.message_id now contain just
+ a message id, without CFWS and other garbage that might appear in
+ a Message-ID header field; this facilitates queries, and pen pals
+ matching of IDs in In-Reply-To or References header fields
+ of a reply to an original Message-ID;
+
+- updated $map_full_type_to_short_type_re to avoid mapping file(1) result
+ 'MS-DOS executable (built-in)' to types 'exe-ms' and 'exe'; the file(1)
+ utility generously declares any text file starting with LZ to be a
+ 'MS-DOS executable (built-in)'; thanks to Noel Jones, Jakob Curdes
+ and Clifton Royston for troubleshooting;
+
+- add X-Spam-* header fields to quarantined mail if spam score is at or
+ above tag_level. Previously message needed to be recognized as spammy
+ or spam (tag2 or kill level) in order to receive spam header fields
+ in quarantined copy. This also makes it more consistent with adding
+ such header fields to passed mail; suggested by Michael Gaskins;
+
+- add X-Amavis-OS-Fingerprint header field to quarantined mail;
+
+- header field X-Spam-Score in a passed or quarantined mail now reflects
+ score boost even when SA score is unknown (e.g. when SA was not called),
+ and reflects white and blacklisting by pushing score to 0 or 64, to
+ make it consistent with a bar size in X-Spam-Level header field;
+
+- resignal "timed out" after (almost) every eval {} which has no subsequent
+ call to prolong_timer() to ensure we do not continue running with
+ disabled timer. Exceptions are DESTROY and END handlers, and code which
+ handles timer in some other way (e.g. by keeping evidence of a deadline);
+
+- for the purpose of looking up client IP address in @mynetworks_maps,
+ treat unknown/unavailable IP address as 0.0.0.0; this allows treating
+ directly submitted mail on the MTA host (not submitted through SMTP) as
+ coming from IP address 0.0.0.0 (i.e. "This" Network - according to RFC 1700);
+
+ Note that this is indistinguishable from other reasons when IP address
+ is not made available to amavisd, e.g. when smtp_send_xforward_command
+ option in Postfix smtp service is not enabled, which is why the default
+ setting of @mynetworks does not include a 0.0.0.0/8 network to prevent
+ falsely loading a MYNETS policy bank.
+
+ One should add 0.0.0.0/8 to a @mynetworks list only when XFORWARD is known
+ to work and if some software on the MTA host is submitting its mail to MTA
+ directly, e.g. through a sendmail command, and MYNETS policy bank loading
+ is needed for proper processing of such mail;
+
+- report a more informative message when a file(1) utility fails to produce
+ useful results: joins exit status with a parsing report into one message;
+ thanks to Andres, whose file(1) utility was crashing with SEGV;
+
+- consistency: rearrange implicitly adding $X_HEADER_TAG to a hash
+ %allowed_added_header_fields so that it is possible to turn off
+ insertion of $X_HEADER_TAG header field by turning off associated key in
+ %allowed_added_header_fields even when $X_HEADER_TAG is explicitly defined;
+
+- let %allowed_added_header_fields also control insertion of header fields
+ into quarantined message;
+
+- amavisd-nanny now displays a title line indicating the semantics of columns;
+
+- Courier patch: ensure the information is stored to newly introduced
+ recip_addr_smtp and sender_smtp object attributes, which are needed
+ to preserve pristine address forms for DSN and ORCPT use and for logging;
+ a patch by Martin Orr;
+
+- qmqpqq (qmail): ensure the information is stored to newly introduced
+ recip_addr_smtp and sender_smtp object attributes;
+
+- qmail patch now activates line-by-line sending to qmail to avoid qmail bug
+ ('bare LF' reported when CR and LF are separated by a TCP packet boundary);
+
+- tighten a regexp on matching a p0f fingerprint for Windows XP to avoid
+ matching 'Windows XP SP1+, 2000 SP3'; suggested by Michael Scheidell;
+
+- updated AV entry for CentralCommand Vexira (vascan):
+ removed hard-coded option '--vdb'; by Brian Wong;
+
+- internal: move code dealing with a SA call to a dedicated
+ subroutine call_spamassassin;
+
+- internal: provide new routines to collect scalar and structured results
+ from a subprocess (collect_results, collect_results_structured) and
+ take advantage of them in decoding, in AV and in dspam interface routines,
+ unifying code and providing results size sanity limit and consistent
+ killing of runaway external programs;
+
+- experimental: taking advantage of the above, make it possible to run SA in
+ a spawned process, requested by setting a new config variable $sa_spawned
+ to true (it is off by default); benefits are that a mainstream child process
+ can not be brought down by potential processing problems in SA or its
+ external modules, and timeouts are handled cleanly by a calling process;
+ downside is an increase of process count (worst case: doubled), with
+ corresponding increase in memory footprint, plus about 20 .. 30 ms
+ of additional processing time for each call to SA;
+
+- added a tuning tip on buffer sizes to README.sql for MySQL with InnoDB,
+ by Wayne Smith;
+
+- updated URL of Sophie AV scanner;
+
+
+---------------------------------------------------------------------------
+ November 20, 2006
+amavisd-new-2.4.4 release notes
+
+COMPATIBILITY WITH 2.4.3
+
+- PostgreSQL quarantining: data type of field quarantine.mail_text should
+ be 'bytea' (instead of 'text') to allow storing arbitrary octets without
+ associating them with a character set. See below for a conversion of an
+ existing database. Similarly with MySQL the data type should be 'blob'.
+
+- Note: in a sendmail milter setup with Petr Rehor's helper program
+ amavis-milter, one should set: $prepend_header_fields_hdridx = 1;
+ when dk or dkim signing milters are used in the same setup.
+ See below for details.
+
+
+BUG FIXES:
+
+- do_ascii: fix a bug where timer was not restored after decoding of a
+ textual mail part, so a timeout for subsequent decoding operations
+ on the same message was limited to 10 seconds (and to 30 seconds
+ for a call to SpamAssassin), regardless of $child_timeout setting;
+
+- don't call PerlIO::get_layers with Perl 5.8.0, the function was
+ introduced with 5.8.1; reported by Joel Nimety;
+
+- avoid deep recursion in evaluating a regular expression in header checks
+ which caused very slow testing for presence of a all-whitespace lines
+ in folded header fields for degenerate cases of header; the inefficient
+ expression was introduced with amavisd-new-2.4.0; reported and a sample
+ provided by Kai Risku;
+
+- when spam above kill level is to be passed and spam defanging is enabled,
+ SA summary was inserted twice (once for mail contents category being
+ CC_SPAMMY and once for CC_SPAM), fixed. Reported by Gary V and MHahnen;
+
+- when logging directly to a file, do create a log file if it does
+ not already exist; (bug introduced with 2.4.3)
+
+- make sure a quota limit is untainted when it is given as a command line
+ parameter to external TNEF decoder; reported by MK;
+
+- updated Courier patch to loosen up socket protection and allow
+ group write access to the socket; reported by Bill Taroli;
+
+- SQL logging: cleanly chop an UTF-8 octet sequence according to RFC 3629
+ (avoid truncating character octet sequence tail) when Subject, From or
+ Message-Id header field is longer than 255 characters;
+
+- PostgreSQL: when storing mail text to a quarantine use pg_type=PG_BYTEA
+ attribute on a field 'quarantine.mail_text'; previously the following
+ error could be reported:
+
+ 451 4.5.0 Storing to sql db as mail_id ... failed:
+ writing mail text to SQL failed: Error closing, flush:
+ sql inserting text failed,
+ sql exec: err=7, 22P02, DBD::Pg::st execute failed:
+ ERROR: invalid input syntax for type bytea
+
+- updated documentation in README.sql to suggest using data type 'bytea'
+ instead of inappropriate data type 'text' for a field quarantine.mail_text
+
+ To convert an existing table (when quarantining to SQL) please use:
+
+ ALTER TABLE quarantine ALTER mail_text TYPE bytea
+ USING decode(replace(mail_text,'\\','\\\\'),'escape');
+
+ If conversion of data type for 'quarantine.mail_text' is not done,
+ the following error will be reported when storing a message to a SQL
+ quarantine is attempted:
+
+ TROUBLE in check_mail: quar+notif FAILED:
+ temporarily unable to quarantine:
+ 451 4.5.0 Storing to sql db as mail_id ... failed:
+ writing mail text to SQL failed: Error closing, flush:
+ sql inserting text failed, sql exec: err=7, DBD::Pg::st execute failed:
+ ERROR: column "mail_text" is of type text but expression is of type bytea
+ HINT: You will need to rewrite or cast the expression
+
+ If converting quarantine table is not desirable or possible in a short term,
+ it is possible to continue use existing SQL quarantine table without
+ conversion by specifying the following in amavisd.conf:
+
+ $sql_clause{'ins_quar'} =
+ "INSERT INTO quarantine (mail_id, chunk_ind, mail_text)".
+ " VALUES (?,?,encode(?,'escape'))";
+
+ $sql_clause{'sel_quar'} =
+ "SELECT decode(mail_text,'escape') FROM quarantine".
+ " WHERE mail_id=? ORDER BY chunk_ind";
+
+ This will allow PostgreSQL to convert data types on-the-fly, converting
+ octets (any byte) into escaped text, and vice versa when releasing from
+ a quarantine;
+
+ Problem reported by Justin Hillyard, correct data type suggested by
+ Nikola Milutinovic;
+
+- MySQL: updated documentation in README.sql to suggest using data
+ type 'blob' instead of inappropriate data type 'text' for a field
+ quarantine.mail_text. To convert an existing table please use:
+ ALTER TABLE quarantine CHANGE mail_text mail_text blob;
+ Seems like MySQL does not complain on incompatibility between provided
+ data type and a data type of a field in table, but there are reports that
+ MySQL may silently truncate data which it finds violating character set
+ constraints, so conversion to 'blob' is highly recommended. Truncation
+ of quarantined message at an 8-bit character reported by Lubor Kolar.
+
+
+OTHER CHANGES:
+
+- limit recursion in MIME::Parser to $MAXFILES to prevent MIME parser from
+ fully traversing degenerate cases of broken MIME messages which can take
+ excessive amount of time and memory; reported and a sample provided by
+ Joshua Goodall, solution suggested by David F. Skoll;
+
+- check for already running daemon at startup time, preventing a user
+ mistake of trying to start another instance of the daemon without
+ stopping the currently running process; suggested by Jo Rhett;
+
+- keep sender and recipient addresses in original unparsed form (in addition
+ to an internal form) to be able to always provide exact original address
+ in delivery status notifications, in ORCPT, and when appending extensions
+ in a milter setup (AM.PDP), which requires exact matching to the original
+ form (without stripping route and without fixing poorly SMTP-quoted
+ address forms);
+
+- new configuration variable %allowed_header_tests, also member of policy
+ banks, allows for selectively disabling some of the header checks,
+ e.g. checks for non-encoded 8-bit characters. The %allowed_header_tests
+ hash contains all available header tests as its keys by default
+ (with a value of true); removing a key, or setting its value to false,
+ disables a test, e.g.:
+ $allowed_header_tests{'8bit'} = 0;
+ $allowed_header_tests{'missing'} = 0;
+ Currently available keys (i.e. tests) are:
+ other mime 8bit control empty long syntax missing multiple
+ each corresponding to its own minor contents category of CC_BADH;
+
+ ccat test
+ min name description
+ --- ------- -----------
+ 0 other (catchall for everything else, normally not used)
+ 1 mime Bad MIME (sub)headers or bad MIME structure
+ 2 8bit Invalid non-encoded 8-bit characters in header
+ 3 control Invalid control characters in header (CR or NUL)
+ 4 empty Folded header field made up entirely of whitespace
+ 5 long Header line longer than RFC 2822 limit of 998 characters
+ 6 syntax Header field syntax error
+ 7 missing Missing required header field
+ 8 multiple Duplicate or multiple occurrence of a header field
+ legend:
+ ccat min: minor contents category under a major category CC_BADH,
+ available in templates as a macro ccat_min;
+ test name: corresponding test name - a key in %allowed_header_tests;
+ descr.: description of a header test or MIME subheaders/structure test;
+
+- timing report has a couple of new entries to facilitate troubleshooting:
+ header checks section, separate entry for header and body digests,
+ check_mail initialization, entries 'SMTP greeting' and 'SMTP response';
+
+- when exec in a forked process fails, call POSIX::_exit with exist status
+ 8 (ENOEXEC) instead of the more general 1 to make the failure more obvious;
+
+- initialize logging earlier so that do_log may be called earlier during
+ program startup; also log attempts to stop and to reload, including
+ unsuccessful ones;
+
+- avoid logging by a forked process before exec, when there is a chance the
+ log file descriptor is in a range 0..2;
+
+- sub run_command and run_command_consumer: distinguish between undefined
+ and empty values of argument $stderr_to, undef now prevents reopening of
+ file descriptor 2, making it possible for the caller to keep it attached
+ to the current stderr; this is useful when run_command is called by the
+ master process before logging has been configured;
+
+- SQL: explicitly call DBI::bind_param to be able to specify data types
+ of values passed in @args to Amavis::Out::SQL::Connection::execute;
+
+- bump up buffer size from 16 kB to 64 kB in some cases of copying data
+ from/to a pipe, mostly to reduce the amount of logging;
+
+- av scanner update: 'FRISK F-Prot Antivirus' entry modified to recognize
+ name of a 'security risk' result, thanks to Michael Renner;
+
+- in a commented-out code providing a qmail CF/LF bug workaround, replaced
+ $smtp_handle->datasend by $smtp_data_fh->print, which is more efficient
+ in a line-by-line writing mode needed by qmail; thanks to Ronald Vazquez;
+
+- in a (banning) check for double extensions allow for whitespace around the
+ second filename extension (files amavisd.conf and amavisd.conf-sample);
+ based on a sample provided by Patrick T. Tsang;
+
+- setting $max_requests to 0 disables the limit, process will not be replaced
+ based on the number of requests it has completed (but may still be replaced
+ for other reasons); primarily intended for testing;
+
+- bump up a default value for $max_requests from 10 to 20 to match the
+ suggested/example value in amavisd.conf-sample;
+
+- AM.PDP/milter setup: new configuration setting $prepend_header_fields_hdridx,
+ also a member of policy banks, with a default value of 0. It is used as an
+ argument hdridx in an AM.PDP attribute 'insheader' which in a milter setup
+ is passed on as an argument hdridx to a smfi_insheader call. The value
+ of $prepend_header_fields_hdridx only affects AM.PDP protocol and only if
+ $append_header_fields_to_bottom is false (it is false by default). If more
+ than one milter is used, all milters should be inserting their header fields
+ at the same index (all prepending or appending, avoiding insertion in the
+ middle of a header), otherwise the resulting order of header fields in
+ a modified header becomes surprising, and in combination with signing
+ milters like DKIM or DK the signature verification will most likely fail.
+ The default value of 0 is normal and useful in combination with other
+ content-checking milters. Signing milters like dkim-milter and dk-milter
+ insert their header at index 1 (just below the new Received header fields),
+ and when amavisd-new with Petr Rehor's helper program amavis-milter
+ is used as a milter along with dkim-milter or dk-milter, the value of
+ $prepend_header_fields_hdridx MUST BE SET TO 1, otherwise the generated
+ signature will fail verification at the receiving site!
+
+ Discussion: when sendmail calls its milters, its Received header field
+ is not yet created and passed on to milters, yet it is already counted as
+ one header field for the purpose of smfi_insheader hdridx interpretation.
+ When a milter wants to prepend its header field(s), specifying hdridx of
+ 0 does prepend its header fields above the yet-to-be-inserted Received
+ header field as expected, and specifying 1 inserts its header field(s)
+ just below the yet-to-be-inserted Received header field. If some milters
+ in a chain specify 0 and others a 1 it affects the final order of inserted
+ header fields in unexpected ways. It would be natural to always prepend
+ fields with an index 0, but for signing milters like dk-milter this is
+ not acceptable, as it would be expected to include a not-yet-available
+ Received header field in its signature. For this reason signing milters
+ like dkim-milter and dk-milter insert their header fields (signature)
+ at index 1, and if amavisd-milter wants to coexist in such a setup,
+ it must also insert its header fields at index 1.
+
+ The conclusion: when amavisd (with its helper program) is used in a milter
+ setup along with other milters, it should use the same hdridx value as
+ other milters, which in case of signing dkim-milter and dk-milter is 1.
+ If there are no such milters, either a 1 or a 0 would do, although a value
+ of 0 produces a more natural order of header fields, matching that of
+ a post-queue content filtering setup. See threads:
+ http://archives.neohapsis.com/archives/postfix/2006-10/1777.html
+ http://archives.neohapsis.com/archives/postfix/2006-11/0467.html
+
+
---------------------------------------------------------------------------
September 30, 2006
amavisd-new-2.4.3 release notes
@@ -226,7 +1876,8 @@ OTHER CHANGES AND SMALL FEATURES:
- added global configuration variable $sql_quarantine_chunksize_max,
which determines a maximum size (in bytes) for data written to a field
'quarantine.mail_text' when quarantining to SQL. Must not exceed size
- allowed for a data type on a given SQL server. It also determines a
+ allowed for a data type on a given SQL server (e.g. maximum size
+ for data type 'blob' in MySQL is 65535 bytes). It also determines a
buffer size in amavisd. Too large a value may exceed process virtual
memory limits or just waste memory, too small a value splits large
mail into too many chunks, which may be less efficient to process;
@@ -261,7 +1912,13 @@ OTHER CHANGES AND SMALL FEATURES:
@additional_perl_modules = qw(
/usr/local/etc/mail/spamassassin/FuzzyOcr.pm
/usr/local/etc/mail/spamassassin/ImageInfo.pm
- String::Approx Plugin::TextCat BayesStore::SDBM
+ /usr/local/etc/mail/spamassassin/WebRedirect.pm
+ String::Approx Net::HTTP Net::HTTP::Methods
+ URI URI::http URI::_generic URI::_query URI::_server
+ HTTP::Date HTTP::Headers HTTP::Message HTML::HeadParser
+ HTTP::Request HTTP::Response HTTP::Status
+ LWP LWP::Protocol LWP::Protocol::http
+ LWP::UserAgent LWP::MemberMixin LWP::Debug
);
Make sure these files are owned by root and not writable by unprivileged
users such as amavis!
@@ -403,7 +2060,7 @@ OTHER CHANGES AND SMALL FEATURES:
- kill external tnef decoder if running for too long;
- abort Convert::UUlib::LoadFile or other Convert::UUlib processing in
- do_ascii() if running for too long; problem case reported by Martin Grimm;
+ do_ascii() if running for too long; problem case provided by Martin Grimm;
- add Net::Server hooks post_configure_hook() and post_bind_hook(), making
it easier to affect protection of Unix sockets created by Net::Server
@@ -628,6 +2285,8 @@ BUG FIXES AND CHANGES since 2.4.1:
item, misses extracting members from my test cases), so feel free to choose
between the two poor choices, I still prefer zoo(1), partly also because it
covers cases which clamd decoding misses;
+
+- kill external zoo or unzoo decoder if running for too long;
- internal: saving recipient addresses to SQL table maddr is now done
earlier to make information available to pen pals code;
@@ -675,17 +2334,18 @@ NEW FEATURES:
Pre-requisites:
* both the outgoing and the incoming mail must pass through amavisd
- (although outgoing mail may have spam checks disabled if desired);
+ (although outgoing mail may have checks disabled or made more permissive
+ if desired);
* SQL logging must be enabled (@storage_sql_dsn) and records should
- be kept for at least several days (some (holiday months) statistics:
- 85% of replied mail (or followups) is sent within 2 weeks since
- previous correspondence, 40% within 24 hours, 20% within 2 hours,
- 10% within 20 minutes);
+ be kept for at least several days (some statistics (2006-11 update):
+ 90% of replied mail (or followups) is sent within 2 weeks since
+ previous correspondence, 40% within 24 hours, 20% within 3 hours,
+ 10% within 30 minutes, 5% within 12 minutes);
* @mynetworks and @local_domains_maps must reflect reality;
* the information about client IP address must be available to amavisd,
i.e. Postfix XFORWARD protocol extension must be enabled, or AM.PDP+milter;
* configuration variable $penpals_bonus_score must be set to a positive
- value (such as 1.0, increase to 3 or 4 or 5 after seeing that it works),
+ value (such as 1.0, increase to perhaps 5 or 8 after seeing that it works),
zero disables the feature and is a default;
* $sql_clause{'sel_penpals'} must contain a SELECT clause (which by
default it does, unless overridden by an old assignment to %sql_clause
@@ -704,7 +2364,7 @@ NEW FEATURES:
How it works:
* SQL logging stores records about all mail messages processed by amavisd,
- their sender, recipients, delivery status, mail contents (no changes
+ their sender, recipients, delivery status, mail contents type (no changes
there, this feature was introduced with amavisd-new-2.3.0); for the
purpose of pen pals scheme only records with local-domain senders matter;
* when a message is received, a SQL lookup against a SQL logging database
@@ -785,7 +2445,7 @@ NEW FEATURES:
* there may be multiple MTA+amavisd servers, but all must use the same
logging SQL database;
* forwarding is compatible with the pen pals scheme;
- * broken forwarding like suggested for SPF, where sender address is replaced
+ * broken forwarding like SRS (with SPF), where sender address is replaced
by a forwarding mailbox address is counterproductive; for example: a local
user may also have an external mailbox at some remote provider with poor
spam protection; forwarding from the remote to a local mailbox is set up
@@ -796,7 +2456,7 @@ NEW FEATURES:
previous correspondence with his remote mailbox;
Testing:
- * set $penpals_bonus_score initially to a low value such as 1 or 0.5
+ * set $penpals_bonus_score initially to a low value such as 1
to avoid surprises;
* set $penpals_threshold_low and $penpals_threshold_high to undef
to perform pen pals lookups regardless of the score;
@@ -1378,14 +3038,16 @@ OTHER CHANGES:
* one may now add scoring rules to SA local.cf file, e.g.:
- header L_P0F_WXP X-Amavis-OS-Fingerprint =~ /^Windows XP/
- score L_P0F_WXP 3.5
+ header L_P0F_WXP X-Amavis-OS-Fingerprint =~ /^Windows XP(?![^(]*\b2000 SP)/
+ score L_P0F_WXP 2.5
header L_P0F_W X-Amavis-OS-Fingerprint =~ /^Windows(?! XP)/
- score L_P0F_W 1.7
+ score L_P0F_W 1.4
header L_P0F_UNKN X-Amavis-OS-Fingerprint =~ /^UNKNOWN/
score L_P0F_UNKN 0.8
- header L_P0F_Unix X-Amavis-OS-Fingerprint =~ /^((Free|Open|Net)BSD)|Solaris|HP-UX|Tru64/
+ header L_P0F_Unix X-Amavis-OS-Fingerprint =~ /^((Free|Open|Net)BSD|Solaris|HP-UX|Tru64|AIX)/
score L_P0F_Unix -1.0
+ header L_P0F_Linux X-Amavis-OS-Fingerprint =~ /^Linux/
+ score L_P0F_Linux -0.1
It is also possible to add score based on estimated IP distance, for
example to slightly favorize nearer hosts (this is probably good for Europe
@@ -2144,8 +3806,11 @@ is improved.
SECURITY:
-- require minimal version 1.05 of Convert::UUlib to avoid a known security
- problem in the underlying uulib (likely to be exploitable);
+- require minimal version 1.05 of Convert::UUlib to avoid
+ a security problem in the underlying uulib:
+ http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2005-1349
+ which is now known to be exploitable (2006-12-05), credits to
+ Jean-Sébastien Guay-Leroux;
INCOMPATIBILITY with 2.2.1 and older versions:
@@ -2589,7 +4254,7 @@ OTHER CHANGES:
a warning in pre-3.1 versions of SA;
- documentation note: Macintosh.tar.gz installation instructions
- for Macintosh are not recent, they apply to OSX 10.2.0-10.3.9
+ for Macintosh are not recent, they apply to Mac OS X 10.2.0-10.3.9
---------------------------------------------------------------------------
@@ -4103,6 +5768,10 @@ OTHER EXTERNALLY VISIBLE CHANGES
Disabled by default, see variable $auth_supported_out (later renamed
to $auth_required_out).
+- when passing envelope sender address to SpamAssassin, supply it as a
+ rfc2822-standard header field Return-Path, and no longer as X-Envelope-From
+ (the change came with a pre-release amavisd-new-20040301).
+
- provisional/experimental support for DSPAM spam checker (pre 3.0):
if configuration variable $dspam is nonempty and represents a path to
a 'dspam' program, a message is passed to dspam and its inserted headers
diff --git a/TODO b/TODO
--- a/TODO
+++ b/TODO
@@ -1,37 +1,15 @@ SMTP, LMTP, mail address handling
SMTP, LMTP, mail address handling
- correctly RFC2822-quote addresses in From/To/Cc in DSN;
-- avoid un-quoting and re-quoting addresses: keep them in the original form
- besides the decoded form;
-- resolve Net::SMTP deficiencies:
- * it should support pipelining;
- * enqueue SMTP responses and make them available in order or arrival,
- (needed for pipelining);
- * it should not report error if an optional parameter (e.g. SIZE)
- is available but MTA does not support it - should be silently ignored;
- * needs documented API to query a list of services offered by a server
- (EHLO response);
- * missing parameter - passing AUTH 'submitter' in MAIL FROM;
- * missing parameter - passing ORCPT in RCPT TO (needed for DSN - RFC 3461);
- * fix broken error report in IO::Socket when non-blocking mode is used
- (when timeout is nonzero): instead of 'Invalid argument' it should tell
- that the server is not reachable;
- * make use of IO::Socket::INET6 to provide IPv6 support;
- * investigate possibility of enhancing it to support LMTP;
- be able to do multiple-transaction sessions on the
outgoing side (SMTP client);
- split_localpart(): check 'owner-special' handling, e.g. foo-request-spam@
- separate the application from SMTP protocol handling
- (re-investigate Net-Server-Mail and other Perl module attempts)
-- provide LMTP client code (Net-LMTP is unsuitable);
- fallback relays, MX backups?
- one_response_for_all: report (propagate) all MTA SMTP responses if different
MAIL MODIFICATIONS, EXPAND, NOTIFICATIONS:
- make possible to use proper MIME content structure in notifications;
- optionally attach (chopped?) mail body to DSN?
-- implement defanging on a per-recipient basis;
-- more versatile mail defanging: be able to rebuild mail from desired
- parts while throwing away or modifying undesired mail parts;
- use modified headers (and body?) as provided by SA ?
- macro expander: do not replace formal arguments %<n> which are within
quoted replacement text (in the regexp macro as well);
@@ -52,7 +30,7 @@ VIRUS AND OTHER MALWARE SCANNING:
- split calls to virus scanner into multiple calls for long lists of files
to be scanned, in oder not to exceed the command line / arguments size limit;
- configurable SAVI-Perl;
-- clamscan (and others) need option '--mbox' when given full mail file,
+- some virus scanners need option '--mbox' when given full mail file,
but not on already decoded parts (Michael Boelen);
- some scanners need proper file name extension to be able to recognize
and decode a file correctly;
@@ -68,7 +46,6 @@ MAIL DECODING/DE-ARCHIVING:
- file(1) is unable to differentiate or recognize various types
of pgp/gpg mail (signed/encrypted/armored/signature/key);
- per-recipient bypass_decoding;
-- support 7-Zip archives;
- yEnc encoding www.yenc.org (NNTP); Appledouble encoding, Macintosh StuffIt;
- store_mgr: stop_expensive_decoding_at=n ?
- seek some solution to prevent decoder from attempting to create files
@@ -77,12 +54,10 @@ MAIL DECODING/DE-ARCHIVING:
errors when decoding;
INTERNALS, CODING, ...
-- save am_id to conn object or to msginfo?
- amavis-milter.c: be able to approve locally originating DSN without
calling amavisd to avoid deadlock (or the need to force '-odd');
- use timers in a manner to provide some resiliency to clock jumps;
- use multi-timers Perl module? make timer independent of its use by SA;
-- syslog-ng problem (SA bug report 3625, syslog-ng may fork during posting);
- can we avoid keeping two copies of header (MIME::Entity and orig_headers)?
QUARANTINE
@@ -93,6 +68,7 @@ QUARANTINE
* to do the correct intersect between per-recipient quarantine_to
and per-recipient kill level and other blockings;
- disable quarantine (and virus admin notifications) based on virus name;
+- (optionally) disable quarantine for spam lovers implicitly/automatically;
- strip original X-Spam-* headers when releasing a quarantined message;
- update msgrcpt.rs field after a quarantine release
- support quarantining by MTA (milter, HOLD)
@@ -123,21 +99,13 @@ DOCUMENTATION
- cleaner web page;
SOME OF THE MORE SELF-CONTAINED PROJECTS
-- more versatile mail defanging (see above);
- provide a true SNMP agent (see TODO-SNMP-AGENT);
- more sophisticated tools for plotting and analyzing collected
statistics (such as SNMP counters and timing breakdown), and for
providing early warnings in case of problems (including SNMP alerts);
-- providing missing features and/or taking over the maintainership
- of Perl modules Net::SMTP (e.g. proper pipelining);
- Net::Server IPv6 support;
- write a Perl module for interfacing with libarchive
(http://people.freebsd.org/~kientzle/libarchive/), which is an
ambitious (currently primarily FreeBSD) programming library that can
create and read several streaming archive formats, including most
popular tar variants and the POSIX cpio format.
-- a volunteer activist to act as a contact/development person towards
- SpamAssassin project, with a goal to split the SpamAssassin API into
- two independent phases: analyzing the message (once per message),
- and providing appropriately evaluated scores and mail header modifications
- based on recipient preferences (once per recipient);
diff --git a/amavisd b/amavisd
--- a/amavisd
+++ b/amavisd
@@ -10,7 +10,8 @@
# on amavisd-snapshot-20020300).
#
# All work since amavisd-snapshot-20020300:
-# Copyright (C) 2002,2003,2004,2005,2006 Mark Martinec, All Rights Reserved.
+# Copyright (C) 2002,2003,2004,2005,2006,2007 Mark Martinec,
+# All Rights Reserved.
# with contributions from the amavis-* mailing lists and individuals,
# as acknowledged in the release notes.
#
@@ -82,6 +83,7 @@
# Amavis::Unpackers::MIME
# Amavis::Notify
# Amavis::Cache
+# Amavis::Custom
# Amavis
#optionally compiled-in packages: ---------------------------------------------
# Amavis::DB::SNMP
@@ -95,6 +97,7 @@
# Amavis::In::AMCL
# Amavis::In::SMTP
#( Amavis::In::Courier )
+# Amavis::Out::SMTP::Protocol
# Amavis::Out::SMTP
# Amavis::Out::Pipe
# Amavis::Out::BSMTP
@@ -125,12 +128,23 @@ use re 'taint';
#
sub fetch_modules($$@) {
my($reason, $required, @modules) = @_;
+ my($have_sawampersand) = Devel::SawAmpersand->UNIVERSAL::can("sawampersand");
+ my($amp) = $have_sawampersand && Devel::SawAmpersand::sawampersand() ? 1 : 0;
+ warn "fetch_modules: PL_sawampersand flag was already turned on" if $amp;
my(@missing);
for my $m (@modules) {
local($_) = $m;
- $_ .= /^auto::/ ? '.al' : '.pm' if !m{^/} && !m{\.(pm|pl|al)\z};
+ $_ .= /^auto::/ ? '.al' : '.pm' if !m{^/} && !m{\.(pm|pl|al|ix)\z};
s{::}{/}g;
- eval { require $_ } or push(@missing, $m);
+ eval { require $_ }
+ or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ push(@missing,$m);
+ # printf STDERR ("fetch_modules: error loading module %s :\n%s\n", $_,
+ # join("\n", map {"> $_"} split(/\n/,$eval_stat)));
+ };
+ if ($have_sawampersand && !$amp && Devel::SawAmpersand::sawampersand())
+ { $amp = 1; warn "Loading of module $m turned on PL_sawampersand flag" }
}
die "ERROR: MISSING $reason:\n" . join('', map { " $_\n" } @missing)
if $required && @missing;
@@ -142,12 +156,11 @@ BEGIN {
Exporter POSIX Fcntl Socket Errno Carp Time::HiRes
IO::Handle IO::File IO::Socket IO::Socket::UNIX IO::Socket::INET
IO::Wrap IO::Stringy Digest::MD5 Unix::Syslog File::Basename
- Mail::Field Mail::Address Mail::Header Mail::Internet Compress::Zlib
- MIME::Base64 MIME::QuotedPrint MIME::Words
+ Compress::Zlib MIME::Base64 MIME::QuotedPrint MIME::Words
MIME::Head MIME::Body MIME::Entity MIME::Parser MIME::Decoder
MIME::Decoder::Base64 MIME::Decoder::Binary MIME::Decoder::QuotedPrint
MIME::Decoder::NBit MIME::Decoder::UU MIME::Decoder::Gzip64
- Net::Cmd Net::SMTP Net::Server Net::Server::PreForkSimple
+ Net::Server Net::Server::PreFork
));
# with earlier versions of Perl one may need to add additional modules
# to the list, such as: auto::POSIX::setgid auto::POSIX::setuid ...
@@ -170,31 +183,41 @@ sub D_DISCARD() { 0 }
sub D_DISCARD() { 0 }
sub D_PASS () { 1 }
-# major contents_category constants, in order of importance
+# major contents_category constants, in increasing order of importance
sub CC_CATCHALL() { 0 }
sub CC_CLEAN () { 1 } # tag_level = "CC_CLEAN,1"
-sub CC_TEMPFAIL () { 2 }
+sub CC_MTA () { 2 } # trouble passing mail back to MTA
sub CC_OVERSIZED() { 3 }
sub CC_BADH () { 4 }
-sub CC_SPAMMY() { 5 } # tag2_level (and: "CC_SPAMMY,1" = tag3_level)
+sub CC_SPAMMY() { 5 } # tag2_level (and: tag3_level = "CC_SPAMMY,1")
sub CC_SPAM () { 6 } # kill_level
sub CC_UNCHECKED() { 7 }
sub CC_BANNED() { 8 }
sub CC_VIRUS () { 9 }
+#
+*CC_TEMPFAIL = \&CC_MTA; # alias - old name, cc 2 was repurposed/generalized)
+#
+# in other words: major_ccat minor_ccat %subject_tag_maps_by_ccat
+## if score >= kill level => CC_SPAM 0
+## elsif score >= tag3 level => CC_SPAMMY 1 @spam_subject_tag3_maps
+## elsif score >= tag2 level => CC_SPAMMY 0 @spam_subject_tag2_maps
+## elsif score >= tag level => CC_CLEAN 1 @spam_subject_tag_maps
+## else => CC_CLEAN 0
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
%EXPORT_TAGS = (
- 'dynamic_confvars' => # per - policy bank settings
+ 'dynamic_confvars' => # per- policy bank settings
[qw(
$policy_bank_name $protocol @inet_acl
$myhostname $syslog_ident $syslog_facility $syslog_priority
$log_level $log_templ $log_recip_templ
- $forward_method $notify_method $resend_method $release_method
- $os_fingerprint_method @smtpd_discard_ehlo_keywords
+ $forward_method $notify_method $resend_method
+ $release_method $requeue_method
+ $os_fingerprint_method $originating @smtpd_discard_ehlo_keywords
$propagate_dsn_if_possible $terminate_dsn_on_notify_success
$amavis_auth_user $amavis_auth_pass $auth_reauthenticate_forwarded
$auth_required_out $auth_required_inp $auth_required_release
@@ -207,14 +230,17 @@ BEGIN {
$undecipherable_subject_tag $localpart_is_case_sensitive
$recipient_delimiter $replace_existing_extension
$hdr_encoding $bdy_encoding $hdr_encoding_qb
- $insert_received_line $append_header_fields_to_bottom
+ $allow_disclaimers $insert_received_line
+ $append_header_fields_to_bottom $prepend_header_fields_hdridx
$allow_fixing_improper_header $allow_fixing_improper_header_folding
- %allowed_added_header_fields
+ %allowed_added_header_fields %allowed_header_tests
$X_HEADER_TAG $X_HEADER_LINE $notify_xmailer_header
$remove_existing_x_scanned_headers $remove_existing_spam_headers
%sql_clause %local_delivery_aliases $banned_namepath_re
$per_recip_whitelist_sender_lookup_tables
$per_recip_blacklist_sender_lookup_tables
+ @anomy_sanitizer_args @altermime_args_defang
+ @altermime_args_disclaimer @disclaimer_options_bysender_maps
@local_domains_maps @mynetworks_maps
@newvirus_admin_maps @banned_filename_maps
@@ -228,29 +254,32 @@ BEGIN {
@message_size_limit_maps @debug_sender_maps
@bypass_virus_checks_maps @bypass_spam_checks_maps
@bypass_banned_checks_maps @bypass_header_checks_maps
- @viruses_that_fake_sender_maps
+ @viruses_that_fake_sender_maps @virus_name_to_spam_score_maps
%final_destiny_by_ccat %lovers_maps_by_ccat
- %defang_by_ccat %subject_tag_maps_by_ccat
+ %defang_maps_by_ccat %subject_tag_maps_by_ccat
%quarantine_method_by_ccat %quarantine_to_maps_by_ccat
%notify_admin_templ_by_ccat %notify_recips_templ_by_ccat
%notify_sender_templ_by_ccat %warnsender_by_ccat
%hdrfrom_notify_admin_by_ccat %mailfrom_notify_admin_by_ccat
%hdrfrom_notify_recip_by_ccat %mailfrom_notify_recip_by_ccat
%hdrfrom_notify_sender_by_ccat
- %admin_maps_by_ccat %dsn_bcc_by_ccat
- %warnrecip_maps_by_ccat %addr_extension_maps_by_ccat
+ %admin_maps_by_ccat %warnrecip_maps_by_ccat
+ %always_bcc_by_ccat %dsn_bcc_by_ccat
+ %addr_extension_maps_by_ccat %addr_rewrite_maps_by_ccat
)],
'confvars' => # global settings (not per-policy, not per-recipient)
[qw(
$myproduct_name $myversion_id $myversion_id_numeric $myversion_date
- $myversion @additional_perl_modules
+ $myversion $instance_name @additional_perl_modules
$MYHOME $TEMPBASE $QUARANTINEDIR $quarantine_subdir_levels
$daemonize $courierfilter_shutdown $pid_file $lock_file $db_home
$enable_db $enable_global_cache
$daemon_user $daemon_group $daemon_chroot_dir $path
- $DEBUG $DO_SYSLOG $LOGFILE
- $max_servers $max_requests $child_timeout $smtpd_timeout
+ $DEBUG $DO_SYSLOG $LOGFILE $nanny_details_level
+ $max_servers $max_requests
+ $min_servers $min_spare_servers $max_spare_servers
+ $child_timeout $smtpd_timeout
%current_policy_bank %policy_bank %interface_policy
$unix_socketname $inet_socket_port $inet_socket_bind
$relayhost_is_client $smtpd_recipient_limit
@@ -268,17 +297,17 @@ BEGIN {
@keep_decoded_original_maps @map_full_type_to_short_type_maps
%banned_rules
$penpals_threshold_low $penpals_threshold_high
- $file
+ $file $altermime $enable_anomy_sanitizer
)],
'sa' => # global SpamAssassin settings
[qw(
- $helpers_home $dspam
+ $helpers_home $dspam $sa_spawned
$sa_local_tests_only $sa_auto_whitelist $sa_timeout $sa_debug
)],
'platform' => [qw(
- $can_truncate $unicode_aware $eol
+ $can_truncate $unicode_aware $eol $my_pid
&D_REJECT &D_BOUNCE &D_DISCARD &D_PASS
- &CC_CATCHALL &CC_CLEAN &CC_TEMPFAIL &CC_OVERSIZED
+ &CC_CATCHALL &CC_CLEAN &CC_MTA &CC_OVERSIZED
&CC_BADH &CC_SPAMMY &CC_SPAM &CC_UNCHECKED &CC_BANNED &CC_VIRUS
%ccat_display_names
)],
@@ -296,7 +325,7 @@ BEGIN {
$final_banned_destiny $final_bad_header_destiny
@virus_lovers_maps @spam_lovers_maps
@banned_files_lovers_maps @bad_header_lovers_maps
- $dsn_bcc
+ $always_bcc $dsn_bcc
$mailfrom_notify_sender $mailfrom_notify_recip
$mailfrom_notify_admin $mailfrom_notify_spamadmin
$hdrfrom_notify_sender $hdrfrom_notify_recip
@@ -412,9 +441,10 @@ sub ca($) {
# essential initializations, right at the program start time, may run as root!
#
+use vars qw($read_config_files_depth @actual_config_files);
BEGIN { # init_primary: version, $unicode_aware, base policy bank
$myproduct_name = 'amavisd-new';
- $myversion_id = '2.4.3'; $myversion_date = '20060930';
+ $myversion_id = '2.5.2'; $myversion_date = '20070627';
$myversion = "$myproduct_name-$myversion_id ($myversion_date)";
$myversion_id_numeric = # x.yyyzzz, allows numerical compare, like Perl $]
@@ -424,8 +454,9 @@ BEGIN { # init_primary: version, $unico
$eol = "\n"; # native record separator in files: LF or CRLF or even CR
$unicode_aware =
$] >= 5.008 && length("\x{263a}")==1 && eval { require Encode };
-
- # initialize policy bank hash containing dynamic config settings
+ $read_config_files_depth = 0;
+ eval { require Devel::SawAmpersand }; # load if avail, don't bother otherwise
+ # initialize policy bank hash to contain dynamic config settings
for my $tag (@EXPORT_TAGS{'dynamic_confvars', 'legacy_dynamic_confvars'}) {
for my $v (@$tag) {
local($1,$2);
@@ -461,7 +492,7 @@ BEGIN {
# Net::Server pre-forking settings - defaults, overruled by amavisd.conf
$max_servers = 2; # number of pre-forked children
- $max_requests = 10; # retire a child after that many accepts
+ $max_requests = 20; # retire a child after that many accepts, 0=unlimited
# timeout for our processing:
$child_timeout = 8*60; # abort child if it does not complete a task in n sec
@@ -498,7 +529,8 @@ BEGIN {
$SYSLOG_LEVEL = 'mail.debug';
$enable_db = 0; # load optional modules Amavis::DB & Amavis::DB::SNMP
- $enable_global_cache = 0; # enable use of bdb-based Amavis::Cache
+ $enable_global_cache = 0; # enable use of bdb-based Amavis::Cache
+ $nanny_details_level = 1; # register_proc verbosity: 0, 1, 2
# Where to find SQL server(s) and database to support SQL lookups?
# A list of triples: (dsn,user,passw). Specify more than one
@@ -556,12 +588,14 @@ BEGIN {
@inet_acl = qw( 127.0.0.1 ::1 ); # allow SMTP access only from localhost
@mynetworks = qw( 127.0.0.0/8 ::1 FE80::/10 FEC0::/10
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 );
-
+ $originating = 0; # a boolean, initially reflects client_addr_mynets,
+ # but may be modified later through a policy bank
$notify_method = 'smtp:[127.0.0.1]:10025';
$forward_method = 'smtp:[127.0.0.1]:10025';
$resend_method = undef; # overrides $forward_method on defanging if nonempty
$release_method = undef; # overrides $notify_method on releasing
# from quarantine if nonempty
+ $requeue_method = 'smtp:[127.0.0.1]:25'; # requeueing release from a quarant.
$virus_quarantine_method = 'local:virus-%m';
$spam_quarantine_method = 'local:spam-%m.gz';
$banned_files_quarantine_method = 'local:banned-%m';
@@ -571,6 +605,8 @@ BEGIN {
$insert_received_line = 1; # insert Received: header field? (not with milter)
$append_header_fields_to_bottom = 0;
+ $prepend_header_fields_hdridx = 0; # normally 0, use 1 for co-existence
+ # with signing DK and DKIM milters
$remove_existing_x_scanned_headers = 0;
$remove_existing_spam_headers = 1;
@@ -651,6 +687,8 @@ BEGIN {
$QUARANTINEDIR = undef; # no quarantine unless overridden by config
$undecipherable_subject_tag = '***UNCHECKED*** ';
+
+ $sa_spawned = 0; # true: run SA in a subprocess; false: call SA directly
# string to prepend to Subject header field when message qualifies as spam
# $sa_spam_subject_tag1 = undef; # example: '***possible SPAM*** '
@@ -665,6 +703,11 @@ BEGIN {
$sa_timeout = 30;# timeout low boundary in seconds for a call to SpamAssassin
$file = 'file'; # path to the file(1) utility for classifying contents
+ $altermime = 'altermime'; # path to the altermime utility (optional)
+ @altermime_args_defang = qw(--verbose --removeall);
+ @altermime_args_disclaimer = qw(--disclaimer=/etc/altermime-disclaimer.txt);
+ # @altermime_args_disclaimer = qw(--disclaimer=/etc/_OPTION_);
+ # @disclaimer_options_bysender_maps = ( 'altermime-disclaimer.txt' );
$MIN_EXPANSION_FACTOR = 5; # times original mail size
$MAX_EXPANSION_FACTOR = 500; # times original mail size
@@ -775,8 +818,7 @@ BEGIN {
#
BEGIN {
$allowed_added_header_fields{lc($_)} = 1 for qw(
- Received X-Quarantine-ID X-Virus-Scanned
- X-Amavis-Alert X-Amavis-Hold X-Amavis-Modified
+ Received X-Quarantine-ID X-Amavis-Alert X-Amavis-Hold X-Amavis-Modified
X-Amavis-PenPals X-Amavis-OS-Fingerprint X-Amavis-PolicyBank
X-Spam-Status X-Spam-Level X-Spam-Flag X-Spam-Score
X-Spam-Report X-Spam-Checker-Version X-Spam-Tests
@@ -784,8 +826,13 @@ BEGIN {
X-DSPAM-Confidence X-DSPAM-Probability X-DSPAM-User X-DSPAM-Factors
);
$allowed_added_header_fields{lc('X-Spam-Checker-Version')} = 0;
- # $allowed_added_header_fields{lc(c('X_HEADER_TAG'))} = 1;
+ # $allowed_added_header_fields{lc(c('X_HEADER_TAG'))} = 1; #later:read_config
# $allowed_added_header_fields{lc('Received')} = 0 if !$insert_received_line;
+
+ # controls which header tests are performed in check_header_validity,
+ # keys correspond to minor contents categories for CC_BADH
+ $allowed_header_tests{lc($_)} = 1 for qw(
+ other mime 8bit control empty long syntax missing multiple);
# provide names for content categories - to be used only for logging,
# SNMP counter names and display purposes
@@ -793,7 +840,9 @@ BEGIN {
CC_CATCHALL, 'CatchAll', # last resort, should not normally appear
CC_CLEAN, 'Clean',
CC_CLEAN.',1', 'CleanTag', # tag_level
- CC_TEMPFAIL, 'TempFail',
+ CC_MTA, 'MTA-Trouble', # unable to forward (general)
+ CC_MTA.',1', 'MTA-TempFailed', # MTA response was 4xx
+ CC_MTA.',2', 'MTA-Rejected', # MTA response was 5xx
CC_OVERSIZED, 'Oversized',
CC_BADH, 'BadHdr',
CC_BADH.',1', 'BadHdrMime',
@@ -810,32 +859,36 @@ BEGIN {
CC_VIRUS, 'Virus',
);
- # The SQL select clause to fetch per-recipient policy settings
+ # The SQL select clause to fetch per-recipient policy settings.
# The %k will be replaced by a comma-separated list of query addresses
- # (e.g. full address, domain only, catchall). Use ORDER, if there
- # is a chance that multiple records will match - the first match wins
- # If field names are not unique (e.g. 'id'), the later field overwrites the
- # earlier in a hash returned by lookup, which is why we use '*,users.id'.
- # This is a separate legacy variable for upwards compatibility, now only
- # referenced by the program through %sql_clause entry 'sel_policy'.
+ # for a recipient (e.g. full address, domain only, catchall), %a will be
+ # replaced by an exact recipient address (same as the first entry in %k,
+ # suitable for pattern matching). Use ORDER, if there is a chance that
+ # multiple records will match - the first match wins (i.e. the first
+ # returned record). If field names are not unique (e.g. 'id'), the later
+ # field overwrites the earlier in a hash returned by lookup, which is
+ # why we use '*, users.id'. This is a separate legacy variable for
+ # upwards compatibility, now only referenced by the program through
+ # a %sql_clause entry 'sel_policy'.
$sql_select_policy =
'SELECT *, users.id'.
' FROM users LEFT JOIN policy ON users.policy_id=policy.id'.
' WHERE users.email IN (%k) ORDER BY users.priority DESC';
- # The SQL select clause to check sender in per-recipient whitelist/blacklist
+ # The SQL select clause to check sender in per-recipient whitelist/blacklist.
# The first SELECT argument '?' will be users.id from recipient SQL lookup,
- # the %k will be sender addresses (e.g. full address, domain only, catchall).
+ # the %k will be sender addresses (e.g. full address, domain only, catchall),
+ # the %a will be an exact sender address (same as the first entry in %k).
# Only the first occurrence of '?' will be replaced by users.id, subsequent
# occurrences of '?' will see empty string as an argument. There can be zero
- # or more occurrences of %k, lookup keys will be multiplied accordingly.
+ # or more occurrences of %k or %a, lookup keys will be replicated accordingly
# Up until version 2.2.0 the '?' had to be placed before the '%k';
# starting with 2.2.1 this restriction is lifted.
# This is a separate legacy variable for upwards compatibility, now only
# referenced by the program through %sql_clause entry 'sel_wblist'.
$sql_select_white_black_list =
- 'SELECT wb FROM wblist LEFT JOIN mailaddr ON wblist.sid=mailaddr.id'.
- ' WHERE (wblist.rid=?) AND (mailaddr.email IN (%k))'.
+ 'SELECT wb FROM wblist JOIN mailaddr ON wblist.sid=mailaddr.id'.
+ ' WHERE wblist.rid=? AND mailaddr.email IN (%k)'.
' ORDER BY mailaddr.priority DESC';
%sql_clause = (
@@ -850,26 +903,27 @@ BEGIN {
' policy, client_addr, size, host) VALUES (?,?,?,?,?,?,?,?,?,?)',
'upd_msg' =>
'UPDATE msgs SET content=?, quar_type=?, quar_loc=?, dsn_sent=?,'.
- ' spam_level=?, message_id=?, from_addr=?, subject=? WHERE mail_id=?',
- # 'ins_rcp' =>
- # 'INSERT INTO msgrcpt (mail_id, rid, time_num,'.
- # ' ds, rs, bl, wl, bspam_level, smtp_resp) VALUES (?,?,?,?,?,?,?,?,?)',
+ ' spam_level=?, message_id=?, from_addr=?, subject=?'. # ,p0f=?
+ ' WHERE mail_id=?',
'ins_rcp' =>
'INSERT INTO msgrcpt (mail_id, rid,'.
' ds, rs, bl, wl, bspam_level, smtp_resp) VALUES (?,?,?,?,?,?,?,?)',
- # 'ins_quar' =>
- # 'INSERT INTO quarantine (mail_id, chunk_ind, time_num, mail_text)'.
- # ' VALUES (?,?,?,?)',
'ins_quar' =>
'INSERT INTO quarantine (mail_id, chunk_ind, mail_text)'.
' VALUES (?,?,?)',
'sel_quar' =>
'SELECT mail_text FROM quarantine WHERE mail_id=? ORDER BY chunk_ind',
- 'sel_penpals' =>
+ 'sel_penpals' => # no message-id references list
"SELECT msgs.time_num, msgs.mail_id, subject".
" FROM msgs JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id".
- " WHERE sid=? AND rid=? AND ds='P' AND content!='V'".
- " ORDER BY time_num DESC LIMIT 1",
+ " WHERE sid=? AND rid=? AND content!='V' AND ds='P'".
+ " ORDER BY msgs.time_num DESC LIMIT 1",
+ 'sel_penpals_msgid' => # with a nonempty message-id references list
+ "SELECT msgs.time_num, msgs.mail_id, subject, message_id, rid".
+ " FROM msgs JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id".
+ " WHERE sid=? AND content!='V' AND ds='P' AND message_id IN (%m)".
+ " AND rid!=sid".
+ " ORDER BY rid=? DESC, msgs.time_num DESC LIMIT 1",
);
# NOTE on $sql_clause{'upd_msg'}: MySQL clobbers timestamp on update
# (unless DEFAULT 0 is used) setting it to current local time and
@@ -903,6 +957,7 @@ BEGIN {
[qr/^ISO-8859.*\btext\b/ => 'txt'],
[qr/^Non-ISO.*ASCII\b.*\btext\b/ => 'txt'],
[qr/^Unicode\b.*\btext\b/i => 'txt'],
+ [qr/^UTF.* Unicode text\b/i => 'txt'],
[qr/^'diff' output text\b/ => 'txt'],
[qr/^GNU message catalog\b/ => 'mo'],
[qr/^PGP encrypted data\b/ => 'pgp'],
@@ -927,6 +982,7 @@ BEGIN {
[qr/^MPEG\b/ =>['movie','mpg'] ],
[qr/^Microsoft ASF\b/ =>['movie','wmv'] ],
[qr/^RIFF\b.*\bAVI\b/ =>['movie','avi'] ],
+ [qr/^RIFF\b.*\banimated cursor\b/ =>['movie','ani'] ],
[qr/^RIFF\b.*\bWAVE audio\b/ =>['audio','wav'] ],
[qr/^Macromedia Flash data\b/ => 'swf'],
@@ -937,6 +993,7 @@ BEGIN {
[qr/^PDF document\b/ => 'pdf'],
[qr/^Rich Text Format data\b/ => 'rtf'],
[qr/^Microsoft Office Document\b/i => 'doc'], # OLE2: doc, ppt, xls, ...
+ [qr/^Microsoft Installer\b/i => 'doc'], # file(1) may misclassify
[qr/^ms-windows meta(file|font)\b/i => 'wmf'],
[qr/^LaTeX\b.*\bdocument text\b/ => 'lat'],
[qr/^TeX DVI file\b/ => 'dvi'],
@@ -951,6 +1008,7 @@ BEGIN {
[qr/^lzop compressed\b/ => 'lzo'],
[qr/^compress'd/ => 'Z'],
[qr/^Zip archive\b/i => 'zip'],
+ [qr/^7-zip archive\b/i => '7z'],
[qr/^RAR archive\b/i => 'rar'],
[qr/^LHa.*\barchive\b/i => 'lha'], # (also known as .lzh)
[qr/^ARC archive\b/i => 'arc'],
@@ -974,9 +1032,11 @@ BEGIN {
[qr/^MS Windows\b.*\bDLL\b/ => ['exe','dll'] ],
[qr/\bexecutable for MS Windows\b.*\bDLL\b/ => ['exe','dll'] ],
+ [qr/^MS-DOS executable \(built-in\)/ => 'asc'], # starts with LZ
[qr/^(MS-)?DOS executable\b.*\bDLL\b/ => ['exe','dll'] ],
[qr/^MS Windows\b.*\bexecutable\b/ => ['exe','exe-ms'] ],
[qr/\bexecutable for MS Windows\b/ => ['exe','exe-ms'] ],
+ [qr/^COM executable for DOS\b/ => 'asc'], # misclassified?
[qr/^(MS-)?DOS executable\b(?!.*\(COM\))/ => ['exe','exe-ms'] ],
[qr/^PA-RISC.*\bexecutable\b/ => ['exe','exe-unix'] ],
[qr/^ELF .*\bexecutable\b/ => ['exe','exe-unix'] ],
@@ -996,6 +1056,7 @@ BEGIN {
# MS-DOS executable PE for MS Windows (DLL) (GUI) Intel 80386 32-bit
# MS-DOS executable PE for MS Windows (DLL) (GUI) Alpha 32-bit
# MS-DOS executable, NE for MS Windows 3.x (driver)
+ # MS-DOS executable (built-in) (any file starting with LZ!)
# PE executable for MS Windows (DLL) (GUI) Intel 80386 32-bit
# PE executable for MS Windows (GUI) Intel 80386 32-bit
# NE executable for MS Windows 3.x
@@ -1042,10 +1103,11 @@ BEGIN {
['cpio', \&Amavis::Unpackers::do_pax_cpio, \$cpio],
['tar', \&Amavis::Unpackers::do_pax_cpio, \$pax],
['tar', \&Amavis::Unpackers::do_pax_cpio, \$cpio],
- ['tar', \&Amavis::Unpackers::do_tar],
+# ['tar', \&Amavis::Unpackers::do_tar], # no longer supported
['deb', \&Amavis::Unpackers::do_ar, \$ar],
# ['a', \&Amavis::Unpackers::do_ar, \$ar], #unpacking .a seems an overkill
['zip', \&Amavis::Unpackers::do_unzip],
+ ['7z', \&Amavis::Unpackers::do_7zip, ['7zr','7za','7z'] ],
['rar', \&Amavis::Unpackers::do_unrar, \$unrar],
['arj', \&Amavis::Unpackers::do_unarj, \$unarj],
['arc', \&Amavis::Unpackers::do_arc, \$arc],
@@ -1139,17 +1201,16 @@ BEGIN {
CC_SPAM, sub { ca('spam_lovers_maps') },
CC_BADH, sub { ca('bad_header_lovers_maps') },
);
- %defang_by_ccat = (
- CC_VIRUS, sub { c('defang_virus') || c('defang_all') },
- CC_BANNED, sub { c('defang_banned') || c('defang_all') },
- CC_UNCHECKED, sub { c('defang_undecipherable') || c('defang_all') },
- CC_SPAM, sub { c('defang_spam') || c('defang_all') },
- CC_SPAMMY, sub { c('defang_spam') || c('defang_all') },
+ %defang_maps_by_ccat = (
+ CC_VIRUS, sub { c('defang_virus') },
+ CC_BANNED, sub { c('defang_banned') },
+ CC_UNCHECKED, sub { c('defang_undecipherable') },
+ CC_SPAM, sub { c('defang_spam') },
+ CC_SPAMMY, sub { c('defang_spam') },
# CC_BADH.',3', 1, # NUL or CR character in header
# CC_BADH.',5', 1, # header line longer than 998 characters
# CC_BADH.',6', 1, # header field syntax error
- CC_BADH, sub { c('defang_bad_header') || c('defang_all') },
- CC_CATCHALL, sub { c('defang_all') },
+ CC_BADH, sub { c('defang_bad_header') },
);
%subject_tag_maps_by_ccat = (
CC_VIRUS, [ '***INFECTED*** ' ],
@@ -1164,7 +1225,7 @@ BEGIN {
CC_VIRUS, sub { c('virus_quarantine_method') },
CC_BANNED, sub { c('banned_files_quarantine_method') },
CC_SPAM, sub { c('spam_quarantine_method') },
- CC_SPAMMY, sub { c('clean_quarantine_method') }, # formally is clean
+ CC_SPAMMY, sub { c('clean_quarantine_method') }, #formally a clean msg
CC_BADH, sub { c('bad_header_quarantine_method') },
CC_CLEAN, sub { c('clean_quarantine_method') },
);
@@ -1181,6 +1242,9 @@ BEGIN {
CC_BANNED, sub { ca('banned_admin_maps') },
CC_SPAM, sub { ca('spam_admin_maps') },
CC_BADH, sub { ca('bad_header_admin_maps') },
+ );
+ %always_bcc_by_ccat = (
+ CC_CATCHALL, sub { c('always_bcc') },
);
%dsn_bcc_by_ccat = (
CC_CATCHALL, sub { c('dsn_bcc') },
@@ -1207,7 +1271,7 @@ BEGIN {
CC_CATCHALL, sub { cr('notify_virus_admin_templ') },
);
%notify_recips_templ_by_ccat = (
- CC_SPAM, sub { cr('notify_spam_recips_templ') },
+ CC_SPAM, sub { cr('notify_spam_recips_templ') }, #usualy empty
CC_CATCHALL, sub { cr('notify_virus_recips_templ') },
);
%notify_sender_templ_by_ccat = ( # bounce templates
@@ -1236,6 +1300,7 @@ BEGIN {
CC_BADH, sub { ca('addr_extension_bad_header_maps') },
# CC_OVERSIZED, 'oversized';
);
+ %addr_rewrite_maps_by_ccat = ( );
} # end BEGIN - init_tertiary
@@ -1246,9 +1311,10 @@ sub Amavis::Unpackers::do_uncompress($$$
sub Amavis::Unpackers::do_uncompress($$$);
sub Amavis::Unpackers::do_gunzip($$);
sub Amavis::Unpackers::do_pax_cpio($$$);
-sub Amavis::Unpackers::do_tar($$);
+#sub Amavis::Unpackers::do_tar($$); # no longer supported
sub Amavis::Unpackers::do_ar($$$);
sub Amavis::Unpackers::do_unzip($$;$$);
+sub Amavis::Unpackers::do_7zip($$$;$);
sub Amavis::Unpackers::do_unrar($$$;$);
sub Amavis::Unpackers::do_unarj($$$;$);
sub Amavis::Unpackers::do_arc($$$);
@@ -1264,8 +1330,8 @@ no warnings 'once';
no warnings 'once';
# Define alias names or shortcuts in this module to make it simpler
# to call these routines from amavisd.conf
+*read_l10n_templates = \&Amavis::Util::read_l10n_templates;
*read_text = \&Amavis::Util::read_text;
-*read_l10n_templates = \&Amavis::Util::read_l10n_templates;
*read_hash = \&Amavis::Util::read_hash;
*read_array = \&Amavis::Util::read_array;
*dump_hash = \&Amavis::Util::dump_hash;
@@ -1278,10 +1344,11 @@ no warnings 'once';
*do_uncompress = \&Amavis::Unpackers::do_uncompress;
*do_gunzip = \&Amavis::Unpackers::do_gunzip;
*do_pax_cpio = \&Amavis::Unpackers::do_pax_cpio;
-*do_tar = \&Amavis::Unpackers::do_tar;
+*do_tar = \&Amavis::Unpackers::do_tar; # no longer supported
*do_ar = \&Amavis::Unpackers::do_ar;
*do_unzip = \&Amavis::Unpackers::do_unzip;
*do_unrar = \&Amavis::Unpackers::do_unrar;
+*do_7zip = \&Amavis::Unpackers::do_7zip;
*do_unarj = \&Amavis::Unpackers::do_unarj;
*do_arc = \&Amavis::Unpackers::do_arc;
*do_zoo = \&Amavis::Unpackers::do_zoo;
@@ -1293,6 +1360,17 @@ no warnings 'once';
*do_unstuff = \&Amavis::Unpackers::do_unstuff;
*do_executable = \&Amavis::Unpackers::do_executable;
sub new_RE { Amavis::Lookup::RE->new(@_) }
+*defang_by_ccat = \%defang_maps_by_ccat; # compatibility with old name
+use vars qw(%defang_by_ccat);
+
+ at virus_name_to_spam_score_maps =
+ (new_RE( [ qr'^(Email|HTML)\.(Phishing|Spam|Scam[a-z0-9]?)\.'i => 0.1 ],
+ [ qr'^(Email|Html)\.Malware\.Sanesecurity\.' => undef ],
+ [ qr'^(Email|Html)(\.[^., ]*)*\.Sanesecurity\.' => 0.1 ],
+ # [ qr'^(Email|Html)\.(Hdr|Img|ImgO|Bou|Stk|Loan|Cred|Job|Dipl|Doc)
+ # (\.[^., ]*)* \.Sanesecurity\.'x => 0.1 ],
+ [ qr'^(MSRBL-Images/|MSRBL-SPAM\.)' => 0.1 ],
+ ));
# prepend a lookup table label object for logging purposes
sub label_default_maps() {
@@ -1308,9 +1386,9 @@ sub label_default_maps() {
@virus_quarantine_to_maps @banned_quarantine_to_maps
@spam_quarantine_to_maps @spam_quarantine_bysender_to_maps
@bad_header_quarantine_to_maps @clean_quarantine_to_maps
- @archive_quarantine_to_maps
+ @archive_quarantine_to_maps @banned_filename_maps
@keep_decoded_original_maps @map_full_type_to_short_type_maps
- @banned_filename_maps @viruses_that_fake_sender_maps
+ @viruses_that_fake_sender_maps @virus_name_to_spam_score_maps
@spam_tag_level_maps @spam_tag2_level_maps @spam_tag3_level_maps
@spam_kill_level_maps @spam_modifies_subj_maps
@spam_dsn_cutoff_level_maps @spam_dsn_cutoff_level_bysender_maps
@@ -1331,24 +1409,43 @@ sub label_default_maps() {
}
}
-# read and evaluate configuration files (one or more)
-sub read_config(@) {
- my(@config_files) = @_;
- for my $config_file (@config_files) {
+# return a list of actually read&evaluated configuration files
+sub get_config_files_read() { @actual_config_files }
+
+# read and evaluate a configuration file, some sanity checking and housekeeping
+sub read_config_file($$) {
+ my($config_file,$is_optional) = @_;
+ my($errn) = stat($config_file) ? 0 : 0+$!; # symlinks-friendly
+ if ($errn == ENOENT && $is_optional) {
+ # don't complain if missing
+ } else {
my($msg);
- my($errn) = stat($config_file) ? 0 : 0+$!; # symlinks-friendly
- if ($errn == ENOENT) { $msg = "does not exist" }
- elsif ($errn) { $msg = "is inaccessible: $!" }
- elsif (-d _) { $msg = "is a directory" }
- elsif (!-f _) { $msg = "is not a regular file" }
- elsif ($> && -o _) { $msg = "should not be owned by EUID $>"}
- elsif ($> && -w _) { $msg = "is writable by EUID $>, EGID $)" }
- if (defined $msg) { die "Config file \"$config_file\" $msg," }
+ if ($errn == ENOENT) { $msg = "does not exist" }
+ elsif ($errn) { $msg = "is inaccessible: $!" }
+ elsif (-d _) { $msg = "is a directory" }
+ elsif (!-f _) { $msg = "is not a regular file" }
+ elsif ($> && -o _) { $msg = "should not be owned by EUID $>"}
+ elsif ($> && -w _) { $msg = "is writable by EUID $>, EGID $)" }
+ if (defined $msg) { die "Config file \"$config_file\" $msg," }
+ $read_config_files_depth++; push(@actual_config_files, $config_file);
+ if ($read_config_files_depth >= 100) {
+ print STDERR "read_config_files: recursion depth limit exceeded\n";
+ exit 1; # avoid unwinding deep recursion, abort right away
+ }
$! = 0;
if (defined(do $config_file)) {}
elsif ($@ ne '') { die "Error in config file \"$config_file\": $@" }
elsif ($! != 0) { die "Error reading config file \"$config_file\": $!" }
- }
+ $read_config_files_depth-- if $read_config_files_depth > 0;
+ }
+ 1;
+}
+
+sub include_config_files(@) { read_config_file($_,0) for @_; 1 }
+sub include_optional_config_files(@) { read_config_file($_,1) for @_; 1 }
+
+# supply remaining defaults after config files have already been read/evaluated
+sub supply_after_defaults() {
$daemon_chroot_dir = ''
if !defined $daemon_chroot_dir || $daemon_chroot_dir eq '/';
# provide some sensible defaults for essential settings (post-defaults)
@@ -1363,10 +1460,13 @@ sub read_config(@) {
$syslog_priority = $2 if $syslog_priority eq '';
}
$X_HEADER_LINE= "$myproduct_name at $mydomain" if !defined $X_HEADER_LINE;
- if (!defined($X_HEADER_TAG)) { $X_HEADER_TAG = 'X-Virus-Scanned' }
- else { # implicitly add to %allowed_added_header_fields for compatibility
+ $X_HEADER_TAG = 'X-Virus-Scanned' if !defined $X_HEADER_TAG;
+ if ($X_HEADER_TAG =~ /^[!-9;-\176]+\z/) {
+ # implicitly add to %allowed_added_header_fields for compatibility,
+ # unless the hash entry already exists
my($allowed_hdrs) = cr('allowed_added_header_fields');
- $allowed_hdrs->{lc($X_HEADER_TAG)} = 1 if $allowed_hdrs;
+ $allowed_hdrs->{lc($X_HEADER_TAG)} = 1
+ if $allowed_hdrs && !exists($allowed_hdrs->{lc($X_HEADER_TAG)});
}
$gunzip = "$gzip -d" if !defined $gunzip && $gzip ne '';
$bunzip2 = "$bzip2 -d" if !defined $bunzip2 && $bzip2 ne '';
@@ -1408,6 +1508,7 @@ sub read_config(@) {
# to a single 'banned names/types' lookup table
%banned_rules = ('DEFAULT'=>$banned_filename_re); # backwards compatibile
}
+ 1;
}
1;
@@ -1420,7 +1521,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT = qw(&lock &unlock);
}
@@ -1449,9 +1550,10 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
- @EXPORT_OK = qw(&init &write_log &open_log &close_log);
+ @EXPORT_OK = qw(&init &collect_log_stats &log_to_stderr &log_fd
+ &write_log &open_log &close_log);
}
use subs @EXPORT_OK;
@@ -1461,7 +1563,7 @@ use File::Basename;
use File::Basename;
BEGIN {
- import Amavis::Conf qw(:platform c cr ca $myversion $daemon_user);
+ import Amavis::Conf qw(:platform $DEBUG c cr ca $myversion $daemon_user);
import Amavis::Lock;
}
@@ -1470,26 +1572,39 @@ use vars qw($log_to_stderr $do_syslog $l
use vars qw($log_to_stderr $do_syslog $logfile);
use vars qw($current_syslog_ident $current_syslog_facility);
use vars qw(%syslog_prio_name_to_num); # maps syslog priority names to numbers
-
-sub init($$$) {
- ($log_to_stderr, $do_syslog, $logfile) = @_;
+use vars qw($log_lines $log_warnings %log_status_counts);
+
+sub init($$) {
+ ($do_syslog, $logfile) = @_;
+ $log_lines = 0; $log_warnings = 0; undef %log_status_counts;
# initialize mapping of syslog priority names to numbers
for my $pn qw(DEBUG INFO NOTICE WARNING ERR CRIT ALERT EMERG) {
my($prio) = eval("LOG_$pn");
$syslog_prio_name_to_num{$pn} = $prio =~ /^\d+\z/ ? $prio : LOG_WARNING;
}
+ $myname = $0;
open_log();
if (!$do_syslog && $logfile eq '')
{ print STDERR "Logging to STDERR (no \$LOGFILE and no \$DO_SYSLOG)\n" }
- $myname = $0;
- my($msg) = "starting. $myname at " . c('myhostname') . " $myversion";
- $msg .= ", eol=\"$eol\"" if $eol ne "\n";
- $msg .= ", Unicode aware" if $unicode_aware;
- $msg .= ", LC_ALL=$ENV{LC_ALL}" if $ENV{LC_ALL} ne '';
- $msg .= ", LC_TYPE=$ENV{LC_TYPE}" if $ENV{LC_TYPE} ne '';
- $msg .= ", LC_CTYPE=$ENV{LC_CTYPE}" if $ENV{LC_CTYPE} ne '';
- $msg .= ", LANG=$ENV{LANG}" if $ENV{LANG} ne '';
- write_log(0, undef, $msg);
+}
+
+sub collect_log_stats() {
+ my(@result) = ($log_lines, $log_warnings, {%log_status_counts});
+ $log_lines = 0; $log_warnings = 0; undef %log_status_counts;
+ @result;
+}
+
+# turn debug logging to STDERR on or off
+sub log_to_stderr(;$) {
+ $log_to_stderr = shift if @_ > 0;
+ $log_to_stderr;
+}
+
+# try to obtain file descriptor used by write_log, undef if unknown
+sub log_fd() {
+ $log_to_stderr ? fileno(STDERR)
+ : $do_syslog ? undef # how to obtain fd on syslog?
+ : defined $loghandle ? $loghandle->fileno : fileno(STDERR);
}
sub open_log() {
@@ -1505,7 +1620,7 @@ sub open_log() {
$current_syslog_ident = $id; $current_syslog_facility = $fac;
} elsif ($logfile ne '') {
$loghandle = IO::File->new;
- $loghandle->open($logfile, O_APPEND|O_WRONLY, 0640)
+ $loghandle->open($logfile, O_CREAT|O_APPEND|O_WRONLY, 0640)
or die "Failed to open log file $logfile: $!";
binmode($loghandle,":bytes") or die "Can't cancel :utf8 mode: $!"
if $unicode_aware;
@@ -1524,10 +1639,10 @@ sub close_log() {
sub close_log() {
if ($do_syslog) {
closelog();
- $current_syslog_ident = $current_syslog_facility = undef;
+ undef $current_syslog_ident; undef $current_syslog_facility;
} elsif (defined($loghandle) && $logfile ne '') {
$loghandle->close or die "Error closing log file $logfile: $!";
- $loghandle = undef;
+ undef $loghandle;
}
}
@@ -1552,23 +1667,27 @@ sub write_log($$$;@) {
elsif ($level >= -1) { $prio = LOG_WARNING if $prio > LOG_WARNING }
elsif ($level >= -2) { $prio = LOG_ERR if $prio > LOG_ERR }
else { $prio = LOG_CRIT if $prio > LOG_CRIT }
+ if (c('syslog_ident') ne $current_syslog_ident ||
+ c('syslog_facility') ne $current_syslog_facility) {
+ close_log() if !defined($current_syslog_ident) &&
+ !defined($current_syslog_facility);
+ open_log();
+ }
my($pre) = $alert_mark;
my($logline_size) = 980; # less than (1023 - prefix)
while (length($am_id)+length($pre)+length($errmsg) > $logline_size) {
my($avail) = $logline_size - length($am_id . $pre . "...");
+ $log_lines++; $! = 0;
syslog($prio, "%s", $am_id . $pre . substr($errmsg,0,$avail) . "...");
+ if ($! != 0) { $log_warnings++; $log_status_counts{"$!"}++ }
$pre = $alert_mark . "..."; $errmsg = substr($errmsg, $avail);
}
- if (c('syslog_ident') ne $current_syslog_ident ||
- c('syslog_facility') ne $current_syslog_facility) {
- close_log() if !defined($current_syslog_ident) &&
- !defined($current_syslog_facility);
- open_log();
- }
- syslog($prio, "%s", $am_id . $pre . $errmsg);
+ $log_lines++; $! = 0; syslog($prio, "%s", $am_id . $pre . $errmsg);
+ if ($! != 0) { $log_warnings++; $log_status_counts{"$!"}++ }
} else {
my($prefix) = sprintf("%s %s %s[%s]: ", # prepare syslog-like prefix
strftime("%b %e %H:%M:%S",localtime), c('myhostname'), $myname, $$);
+ $log_lines++;
if (defined $loghandle && !$log_to_stderr) {
lock($loghandle);
seek($loghandle,0,2) or die "Can't position log file to its tail: $!";
@@ -1593,7 +1712,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&init §ion_time &report &get_time_so_far);
}
@@ -1678,29 +1797,31 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&untaint &min &max &unique
&safe_encode &safe_decode &q_encode
- &xtext_encode &xtext_decode
+ &xtext_encode &xtext_decode &orcpt_encode
&snmp_count &snmp_counters_init &snmp_counters_get
&am_id &new_am_id &ll &do_log &debug_oneshot
&add_entropy &fetch_entropy &generate_mail_id
&exit_status_str &proc_status_ok &kill_proc &prolong_timer
&waiting_for_client &switch_to_my_time &switch_to_client_time
- &sanitize_str &fmt_struct
- &ccat_maj &ccat_min &cmp_ccat &cmp_ccat_maj
+ &sanitize_str &fmt_struct &freeze &thaw
+ &ccat_split &ccat_maj &cmp_ccat &cmp_ccat_maj
&setting_by_given_contents_category_all
&setting_by_given_contents_category
&rmdir_recursively &read_text &read_l10n_templates
- &read_hash &read_array &dump_hash &dump_array
- &cloexec &run_command &run_command_consumer
+ &read_hash &read_array &dump_hash &dump_array &cloexec
+ &run_command &run_command_consumer &run_as_subprocess
+ &collect_results &collect_results_structured
&dynamic_destination);
}
use subs @EXPORT_OK;
use POSIX qw(WIFEXITED WIFSIGNALED WIFSTOPPED
WEXITSTATUS WTERMSIG WSTOPSIG);
-use Errno qw(ENOENT EACCES);
+use Errno qw(ENOENT EACCES EAGAIN ESRCH);
+use Time::HiRes ();
use Digest::MD5 2.22; # need 'clone' method
# use Fcntl qw(F_GETFD F_SETFD FD_CLOEXEC); # used in cloexec, if enabled
# use Encode; # Perl 5.8 UTF-8 support
@@ -1708,7 +1829,7 @@ BEGIN {
BEGIN {
import Amavis::Conf qw(:platform $DEBUG c cr ca $child_timeout $smtpd_timeout
$trim_trailing_space_in_lookup_result_fields);
- import Amavis::Log qw(write_log open_log close_log);
+ import Amavis::Log qw(write_log open_log close_log log_fd);
import Amavis::Timing qw(section_time);
}
@@ -1778,9 +1899,8 @@ sub q_encode($$$) {
$octets =~ /^ ( [\001-\011\013\014\016-\177]* [ \t] )? (.*?)
( [ \t] [\001-\011\013\014\016-\177]* )? \z/sx;
my($head,$rest,$tail) = ($1,$2,$3);
- # Q-encode $rest according to RFC 2047
- # more restricted than =?_ so that it may be used in 'phrase'
- $rest =~ s{([^ 0-9a-zA-Z!*/+-])}{sprintf('=%02X',ord($1))}egs;
+ # Q-encode $rest according to RFC 2047 (not for use in comments or phrase)
+ $rest =~ s{([\000-\037\177\200-\377=?_])}{sprintf('=%02X',ord($1))}egs;
$rest =~ tr/ /_/; # turn spaces into _ (rfc2047 allows it)
my($s) = $head; my($len) = 75 - (length($prefix)+length($suffix)) - 2;
while ($rest ne '') {
@@ -1805,11 +1925,24 @@ sub xtext_decode($) {
$str;
}
-# Set or get Amavis internal message id.
-# This message id performs a similar function as queue-id in MTA responses.
+# xtext_encode and prepend 'rfc822;' to form a string to be used as ORCPT
+sub orcpt_encode($) { # rfc3461
+ # rfc3461: Due to limitations in the Delivery Status Notification format,
+ # the value of the original recipient address prior to encoding as "xtext"
+ # MUST consist entirely of printable (graphic and white space) characters
+ # from the US-ASCII [4] repertoire.
+ my($str) = @_; local($1); # argument should be SMTP-quoted address
+ $str = $1 if $str =~ /^<(.*)>\z/s; # strip-off <>
+ $str =~ s/[^\040-\176]/?/gs;
+ 'rfc822;' . xtext_encode($str);
+}
+
+# Set or get Amavis internal task id (also called: message id).
+# This task id performs a similar function as queue-id in MTA responses.
# It may only be used in generating text part of SMTP responses,
# or in generating log entries. It is only unique within a limited timespan.
-use vars qw($amavis_task_id); # internal message id (accessible via &am_id)
+use vars qw($amavis_task_id); # internal task id
+# (accessible via am_id() and later also as $msginfo->log_id)
sub am_id(;$) {
if (@_) { # set, if argument present
@@ -1846,6 +1979,15 @@ sub fetch_entropy() {
# authorize releasing quarantined mail. Both the mail_id and secret are
# 12-char strings of characters [A-Za-z0-9+-], with an additional restriction
# for mail_id which must begin and end with an alphanumeric character.
+# As the number of encoded bits is an integral multiple of 24, no base64
+# trailing padding characters '=' are needed for the time being (rfc4648).
+# Note the difference in base64-like encodings:
+# amavisd almost-base64: 62 +, 63 -
+# rfc4648 base64: 62 +, 63 /
+# rfc4648 base64url: 62 -, 63 _
+# Generally, rfc2822 controls, SP and specials must be avoided: ()<>[]:;@\,."
+# Some day we may want to switch from almost-base64 to base64url to avoid
+# having to quote a '+' in regular expressions and URL.
sub generate_mail_id() {
my($secret_id,$id,$rest);
for (my $j=0; $j<100; $j++) { # provide some sanity loop limit just in case
@@ -1862,12 +2004,13 @@ sub generate_mail_id() {
$entropy = undef;
add_entropy($rest); # carry over unused portion of old entropy accumulator
add_entropy($id); # mix-in the full mail_id before chopping it to 12 chars
- $id = substr($id,0,12); $id =~ tr{/}{-};
+ $id = substr($id,0,12); $id =~ tr{/}{-}; # base64 -> almost-base64
($id,$secret_id);
}
use vars qw(@counter_names);
-# elements may be counter names (increment is 1), or pairs: [name,increment]
+# elements may be counter names (increment is 1), or pairs: [name,increment],
+# or triples: [name,value,type], where type can be: C32, C64, INT or OID
sub snmp_counters_init() { @counter_names = () }
sub snmp_count(@) { push(@counter_names, @_) }
sub snmp_counters_get() { \@counter_names }
@@ -1902,6 +2045,7 @@ sub do_log($$;@) {
$level = 0 if ($DEBUG || $debug_oneshot) && $level > 0;
write_log($level, am_id(), shift, @_);
}
+ 1;
}
# map process termination status number to a string, and append optional
@@ -1933,11 +2077,12 @@ sub proc_status_ok($$@) {
}
# kill a process, typically a runaway external decoder or checker
-sub kill_proc($;$$$) {
- my($pid,$what,$timeout,$proc_fh) = @_;
- $pid > 0 or die "shouldn't be killing process groups: [$pid]";
- $what = defined $what ? " running $what" : '';
- do_log(-1,"killing process [%s]%s", $pid,$what);
+sub kill_proc($;$$$$) {
+ my($pid,$what,$timeout,$proc_fh,$reason) = @_;
+ $pid >= 0 or die "Shouldn't be killing process groups: [$pid]";
+ $pid != 1 or die "Shouldn't be killing process 'init': [$pid]";
+ $what = defined $what ? " running $what" : '';
+ $reason = defined $reason ? " (reason: $reason)" : '';
#
# the following sequence is a must: SIGTERM first, _then_ close a pipe;
# otherwise the following can happen: closing a pipe first (explicitly or
@@ -1946,18 +2091,29 @@ sub kill_proc($;$$$) {
# not closing the pipe after SIGTERM does not necessarily let the process
# notice SIGTERM, so SIGKILL is always needed to stop it, which is not nice
#
- kill('TERM',$pid) # be gentle on the first attempt
- or do_log(1,"Can't send SIGTERM to process [%s]: %s", $pid,$!);
+ my($n) = kill(0,$pid); # does the process really exist?
+ if ($n == 0 && $! != ESRCH) {
+ die sprintf("Can't send SIG 0 to process [%s]%s: %s", $pid,$what,$!);
+ } elsif ($n == 0) {
+ do_log(2, "no need to kill process [%s]%s, already gone", $pid,$what);
+ } else {
+ do_log(-1,"killing process [%s]%s%s", $pid,$what,$reason);
+ kill('TERM',$pid) or $! == ESRCH # be gentle on the first attempt
+ or die sprintf("Can't send SIGTERM to process [%s]%s: %s",$pid,$what,$!);
+ }
# close the pipe if still open, ignoring status
$proc_fh->close if defined $proc_fh;
- my($n) = kill(0,$pid); # is the process still there? get number of processes
+ $n = kill(0,$pid); # is the process still there?
if ($n > 0 && defined($timeout) && $timeout > 0) {
sleep($timeout); $n = kill(0,$pid); # wait a little and recheck
}
- if ($n > 0) { # the process is still there, try a stronger signal
- do_log(-1,"process [%s] is still alive, using a bigger hammer", $pid);
- kill('KILL',$pid)
- or do_log(-2,"Can't send SIGKILL to process [%s]: %s", $pid,$!);
+ if ($n == 0 && $! != ESRCH) {
+ die sprintf("Can't send SIG 0 to process [%s]%s: %s", $pid,$what,$!);
+ } elsif ($n > 0) { # the process is still there, try a stronger signal
+ do_log(-1,"process [%s]%s is still alive, using a bigger hammer",
+ $pid,$what);
+ kill('KILL',$pid) or $! == ESRCH
+ or die sprintf("Can't send SIGKILL to process [%s]%s: %s",$pid,$what,$!);
}
}
@@ -1965,7 +2121,7 @@ sub prolong_timer($;$) {
my($which_section, $child_remaining_time) = @_;
if (defined $child_remaining_time) { # explicitly given
$child_remaining_time = 10 if $child_remaining_time < 10;
- do_log(5, "prolong_timer %s: timer set to = %d s",
+ do_log(5, "prolong_timer %s: timer set to %d s",
$which_section,$child_remaining_time);
} else {
$child_remaining_time = alarm(0); # check how much time is left
@@ -2026,10 +2182,66 @@ sub fmt_struct($) {
ref($arg) eq 'ARRAY' ? '['.join(',',map {fmt_struct($_)} @$arg).']' : $arg;
};
-
-sub ccat_maj($$) { my($a_maj,$a_min) = split(/,/, shift, -1); $a_maj }
-
-sub ccat_min($$) { my($a_maj,$a_min) = split(/,/, shift, -1); $a_min }
+# used by freeze: protect % and ~, as well as NUL and \200 for good measure
+sub st_encode($) {
+ my($str) = @_; local($1);
+ $str =~ s/([%~\000\200])/sprintf("%%%02X",ord($1))/egs;
+ $str;
+}
+
+# simple Storable::freeze lookalike
+sub freeze($); # prototype
+sub freeze($) {
+ my($obj) = @_; my($ty) = ref($obj);
+ if (!defined($obj)) { 'U' }
+ elsif (!$ty) { join('~', '', st_encode($obj)) } # string
+ elsif ($ty eq 'SCALAR') { join('~', 'S', st_encode(freeze($$obj))) }
+ elsif ($ty eq 'REF') { join('~', 'R', st_encode(freeze($$obj))) }
+ elsif ($ty eq 'ARRAY') { join('~', 'A', map {st_encode(freeze($_))} @$obj) }
+ elsif ($ty eq 'HASH') {
+ join('~', 'H',
+ map {(st_encode($_),st_encode(freeze($obj->{$_})))} sort keys %$obj)
+ } else { die "Can't freeze object type $ty" }
+}
+
+# simple Storable::thaw lookalike
+sub thaw($); # prototype
+sub thaw($) {
+ my($str) = @_;
+ return undef if !defined $str;
+ my($ty, at val) = split(/~/,$str,-1);
+ for (@val) { s/%([0-9a-fA-F]{2})/pack("C",hex($1))/eg }
+ if ($ty eq 'U') { undef }
+ elsif ($ty eq '') { $val[0] }
+ elsif ($ty eq 'S') { my($obj)=thaw($val[0]); \$obj }
+ elsif ($ty eq 'R') { my($obj)=thaw($val[0]); \$obj }
+ elsif ($ty eq 'A') { [map {thaw($_)} @val] }
+ elsif ($ty eq 'H') {
+ my($hr) = {};
+ while (@val) { my($k) = shift @val; $hr->{$k} = thaw(shift @val) }
+ $hr;
+ } else { die "Can't thaw object type $ty" }
+}
+
+# accepts either a single contents category (a string: "maj,min" or "maj"),
+# or a list of contents categories, in which case only the first element
+# is considered; returns a passed pair: (major_ccat, minor_ccat)
+sub ccat_split($) {
+ my($ccat) = @_; my($major,$minor);
+ $ccat = $ccat->[0] if ref $ccat; # pick the first element if a list
+ ($major,$minor) = split(/,/,$ccat,-1) if defined $ccat;
+ !wantarray ? $major : ($major,$minor);
+}
+
+# accepts either a single contents category (a string: "maj,min" or "maj"),
+# or a list of contents categories, in which case only the first element
+# is considered; returns major_ccat
+sub ccat_maj($) {
+ my($ccat) = @_; my($major,$minor);
+ $ccat = $ccat->[0] if ref $ccat; # pick the first element if a list
+ ($major,$minor) = split(/,/,$ccat,-1) if defined $ccat;
+ $major;
+}
# compare numerically two strings of the form "maj,min" or just "maj", where
# maj and min are numbers, representing major and minor contents categery
@@ -2051,38 +2263,39 @@ sub cmp_ccat_maj($$) {
# relevant contents categories for which a query is made, it MUST already be
# sorted in descending order; this is a classical subroutine, not a method!
sub setting_by_given_contents_category_all($@) {
- my($settings_href, at ccat) = @_; my(@r);
- if (!defined($settings_href)) {
- # no settings, returns an empty list or undef
- } elsif (ref($settings_href) ne 'HASH') {
- die "setting_by_given_cc_all: unexpected data type: ".ref($settings_href);
- } else {
- for my $e (@ccat, CC_CATCHALL) {
- if (exists $settings_href->{$e}) {
- my($s) = $settings_href->{$e};
- $s = &$s if ref($s) eq 'CODE'; # support lazy evaluation
- push(@r, [$s,$e]); # a pair: [setting, corresponding ccat]
- }
- }
- }
- @r; # a list of pairs
+ my($ccat, at settings_href_list) = @_; my(@r);
+ if (!@settings_href_list) {} # no settings provided
+ else {
+ for my $e ((!defined($ccat) ?() :ref($ccat) ?@$ccat :$ccat), CC_CATCHALL) {
+ if (grep { defined($_) && exists($_->{$e}) } @settings_href_list) {
+ # supports lazy evaluation (setting may be a subroutine)
+ my(@slist) = map { !defined($_) || !exists($_->{$e}) ? undef :
+ do {my($s)=$_->{$e}; ref($s) eq 'CODE' ? &$s : $s}
+ } @settings_href_list;
+ push(@r, [$e, at slist]); # a tuple: [corresponding ccat, settings list]
+ }
+ }
+ }
+ @r; # a list of tuples
}
# similar to setting_by_given_contents_category_all(), but only the first
-# (the most relevant) setting is returned
+# (the most relevant) setting is returned, without a corresponding ccat
sub setting_by_given_contents_category($@) {
- my($settings_href, at ccat) = @_; my($s);
- if (!defined($settings_href)) {
- # no settings, returns an empty list or undef
- } elsif (ref($settings_href) ne 'HASH') {
- die "setting_by_given_cc: unexpected data type: ".ref($settings_href);
- } else {
- for my $e (@ccat, CC_CATCHALL) {
- if (exists $settings_href->{$e}) { $s = $settings_href->{$e}; last }
- }
- }
- $s = &$s if ref($s) eq 'CODE'; # support lazy evaluation
- $s; # only the first entry
+ my($ccat, at settings_href_list) = @_; my(@slist);
+ if (!@settings_href_list) {} # no settings provided
+ else {
+ for my $e ((!defined($ccat) ?() :ref($ccat) ?@$ccat :$ccat), CC_CATCHALL) {
+ if (grep { defined($_) && exists($_->{$e}) } @settings_href_list) {
+ # supports lazy evaluation (setting may be a subroutine)
+ @slist = map { !defined($_) || !exists($_->{$e}) ? undef :
+ do {my($s)=$_->{$e}; ref($s) eq 'CODE' ? &$s : $s}
+ } @settings_href_list;
+ last;
+ }
+ }
+ }
+ !wantarray ? $slist[0] : @slist; # only the first entry
}
# Removes a directory, along with its contents
@@ -2303,10 +2516,18 @@ sub cloexec($;$$) { undef }
# }
# POSIX::open a file or dup an existing fd (Perl open syntax), with a
-# requirement that it gets opened on a prescribed file descriptor $fd_target
+# requirement that it gets opened on a prescribed file descriptor $fd_target;
+# this subroutine is usually called from a forked process prior to exec
sub open_on_specific_fd($$$$) {
my($fd_target,$fname,$flags,$mode) = @_;
my($fd_got); # fd directy given as argument, or obtained from POSIX::open
+ my($logging_safe) = 0;
+ if (ll(5)) {
+ # crude attempt to prevent a forked process from writing log records
+ # to its parent process on STDOUT or STDERR
+ my($log_fd) = log_fd();
+ $logging_safe = 1 if !defined($log_fd) || $log_fd > 2;
+ }
local($1);
if ($fname =~ /^&=?(\d+)\z/) { $fd_got = $1 } # fd directly specified
my($flags_displayed) = $flags == &POSIX::O_RDONLY ? '<'
@@ -2315,7 +2536,7 @@ sub open_on_specific_fd($$$$) {
# close whatever is on a target descriptor but don't shoot self in the foot
# with Net::Server <= 0.90 fd0 was main::stdin, but no longer is in 0.91
do_log(5, "open_on_specific_fd: target fd%s closing, to become %s %s",
- $fd_target,$flags_displayed,$fname);
+ $fd_target,$flags_displayed,$fname) if $logging_safe;
# it pays off to close explicitly, with some luck open will get a target fd
POSIX::close($fd_target); # ignore error, we may have just closed a log
}
@@ -2327,21 +2548,33 @@ sub open_on_specific_fd($$$$) {
if ($fd_got != $fd_target) { # dup, ensuring we get a specified descriptor
eval { # we may have been left without a log file descriptor, must not die
do_log(5, "open_on_specific_fd: target fd%s dup2 from fd%s %s %s",
- $fd_target,$fd_got,$flags_displayed,$fname);
+ $fd_target,$fd_got,$flags_displayed,$fname) if $logging_safe;
};
# POSIX mandates we got the lowest fd available (but some kernels have
# bugs), let's be explicit that we require a specified file descriptor
defined POSIX::dup2($fd_got,$fd_target)
- or "Can't dup2 from $fd_got to $fd_target: $!";
+ or die "Can't dup2 from $fd_got to $fd_target: $!";
if ($fd_got > 2) { # let's get rid of the original fd, unless 0,1,2
my($err); defined POSIX::close($fd_got) or $err = $!;
$err = defined $err ? ": $err" : '';
eval { # we may have been left without a log file descriptor, don't die
- do_log(5, "open_on_specific_fd: source fd%s closed%s", $fd_got,$err);
+ do_log(5, "open_on_specific_fd: source fd%s closed%s",
+ $fd_got,$err) if $logging_safe;
};
}
}
$fd_got;
+}
+
+sub release_parent_resources() {
+ $Amavis::sql_dataset_conn_lookups->dbh_inactive(1)
+ if $Amavis::sql_dataset_conn_lookups;
+ $Amavis::sql_dataset_conn_storage->dbh_inactive(1)
+ if $Amavis::sql_dataset_conn_storage;
+# undef $Amavis::sql_dataset_conn_lookups;
+# undef $Amavis::sql_dataset_conn_storage;
+# undef $Amavis::body_digest_cache; undef $Amavis::snmp_db;
+# undef $Amavis::db_env;
}
# Run specified command as a subprocess (like qx operator, but more careful
@@ -2353,17 +2586,26 @@ sub run_command($$@) {
my($stdin_from, $stderr_to, $cmd, @args) = @_;
my($cmd_text) = join(' ', $cmd, @args);
$stdin_from = '/dev/null' if $stdin_from eq '';
- $stderr_to = '/dev/null' if $stderr_to eq '';
+ $stderr_to = '/dev/null' if defined($stderr_to) && $stderr_to eq '';
my($msg) = join(' ', $cmd, @args, "<$stdin_from",
$stderr_to eq '' ? () : "2>$stderr_to");
# $^F == 2 or do_log(-1,"run_command: SYSTEM_FD_MAX not 2: %d", $^F);
my($pid); my($proc_fh) = IO::File->new;
- eval { $pid = $proc_fh->open('-|') }; # fork, catching errors
- if ($@ ne '') { chomp($@); die "run_command (open pipe): $@ ($!)" }
+ eval {
+ $pid = $proc_fh->open('-|'); 1; # fork, catching errors
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die "run_command (open pipe): $eval_stat";
+ };
defined($pid) or die "run_command: can't fork: $!";
- if (!$pid) { # child
- eval { # must not use die in forked process, or we end up with
- # two running daemons!
+ if (!$pid) { # child
+ alarm(0); my($interrupt) = '';
+ my($h1) = sub { $interrupt = $_[0] };
+ my($h2) = sub { die "Received signal ".$_[0] };
+ @SIG{qw(INT HUP TERM TSTP QUIT USR1 USR2)} = ($h1) x 7;
+ eval { # die must be caught, otherwise we end up with two running daemons
+ local(@SIG{qw(INT HUP TERM TSTP QUIT USR1 USR2)}) = ($h2) x 7;
+ if ($interrupt ne '') { my($i) = $interrupt; $interrupt = ''; die $i }
# use Devel::Symdump ();
# my($dumpobj) = Devel::Symdump->rnew;
# for my $k ($dumpobj->ios) {
@@ -2374,10 +2616,7 @@ sub run_command($$@) {
# close(*{$k}{IO}) and do_log(2, "DID CLOSE %s (fileno=%s)", $k,$fn);
# }
# }
-# $sql_dataset_conn_lookups->dbh_inactive(1) if $sql_dataset_conn_lookups;
-# $sql_dataset_conn_storage->dbh_inactive(1) if $sql_dataset_conn_storage;
-# $sql_dataset_conn_lookups = $sql_dataset_conn_storage = undef;
-
+ release_parent_resources();
open_on_specific_fd(0,$stdin_from,&POSIX::O_RDONLY,0);
open_on_specific_fd(2,$stderr_to,&POSIX::O_WRONLY,0) if $stderr_to ne '';
# eval { close_log() }; # may have been closed by open_on_specific_fd
@@ -2385,13 +2624,16 @@ sub run_command($$@) {
exec {$cmd} ($cmd, at args);
die "run_command: failed to exec $cmd_text: $!";
};
- my($err) = $@; chomp($err);
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err;
eval {
+ local(@SIG{qw(INT HUP TERM TSTP QUIT USR1 USR2)}) = ($h2) x 7;
+ if ($interrupt ne '') { my($i) = $interrupt; $interrupt = ''; die $i }
open_log(); # oops, exec failed, we will need logging after all...
- do_log(-2,"run_command: child process [%s]: %s", $$,$err);
+ # we're in trouble if stderr was attached to a terminal, but no longer is
+ do_log(-1,"run_command: child process [%s]: %s", $$,$err);
};
{ no warnings;
- POSIX::_exit(1); # avoid END and destructor processing
+ POSIX::_exit(8); # avoid END and destructor processing
kill('KILL',$$); exit 1; # still kicking? die!
}
}
@@ -2409,21 +2651,27 @@ sub run_command_consumer($$@) {
my($stdout_to, $stderr_to, $cmd, @args) = @_;
my($cmd_text) = join(' ', $cmd, @args);
$stdout_to = '/dev/null' if $stdout_to eq '';
- $stderr_to = '/dev/null' if $stderr_to eq '';
+ $stderr_to = '/dev/null' if defined($stderr_to) && $stderr_to eq '';
my($msg) = join(' ', $cmd, @args, ">$stdout_to",
$stderr_to eq '' ? () : "2>$stderr_to");
# $^F == 2 or do_log(-1,"run_command_consumer: SYSTEM_FD_MAX not 2: %d", $^F);
my($pid); my($proc_fh) = IO::File->new;
- eval { $pid = $proc_fh->open('|-') }; # fork, catching errors
- if ($@ ne '') { chomp($@); die "run_command_consumer (open pipe): $@ ($!)" }
+ eval {
+ $pid = $proc_fh->open('|-'); 1; # fork, catching errors
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die "run_command_consumer (open pipe): $eval_stat";
+ };
defined($pid) or die "run_command_consumer: can't fork: $!";
- if (!$pid) { # child
- eval { # must not use die in forked process, or we end up with
- # two running daemons!
-# $sql_dataset_conn_lookups->dbh_inactive(1) if $sql_dataset_conn_lookups;
-# $sql_dataset_conn_storage->dbh_inactive(1) if $sql_dataset_conn_storage;
-# $sql_dataset_conn_lookups = $sql_dataset_conn_storage = undef;
-
+ if (!$pid) { # child
+ alarm(0); my($interrupt) = '';
+ my($h1) = sub { $interrupt = $_[0] };
+ my($h2) = sub { die "Received signal ".$_[0] };
+ @SIG{qw(INT HUP TERM TSTP QUIT USR1 USR2)} = ($h1) x 7;
+ eval { # die must be caught, otherwise we end up with two running daemons
+ local(@SIG{qw(INT HUP TERM TSTP QUIT USR1 USR2)}) = ($h2) x 7;
+ if ($interrupt ne '') { my($i) = $interrupt; $interrupt = ''; die $i }
+ release_parent_resources();
open_on_specific_fd(1,$stdout_to,&POSIX::O_WRONLY,0);
open_on_specific_fd(2,$stderr_to,&POSIX::O_WRONLY,0) if $stderr_to ne '';
# eval { close_log() }; # may have been closed by open_on_specific_fd
@@ -2431,20 +2679,175 @@ sub run_command_consumer($$@) {
exec {$cmd} ($cmd, at args);
die "run_command_consumer: failed to exec $cmd_text: $!";
};
- my($err) = $@; chomp($err);
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err;
eval {
+ local(@SIG{qw(INT HUP TERM TSTP QUIT USR1 USR2)}) = ($h2) x 7;
+ if ($interrupt ne '') { my($i) = $interrupt; $interrupt = ''; die $i }
open_log(); # oops, exec failed, we will need logging after all...
- do_log(-2,"run_command_consumer: child process [%s]: %s", $$,$err);
+ # we're in trouble if stderr was attached to a terminal, but no longer is
+ do_log(-1,"run_command_consumer: child process [%s]: %s", $$,$err);
};
{ no warnings;
- POSIX::_exit(1); # avoid END and destructor processing
+ POSIX::_exit(8); # avoid END and destructor processing
kill('KILL',$$); exit 1; # still kicking? die!
}
}
# parent
- ll(5) && do_log(5,"run_command_consumer: [%s] %s", $pid,$msg);
+ ll(5) && do_log(5,"run_command_consumer: [%s]", $pid);
binmode($proc_fh) or die "Can't set pipe to binmode: $!"; # dflt Perl 5.8.1
($proc_fh, $pid); # return pipe file handle to the subprocess and its PID
+}
+
+# run a specified subroutine with given arguments as a (forked) subprocess,
+# collecting results (if any) over a pipe from a subprocess and propagating
+# them back to a caller; (useful to prevent a potential process crash from
+# bringing down the main process, and allows cleaner timeout aborts)
+#
+sub run_as_subprocess($@) {
+ my($code, at args) = @_;
+ my($pid); my($proc_fh) = IO::File->new;
+ eval {
+ $pid = $proc_fh->open('-|'); 1; # fork, catching errors
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die "run_as_subprocess (open pipe): $eval_stat";
+ };
+ defined($pid) or die "run_as_subprocess: can't fork: $!";
+ if (!$pid) { # child
+ alarm(0); # stop the timer, timeouts will be handled by a parent process
+ my($t0) = Time::HiRes::time; my(@result); my($interrupt) = '';
+ my($h1) = sub { $interrupt = $_[0] };
+ my($h2) = sub { die "Received signal ".$_[0] };
+ @SIG{qw(INT HUP TERM TSTP QUIT USR1 USR2)} = ($h1) x 7;
+ $SIG{PIPE} = 'IGNORE'; # write to broken pipe should not throw a signal
+ $0 = 'sub-' . $0; # let it show in ps(1)
+ my($eval_stat);
+ eval { # die must be caught, otherwise we end up with two running daemons
+ local(@SIG{qw(INT HUP TERM TSTP QUIT USR1 USR2)}) = ($h2) x 7;
+ if ($interrupt ne '') { my($i) = $interrupt; $interrupt = ''; die $i }
+ release_parent_resources();
+ binmode(STDOUT) or die "Can't set STDOUT to binmode: $!";
+ do_log(5,"[%s] run_as_subprocess: running as child, stdin=%s, stdout=%s",
+ $$, fileno(STDIN), fileno(STDOUT)) if ll(5);
+ @result = &$code(@args); # invoke the caller-specified subroutine
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ my($dt) = Time::HiRes::time - $t0;
+ eval { # must not use die in forked process, or we end up with two daemons
+ local(@SIG{qw(INT HUP TERM TSTP QUIT USR1 USR2)}) = ($h2) x 7;
+ if ($interrupt ne '') { my($i) = $interrupt; $interrupt = ''; die $i }
+ my($status); my($ll) = 3;
+ if ($eval_stat ne '') { # failure
+ chomp $eval_stat; $ll = -2;
+ $status = sprintf("STATUS: FAILURE %s", $eval_stat);
+ } else { # success
+ $status = sprintf("STATUS: SUCCESS, %d results", scalar(@result));
+ };
+ my($frozen) = Amavis::Util::freeze([$status, at result]);
+ ll($ll) && do_log($ll, "[%s] run_as_subprocess: child done (%.1f ms), ".
+ "sending results: res_len=%d, %s",
+ $$, $dt*1000, length($frozen), $status);
+ # write results back to a parent process over a pipe as a frozen struct.
+ # writing to broken pipe must return an error, not throw a signal
+ local $SIG{PIPE} = sub { die "Broken pipe\n" }; # locale-independent err
+ print STDOUT ($frozen) or die "Can't write result to pipe: $!";
+ close(STDOUT) or die "Can't close STDOUT: $!";
+ POSIX::_exit(0); # normal completion, avoid END and destructor processing
+ };
+ my($eval2_stat) = $@ ne '' ? $@ : "errno=$!";
+ eval {
+ chomp $eval2_stat;
+ if ($interrupt ne '') { my($i) = $interrupt; $interrupt = ''; die $i }
+ # broken pipe is common when parent process is shutting down
+ my($ll) = $eval2_stat =~ /^Broken pipe\b/ ? 1 : -1;
+ do_log($ll,"run_as_subprocess: child process [%s]: %s", $$,$eval2_stat);
+ };
+ POSIX::_exit(8); # avoid END and destructor processing in a subprocess
+ }
+ # parent
+ ll(5) && do_log(5,"run_as_subprocess: spawned a subprocess [%s]", $pid);
+ binmode($proc_fh) or die "Can't set pipe to binmode: $!"; # dflt Perl 5.8.1
+ ($proc_fh, $pid); # return pipe file handle to the subprocess and its PID
+}
+
+# read results from a subprocess over a pipe, returns a ref to a results string
+# and a subprocess exit status; close the pipe and dismiss the subprocess,
+# by force if necessary; if $success_list_ref is defined, check also the
+# subprocess exit status against the provided list and log results
+#
+sub collect_results($$;$$$) {
+ my($proc_fh,$pid, $what,$results_max_size,$success_list_ref) = @_;
+ # $results_max_size is interpreted as follows:
+ # undef .. no limit, read and return all data;
+ # 0 ... no limit, read and discard all data, returns ref to empty string
+ # >= 1 ... read all data, but truncate results string at limit
+ my($child_stat); my($close_err) = 0; my($pid_orig) = $pid;
+ my($result) = ''; my($result_l) = 0; my($skipping) = 0; my($eval_stat);
+ eval { # read results; could be aborted by a read error or a timeout
+ my($nbytes,$buff);
+ while (($nbytes=$proc_fh->read($buff,8192)) > 0) {
+ if (!defined($results_max_size)) { $result .= $buff } # keep all data
+ elsif ($results_max_size == 0 || $skipping) {} # discard data
+ elsif ($result_l <= $results_max_size) { $result .= $buff }
+ else {
+ $skipping = 1; # sanity limit exceeded
+ do_log(-1,'collect_results from [%s] (%s): results size limit '.
+ '(%d bytes) exceeded', $pid_orig,$what,$results_max_size);
+ }
+ $result_l += $nbytes;
+ }
+ defined $nbytes or die "Error reading from a subprocess [$pid_orig]: $!";
+ ll(5) && do_log(5,'collect_results from [%s] (%s), %d bytes, (limit %s)',
+ $pid_orig,$what,$result_l,$results_max_size);
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ if ($results_max_size > 0 && length($result) > $results_max_size)
+ { $result = substr($result,0,$results_max_size) . "..." }
+ if ($eval_stat ne '') { # read error or timeout; abort the subprocess
+ chomp $eval_stat;
+ undef $_[0]; # release the caller's copy of $proc_fh
+ kill_proc($pid,$what,1,$proc_fh, "on reading: $eval_stat") if defined $pid;
+ undef $proc_fh; undef $pid;
+ die "collect_results - reading aborted: $eval_stat";
+ }
+ # normal subprocess exit, close pipe, collect exit status
+ undef $eval_stat;
+ eval {
+ $proc_fh->close or $close_err = $!;
+ $child_stat = $?; undef $proc_fh; undef $pid;
+ undef $_[0]; # release also the caller's copy of $proc_fh
+ 1;
+ } or do { # just in case close itself timed out
+ $eval_stat = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ undef $_[0]; # release the caller's copy of $proc_fh
+ kill_proc($pid,$what,1,$proc_fh, "on closing: $eval_stat") if defined $pid;
+ undef $proc_fh; undef $pid;
+ die "collect_results - closing aborted: $eval_stat";
+ };
+ if (defined $success_list_ref) {
+ proc_status_ok($child_stat,$close_err, @$success_list_ref)
+ or do_log(-1, 'collect_results from [%s] (%s): %s %s', $pid_orig,$what,
+ exit_status_str($child_stat,$close_err), $result);
+ } elsif ($close_err != 0) {
+ die "Can't close pipe to subprocess [$pid_orig]: $close_err";
+ }
+ (\$result,$child_stat);
+}
+
+# read results from a subprocess over a pipe as a frozen data structure;
+# close the pipe and dismiss the subprocess; returns results as a ref to a list
+sub collect_results_structured($$;$$) {
+ my($proc_fh,$pid, $what,$results_max_size) = @_;
+ my($result_ref,$child_stat) =
+ collect_results($proc_fh,$pid, $what,$results_max_size,[0]);
+ my($result_ref) = Amavis::Util::thaw($$result_ref);
+ my(@result) = !ref($result_ref) ? () : @$result_ref;
+ @result >= 1
+ or die "collect_results_structured: no results from subprocess [$pid]";
+ my($status) = shift(@result);
+ $status =~ /^STATUS: (?:SUCCESS|FAILURE)\b/
+ or die "collect_results_structured: subprocess [$pid] returned: $status";
+ (\@result,$child_stat);
}
sub dynamic_destination($$$) {
@@ -2483,14 +2886,15 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT = qw(
&iso8601_timestamp &iso8601_utc_timestamp &rfc2822_timestamp
&format_time_interval &received_line &parse_received
- &fish_out_ip_from_received &split_address &split_localpart &make_query_keys
+ &fish_out_ip_from_received &parse_message_id
+ &split_address &split_localpart &replace_addr_fields &make_query_keys
"e_rfc2821_local &qquote_rfc2821_local
- &parse_quoted_rfc2821 &unquote_rfc2821_local
+ &parse_quoted_rfc2821 &unquote_rfc2821_local &parse_address_list
&wrap_string &wrap_smtp_resp &one_response_for_all
&EX_OK &EX_NOUSER &EX_UNAVAILABLE &EX_TEMPFAIL &EX_NOPERM);
}
@@ -2510,7 +2914,7 @@ BEGIN {
BEGIN {
import Amavis::Conf qw(:platform c cr ca);
- import Amavis::Util qw(ll do_log);
+ import Amavis::Util qw(ll do_log unique);
}
# Given a Unix time, return the local time zone offset at that time
@@ -2579,6 +2983,7 @@ sub rfc2822_timestamp($) {
sub format_time_interval($) {
my($t) = @_;
+ return 'undefined' if !defined($t);
my($sign) = ''; if ($t < 0) { $sign = '-'; $t = - $t };
my($dd) = int($t / (24*3600)); $t = $t - $dd*(24*3600);
my($hh) = int($t / 3600); $t = $t - $hh*3600;
@@ -2649,7 +3054,7 @@ sub parse_received($) {
$fields{$field} = [$item, $v1, $v2, $v3, $comment];
ll(5) && do_log(5, "parse_received: %s = %s/%s/%s/%s",
map { !defined($_) ? '' : length($_) <= 50 ? $_
- : substr($_,0,50)."..." }
+ : substr($_,0,50)."[...]" }
($field, @{$fields{$field}}) ) if $field ne '';
}
}
@@ -2709,11 +3114,57 @@ sub split_localpart($$) {
} elsif ($delimiter eq '-' && $owner_request_special &&
$localpart =~ /^owner-.|.-request\z/si) {
# don't split owner-foo or foo-request
- } elsif ($localpart =~ /^(.+?)\Q$delimiter\E(.*)\z/s) {
- ($localpart, $extension) = ($1, $2);
+ } elsif ($localpart =~ /^(.+?)(\Q$delimiter\E.*)\z/s) {
+ ($localpart, $extension) = ($1, $2); # extension includes a delimiter
# do not split the address if the result would have a null localpart
}
($localpart, $extension);
+}
+
+# replace localpart/extension/domain fields of an original email address
+# with nonempty fields of a replacement
+#
+sub replace_addr_fields($$;$) {
+ my($orig_addr, $repl_addr, $delim) = @_;
+ my($localpart_o, $domain_o, $ext_o, $localpart_r, $domain_r, $ext_r);
+ ($localpart_o,$domain_o) = split_address($orig_addr);
+ ($localpart_r,$domain_r) = split_address($repl_addr);
+ $localpart_r = $localpart_o if $localpart_r eq '';
+ $domain_r = $domain_o if $domain_r eq '';
+ if ($delim ne '') {
+ ($localpart_o,$ext_o) = split_localpart($localpart_o,$delim);
+ ($localpart_r,$ext_r) = split_localpart($localpart_r,$delim);
+ $ext_r = $ext_o if !defined($ext_r);
+ }
+ $localpart_r . (defined $ext_r ? $ext_r : '') . $domain_r;
+}
+
+# given a (potentially multiline) header field Message-ID, Resent-Message-ID.
+# In-Reply-To, or References, parse the rfc2822 syntax extracting all
+# message IDs while ignoring comments, and return them as a list
+#
+sub parse_message_id($) {
+ my($str) = @_;
+ $str =~ tr/\n//d; my(@message_id); my($garbage) = 0;
+ for my $t ( $str =~ /\G ( \( (?: \\. | [^()\\] )* \) | [ \t]+ |
+ < (?: " (?: \\. | [^"\\] )* " |
+ \[ (?: \\. | [^\]\\] )* \] |
+ [^"<>\[\]\\]+ )* > |
+ [^<( \t]+ | . )/gcsx ) {
+ if ($t =~ /^<.*>\z/) { push(@message_id,$t) }
+ elsif ($t =~ /^[ \t]*\z/) {} # ignore FWS
+ elsif ($t =~ /^\(.*\)\z/) # ignore CFWS
+ { do_log(2, "parse_message_id ignored comment: /%s/ in %s", $t,$str) }
+ else { $garbage = 1 }
+ }
+ if (@message_id > 1) {
+ @message_id = @{unique(\@message_id)}; # remove possible duplicates
+ } elsif ($garbage && !@message_id) {
+ local($_) = $str; s/^[ \t]+//; s/[ \t\n]+\z//; # trim and sanitize <...>
+ s/^<//; s/>\z//; s/>/_/g; $_ = '<'.$_.'>'; @message_id = ($_);
+ do_log(5, "parse_message_id sanitizing garbage: /%s/ to %s", $str,$_);
+ }
+ @message_id;
}
# For a given email address (e.g. for User+Foo at sub.exAMPLE.CoM)
@@ -2747,16 +3198,17 @@ sub make_query_keys($$$) {
my($extension); my($delim) = c('recipient_delimiter');
if ($delim ne '') {
($localpart,$extension) = split_localpart($localpart,$delim);
+ # extension includes a delimiter since amavisd-new-2.5.0!
}
$extension = '' if !defined($extension); # mute warnings
my($append_to_user,$prepend_to_domain) = $at_with_user ? ('@','') : ('','@');
my(@keys); # a list of query keys
push(@keys, $addr); # as is
- push(@keys, $localpart.$delim.$extension.'@'.$domain)
+ push(@keys, $localpart.$extension.'@'.$domain)
if $extension ne ''; # user+foo at example.com
push(@keys, $localpart.'@'.$domain); # user at example.com
if ($include_bare_user) { # typically enabled for local users only
- push(@keys, $localpart.$delim.$extension.$append_to_user)
+ push(@keys, $localpart.$extension.$append_to_user)
if $extension ne ''; # user+foo(@)
push(@keys, $localpart.$append_to_user); # user(@)
}
@@ -2783,7 +3235,7 @@ sub make_query_keys($$$) {
$addr, # $1 = User+Foo at Sub.Example.COM
$saved_full_localpart, # $2 = User+Foo
$localpart, # $3 = user
- $delim.$extension, # $4 = +foo
+ $extension, # $4 = +foo
$domain, # $5 = sub.example.com
];
($keys_ref, $rhs);
@@ -2877,9 +3329,77 @@ sub unquote_rfc2821_local($) {
my($source_route,$localpart,$domain) = parse_quoted_rfc2821($mailbox,1);
# make address with '@' in the localpart but no domain (like <"aa at bb.com"> )
# distinguishable from <aa at bb.com> by representing it as aa at bb.com@ in
- # unquoted form; (it still obeys all regular rules, it is not a trick)
+ # unquoted form; (it still obeys all regular rules, it is not a dirty trick)
$domain = '@' if $domain eq '' && $localpart ne '' && $localpart =~ /\@/;
$localpart . $domain;
+}
+
+# Parse a rfc2822.address-list, returning a list of rfc2822(quoted) addresses.
+# Properly deals with group addresses, nested comments, address literals,
+# qcontent, addresses with source route, discards display names and comments.
+# The following header fields accept address-list: To, Cc, Bcc, Reply-To.
+# A header field 'From' accepts a 'mailbox-list' syntax (which is similar,
+# but does not allow groups); a header field 'Sender' accepts a 'mailbox'
+# syntax, i.e. only one address and not a group.
+#
+use vars qw($s $p @addresses);
+sub flush_a() {
+ $s =~ s/^[ \t]+//s; $s =~ s/[ \t]\z//s;
+ $p =~ s/^[ \t]+//s; $p =~ s/[ \t]\z//s;
+ if ($p ne '') { $p =~ s/^<//; $p =~ s/>\z//; push(@addresses,$p) }
+ elsif ($s ne '') { push(@addresses,$s) }
+ $p = ''; $s = '';
+}
+sub parse_address_list($) {
+ local($_) = $_[0];
+ local($1); s/\n([ \t])/$1/gs; s/\n+\z//s; # unfold, chomp
+ my($str_l) = length($_); $p = ''; $s = ''; @addresses = ();
+ my($comm_lvl) = 0; my($in_qcontent) = 0; my($in_literal) = 0;
+ my($in_group) = 0; my($in_angle) = 0; my($after_at) = 0; my($new_pos);
+ for (my $pos=-1; $new_pos=pos($_), $new_pos<$str_l; $pos=$new_pos) {
+ $new_pos > $pos or die "parse_address_list PANIC1 $new_pos"; # just in case
+ # comment (may be nested: rfc2822 section 3.2.3)
+ if ($comm_lvl > 0 && /\G( \) )/gcsx) { $comm_lvl--; next }
+ if (!$in_qcontent && !$in_literal && /\G( \( )/gcsx) { $comm_lvl++; next }
+ if ($comm_lvl > 0 && /\G( (?: \\. | [^()\\] )+ )/gcsx) { next }
+ # quoted content
+ if ($in_qcontent && /\G( " )/gcsx)
+ { $in_qcontent = 0; ($in_angle?$p:$s) .= $1; next }
+ if (!$comm_lvl && !$in_qcontent && !$in_literal && /\G( " )/gcsx)
+ { $in_qcontent = 1; ($in_angle?$p:$s) .= $1; next }
+ if ($in_qcontent && /\G( (?: \\. | [^"\\] )+ )/gcsx)
+ { ($in_angle?$p:$s) .= $1; next }
+ # address literal
+ if ($in_literal && /\G( \] )/gcsx)
+ { $in_literal = 0; ($in_angle?$p:$s) .= $1; next }
+ if (!$comm_lvl && !$in_qcontent && /\G( \[ )/gcsx)
+ { $in_literal = 1 if $after_at; ($in_angle?$p:$s) .= $1; next }
+ if ($in_literal && /\G( (?: \\. | [^\]\\] )+ )/gcsx)
+ { ($in_angle?$p:$s) .= $1; next }
+ # normal content
+ if (!$comm_lvl && !$in_qcontent && !$in_literal) {
+ if (!$in_angle && /\G( < )/gcsx)
+ { $in_angle = 1; $after_at = 0; flush_a() if $p ne ''; $p .= $1; next }
+ if ( $in_angle && /\G( > )/gcsx)
+ { $in_angle = 0; $after_at = 0; $p .= $1; next }
+ if (/\G( , )/gcsx) # top-level addr separator or source route delimiter
+ { !$in_angle ? flush_a() : ($p.=$1); $after_at = 0; next }
+ if (!$in_angle && !$in_group && /\G( : )/gcsx) # group name terminator
+ { $in_group = 1; $s .= $1; $p=$s=''; next } # discard group name
+ if ($after_at && /\G( : )/gcsx) # source route terminator
+ { $after_at = 0; ($in_angle?$p:$s) .= $1; next }
+ if ( $in_group && /\G( ; )/gcsx) # group terminator
+ { $in_group = 0; $after_at = 0; next }
+ if (!$in_group && /\G( ; )/gcsx) # out of place special
+ { ($in_angle?$p:$s) .= $1; $after_at = 0; next }
+ if (/\G( \@ )/gcsx) { $after_at = 1; ($in_angle?$p:$s) .= $1; next }
+ if (/\G( [ \t]+ )/gcsx) { ($in_angle?$p:$s) .= $1; next }
+ if (/\G( [^,:;@<>()"\[\]\\]+ )/gcsx) { ($in_angle?$p:$s) .= $1; next }
+ }
+ if (/\G( . )/gcsx) { ($in_angle?$p:$s) .= $1; next } # other junk
+ die "parse_address_list PANIC2 $new_pos"; # just in case
+ }
+ flush_a(); @addresses;
}
# compute a total displayed line size if a string (possibly containing TAB
@@ -3037,10 +3557,10 @@ sub wrap_smtp_resp($) {
# an indication whether a non delivery notification (NDN, a form of DSN)
# is needed).
#
-sub one_response_for_all($$$) {
- my($msginfo, $dsn_per_recip_capable, $am_id) = @_;
+sub one_response_for_all($$;$) {
+ my($msginfo, $dsn_per_recip_capable, $suppressed) = @_;
my($smtp_resp, $exit_code, $ndn_needed);
-
+ my($am_id) = $msginfo->log_id;
my($delivery_method) = $msginfo->delivery_method;
my($sender) = $msginfo->sender;
my($per_recip_data) = $msginfo->per_recip_data;
@@ -3137,7 +3657,9 @@ sub one_response_for_all($$$) {
$smtp_resp .= join ", and ",
map { my($cnt, $nm) = @$_;
!$cnt ? () : $cnt == @$per_recip_data ? $nm : "$cnt $nm"
- } ([$rej_cnt,'REJECT'], [$bounce_cnt,'BOUNCE'], [$drop_cnt,'DISCARD']);
+ } ([$rej_cnt, 'REJECT'],
+ [$bounce_cnt, $suppressed ? 'DISCARD(bounce.suppressed)' :'BOUNCE'],
+ [$drop_cnt, 'DISCARD']);
}
$ndn_needed =
($bounce_cnt > 0 || ($rej_cnt > 0 && !$dsn_per_recip_capable)) ? 1 : 0;
@@ -3160,7 +3682,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
BEGIN { import Amavis::Util qw(ll do_log fmt_struct) }
@@ -3230,7 +3752,7 @@ sub lookup_re($$;$) {
($key,$r) = ($e, 1);
}
""=~/x{0}/; # braindead Perl: serves as explicit deflt for an empty regexp
- my(@rhs); # match, capturing parenthesized subpatterns in @rhs
+ my(@rhs); # match, capturing parenthesized subpatterns into @rhs
if (!ref($addr)) { @rhs = $addr =~ /$key/ }
else { for (@$addr) { @rhs = /$key/; last if @rhs } } # inner loop
if (@rhs) { # regexp matches
@@ -3278,7 +3800,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&lookup_ip_acl);
}
@@ -3437,7 +3959,9 @@ sub ip_to_vec($;$) {
#
sub lookup_ip_acl($@) {
my($ip, @nets_ref) = @_;
- my($ip_vec,$ip_mask) = eval { ip_to_vec($ip,0) }; my($eval_stat) = $@;
+ my($ip_vec,$ip_mask); my($eval_stat);
+ eval { ($ip_vec,$ip_mask) = ip_to_vec($ip,0); 1 }
+ or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
my($label,$fullkey,$result); my($found) = 0;
for my $tb (@nets_ref) {
my($t) = ref($tb) eq 'REF' ? $$tb : $tb; # allow one level of indirection
@@ -3480,7 +4004,7 @@ sub lookup_ip_acl($@) {
$result = 1 - $result if (length($1) & 1); # negate if odd
}
($acl_ip_vec, $acl_mask, $acl_mask_len) = ip_to_vec($key,1);
- if ($acl_mask_len == 0) { $found=1 } # even invalid addr matches ::/0
+ if ($acl_mask_len == 0) { $found=1 } #even an invalid addr matches ::/0
elsif (!defined($ip_vec)) {} # no other matches for invalid address
elsif (($ip_vec & $acl_mask) eq ($acl_ip_vec & $acl_mask)) { $found=1 }
last if $found;
@@ -3489,7 +4013,7 @@ sub lookup_ip_acl($@) {
my($acl_ip_vec, $acl_mask, $acl_mask_len);
for my $e (@$t) {
($fullkey, $acl_ip_vec, $acl_mask, $acl_mask_len, $result) = @$e;
- if ($acl_mask_len == 0) { $found=1 } # even invalid address matches /0
+ if ($acl_mask_len == 0) { $found=1 } #even an invalid addr matches ::/0
elsif (!defined($ip_vec)) {} # no other matches for invalid address
elsif (($ip_vec & $acl_mask) eq ($acl_ip_vec & $acl_mask)) { $found=1 }
last if $found;
@@ -3508,7 +4032,9 @@ sub lookup_ip_acl($@) {
!$found ? ", no match" : " matches \"$fullkey\", result=$result");
if ($eval_stat eq '') { undef $eval_stat }
else {
- chomp($eval_stat); $eval_stat = "lookup_ip_acl$label: $eval_stat";
+ chomp $eval_stat;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ $eval_stat = "lookup_ip_acl$label: $eval_stat";
do_log(2, "%s", $eval_stat);
}
!wantarray ? $result : ($result, $fullkey, $eval_stat);
@@ -3553,7 +4079,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&lookup);
}
@@ -3618,8 +4144,8 @@ sub lookup_hash($$;$) {
# false (undef). Search is always case-insensitive on domain part,
# local part matching depends on $localpart_is_case_sensitive setting.
#
-# lookup_acl is not aware of address extensions and they are not
-# handled specially.
+# NOTE: lookup_acl is not aware of address extensions and they are
+# not handled specially!
#
# If a list element contains a '@', the full e-mail address is compared,
# otherwise if a list element has a leading dot, the domain name part is
@@ -3809,7 +4335,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&expand &tokenize);
}
@@ -3896,15 +4422,18 @@ sub evalmacro($$;@) {
while (@args >= 2) { # at least a regexp and a 'then' argument still there
@repl = ();
my($regexp) = tokens_list_to_str(shift(@args)); # collect a regexp arg
- $regexp =~ s{( \@ | \$ (?!\z) )}{\\$1}gsx; # protect $ and @
""=~/x{0}/; #braindead Perl: serves as explicit deflt for an empty regexp
eval { # guard against invalid regular expression
local($1,$2,$3,$4,$5,$6,$7,$8,$9);
$match = $str=~/$regexp/ ? 1 : 0;
@repl = ($1,$2,$3,$4,$5,$6,$7,$8,$9) if $match;
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ do_log(2,"invalid macro regexp arg: %s", $eval_stat);
+ $match = 0; @repl = ();
};
- if ($@ ne '')
- { $match = 0; @repl = (); do_log(2,"invalid macro regexp arg: %s",$@) }
if ($match) { last } else { shift(@args) } # skip 'then' arg if no match
}
if (@args > 0) {
@@ -4099,15 +4628,15 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
-use Errno qw(ENOENT);
+use Errno qw(ENOENT EACCES);
use IO::File qw(O_RDONLY O_WRONLY O_RDWR O_APPEND O_CREAT O_EXCL);
BEGIN {
- import Amavis::Conf qw(:confvars :platform);
+ import Amavis::Conf qw(:platform :confvars);
import Amavis::Timing qw(section_time);
import Amavis::Util qw(ll do_log add_entropy rmdir_recursively);
import Amavis::rfc2821_2822_Tools qw(iso8601_timestamp);
@@ -4139,30 +4668,37 @@ sub preserve { # Whether to preserve di
# Clean up the tempdir on shutdown
sub DESTROY {
my($self) = shift;
- eval { do_log(5,"Amavis::TempDir::DESTROY called") };
- eval {
- $self->{fh_pers}->close
- or die "Error closing temp file: $!" if $self->{fh_pers};
- $self->{fh_pers} = undef;
- my($errn) = $self->{tempdir_path} eq '' ? ENOENT
- : (stat($self->{tempdir_path}) ? 0 : 0+$!);
- if (defined $self->{tempdir_path} && $errn != ENOENT) {
- # this will not be included in the TIMING report,
- # but it only occurs infrequently and doesn't take that long
- if ($self->{preserve} && !$self->{empty}) {
- do_log(-1,"TempDir removal: tempdir is to be PRESERVED: %s",
- $self->{tempdir_path});
- } else {
- do_log(3, "TempDir removal: %s is being removed: %s%s",
- $self->{empty} ? 'empty tempdir' : 'tempdir',
- $self->{tempdir_path},
- $self->{preserve} ? ', nothing to preserve' : '');
- rmdir_recursively($self->{tempdir_path});
- }
- }
- };
- if ($@ ne '')
- { my($eval_stat) = $@; eval { do_log(1,"TempDir removal: %s",$eval_stat) }}
+ if (defined($my_pid) && $$ != $my_pid) {
+ eval { do_log(5,"Amavis::TempDir DESTROY skip, clone [%s] (born as [%s])",
+ $$,$my_pid) };
+ } else {
+ eval { do_log(5,"Amavis::TempDir DESTROY called") };
+ eval {
+ $self->{fh_pers}->close
+ or do_log(-1,"Error closing temp file: %s",$!) if $self->{fh_pers};
+ $self->{fh_pers} = undef;
+ my($errn) = $self->{tempdir_path} eq '' ? ENOENT
+ : (stat($self->{tempdir_path}) ? 0 : 0+$!);
+ if (defined $self->{tempdir_path} && $errn != ENOENT) {
+ # this will not be included in the TIMING report,
+ # but it only occurs infrequently and doesn't take that long
+ if ($self->{preserve} && !$self->{empty}) {
+ do_log(-1,"TempDir removal: tempdir is to be PRESERVED: %s",
+ $self->{tempdir_path});
+ } else {
+ do_log(3, "TempDir removal: %s is being removed: %s%s",
+ $self->{empty} ? 'empty tempdir' : 'tempdir',
+ $self->{tempdir_path},
+ $self->{preserve} ? ', nothing to preserve' : '');
+ rmdir_recursively($self->{tempdir_path});
+ }
+ };
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ eval { do_log(1,"TempDir removal: %s",$eval_stat) };
+ };
+ }
}
# Creates the temporary directory, and checks that inode does not change
@@ -4239,7 +4775,7 @@ sub prepare_file {
or die "Can't create file $fname: $!";
if ($unicode_aware) {
binmode($newfh,":bytes") or die "Can't cancel :utf8 mode on $fname: $!";
- if (ll(5)) {
+ if (ll(5) && $] >= 5.008001) { # get_layers was added with Perl 5.8.1
my(@layers) = PerlIO::get_layers($newfh);
do_log(5,"TempDir::prepare_file: layers: %s", join(",", at layers));
}
@@ -4256,11 +4792,7 @@ sub prepare_file {
# Cleans the temporary directory for reuse, unless it is set to be preserved
sub clean {
my($self) = @_;
- my($to_be_preserved) = $self->{preserve};
- # turn preserve flag on (temporarily) so that DESTROY will retain files
- # in case this subroutine aborts and does not reach its normal exit
- $self->{preserve} = 1;
- if ($to_be_preserved && !$self->{empty}) {
+ if ($self->{preserve} && !$self->{empty}) {
# keep evidence in case of trouble
do_log(-1,"PRESERVING EVIDENCE in %s", $self->{tempdir_path});
if ($self->{fh_pers}) {
@@ -4284,7 +4816,7 @@ sub clean {
if (defined $self->{tempdir_path}) { # prepare for the next one
$self->strip; $self->{empty} = 1;
}
- $self->{preserve} = 0; # reset
+ $self->{preserve} = 0; # reset
}
# Remove all files and subdirectories from the temporary directory, leaving
@@ -4314,7 +4846,7 @@ sub strip {
#
sub check {
my($self) = shift;
- my($dir) = $self->{tempdir_path};
+ my($eval_stat); my($dir) = $self->{tempdir_path};
local(*DIR); opendir(DIR,$dir) or die "Can't open directory $dir: $!";
eval {
$! = 0; my($f);
@@ -4326,11 +4858,16 @@ sub check {
elsif ($f eq 'email.txt' && -f _) {}
else { die("Unexpected " . (-d _?'directory':'file') . " $dir/$f") }
}
- # checking status on directory read ops doesn't work as expected, Perl bug
- # $!==0 or die "Error reading directory $dir: $!";
- };
+ # checking status on directory read ops doesn't work as expected, Perl bug
+ # $!==0 or die "Error reading directory $dir: $!";
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
closedir(DIR) or die "Error closing directory $dir: $!";
- if ($@ ne '') { chomp($@); die "TempDir::check: $@\n" }
+ if ($eval_stat ne '') {
+ chomp $eval_stat;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ die "TempDir::check: $eval_stat\n";
+ }
1;
}
@@ -4348,7 +4885,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
use Errno qw(EIO);
@@ -4362,8 +4899,13 @@ sub new {
sub close {
my($self) = shift;
- my($status); eval { $status = $self->{fh}->gzclose }; delete $self->{fh};
- if ($status != Z_OK || $@ ne '') {
+ my($status); my($eval_stat);
+ eval { $status = $self->{fh}->gzclose; 1 }
+ or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ delete $self->{fh};
+ if ($status != Z_OK || $eval_stat ne '') {
+ chomp $eval_stat;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
die "gzclose error: $gzerrno"; # can't stash arbitrary text into $!
$! = EIO; return undef; # not reached
}
@@ -4453,7 +4995,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
@@ -4483,7 +5025,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
@@ -4494,57 +5036,65 @@ BEGIN {
}
sub new # NOTE: this class is a list for historical reasons, not a hash
- { my($class) = @_; bless [(undef) x 24], $class }
+ { my($class) = @_; bless [(undef) x 28], $class }
# subs to set or access individual elements of a n-tuple by name
sub recip_addr # raw (unquoted) recipient envelope e-mail address
{ my($self)=shift; !@_ ? $$self[0] : ($$self[0]=shift) }
+sub recip_addr_smtp # SMTP-encoded recipient envelope e-mail address in <>
+ { my($self)=shift; !@_ ? $$self[1] : ($$self[1]=shift) }
sub recip_addr_modified # recip. addr. with possible addr. extension inserted
- { my($self)=shift; !@_ ? $$self[1] : ($$self[1]=shift) }
-#sub recip_is_local # recip_addr matches @local_domains_maps (not implemented)
-# { my($self)=shift; !@_ ? $$self[2] : ($$self[2]=shift) }
+ { my($self)=shift; !@_ ? $$self[2] : ($$self[2]=shift) }
+sub recip_is_local # recip_addr matches @local_domains_maps
+ { my($self)=shift; !@_ ? $$self[3] : ($$self[3]=shift) }
sub recip_maddr_id # maddr.id field from SQL if logging to SQL is enabled
- { my($self)=shift; !@_ ? $$self[3] : ($$self[3]=shift) }
+ { my($self)=shift; !@_ ? $$self[4] : ($$self[4]=shift) }
sub recip_penpals_age # penpals age in sec from SQL if logging to SQL enabled
- { my($self)=shift; !@_ ? $$self[4] : ($$self[4]=shift) }
+ { my($self)=shift; !@_ ? $$self[5] : ($$self[5]=shift) }
+sub recip_penpals_score # penpals score (also added to recip_score_boost)
+ { my($self)=shift; !@_ ? $$self[6] : ($$self[6]=shift) }
sub dsn_notify # ESMTP RCPT command NOTIFY option (DSN-rfc3461, listref)
- { my($self)=shift; !@_ ? $$self[5] : ($$self[5]=shift) }
+ { my($self)=shift; !@_ ? $$self[7] : ($$self[7]=shift) }
sub dsn_orcpt # ESMTP RCPT command ORCPT option (DSN-rfc3461, encoded)
- { my($self)=shift; !@_ ? $$self[6] : ($$self[6]=shift) }
+ { my($self)=shift; !@_ ? $$self[8] : ($$self[8]=shift) }
sub dsn_suppress_reason # if defined disable sending DSN and supply a reason
- { my($self)=shift; !@_ ? $$self[7] : ($$self[7]=shift) }
+ { my($self)=shift; !@_ ? $$self[9] : ($$self[9]=shift) }
sub recip_destiny # D_REJECT, D_BOUNCE, D_DISCARD, D_PASS
- { my($self)=shift; !@_ ? $$self[8] : ($$self[8]=shift) }
+ { my($self)=shift; !@_ ? $$self[10] : ($$self[10]=shift) }
sub recip_done # false: not done, true: done (1: faked, 2: truly sent)
- { my($self)=shift; !@_ ? $$self[9] : ($$self[9]=shift) }
+ { my($self)=shift; !@_ ? $$self[11] : ($$self[11]=shift) }
sub recip_smtp_response # rfc2821 response (3-digit + enhanced resp + text)
- { my($self)=shift; !@_ ? $$self[10] : ($$self[10]=shift) }
+ { my($self)=shift; !@_ ? $$self[12] : ($$self[12]=shift) }
sub recip_remote_mta_smtp_response # smtp response as issued by remote MTA
- { my($self)=shift; !@_ ? $$self[11] : ($$self[11]=shift) }
+ { my($self)=shift; !@_ ? $$self[13] : ($$self[13]=shift) }
sub recip_remote_mta # remote MTA that issued the smtp response
- { my($self)=shift; !@_ ? $$self[12] : ($$self[12]=shift) }
+ { my($self)=shift; !@_ ? $$self[14] : ($$self[14]=shift) }
sub recip_mbxname # mailbox name or file when known (local:, bsmtp: or sql:)
- { my($self)=shift; !@_ ? $$self[13] : ($$self[13]=shift) }
+ { my($self)=shift; !@_ ? $$self[15] : ($$self[15]=shift) }
sub recip_whitelisted_sender # recip considers this sender whitelisted (> 0)
- { my($self)=shift; !@_ ? $$self[14] : ($$self[14]=shift) }
+ { my($self)=shift; !@_ ? $$self[16] : ($$self[16]=shift) }
sub recip_blacklisted_sender # recip considers this sender blacklisted
- { my($self)=shift; !@_ ? $$self[15] : ($$self[15]=shift) }
+ { my($self)=shift; !@_ ? $$self[17] : ($$self[17]=shift) }
sub recip_score_boost # recip adds spam points to the final score
- { my($self)=shift; !@_ ? $$self[16] : ($$self[16]=shift) }
-sub infected # contains a virus (1); check bypassed (undef); clean (0)
- { my($self)=shift; !@_ ? $$self[17] : ($$self[17]=shift) }
+ { my($self)=shift; !@_ ? $$self[18] : ($$self[18]=shift) }
+sub infected # contains a virus (1); check bypassed (undef); clean (0)
+ { my($self)=shift; !@_ ? $$self[19] : ($$self[19]=shift) }
sub banned_parts # banned part descriptions (ref to a list of banned parts)
- { my($self)=shift; !@_ ? $$self[18] : ($$self[18]=shift) }
+ { my($self)=shift; !@_ ? $$self[20] : ($$self[20]=shift) }
sub banned_keys # keys of matching banned rules (a ref to a list)
- { my($self)=shift; !@_ ? $$self[19] : ($$self[19]=shift) }
+ { my($self)=shift; !@_ ? $$self[21] : ($$self[21]=shift) }
sub banned_rhs # right-hand side of matching rules (a ref to a list)
- { my($self)=shift; !@_ ? $$self[20] : ($$self[20]=shift) }
+ { my($self)=shift; !@_ ? $$self[22] : ($$self[22]=shift) }
+sub mail_body_mangle # mail body is being modified (and how) (e.g. defanged)
+ { my($self)=shift; !@_ ? $$self[23] : ($$self[23]=shift) }
sub contents_category # sorted listref of "major,minor" strings(category types)
- { my($self)=shift; !@_ ? $$self[21] : ($$self[21]=shift) }
+ { my($self)=shift; !@_ ? $$self[24] : ($$self[24]=shift) }
+sub blocking_ccat # category type most responsible for blocking msg, or undef
+ { my($self)=shift; !@_ ? $$self[25] : ($$self[25]=shift) }
sub courier_control_file # path to control file containing this recipient
- { my($self)=shift; !@_ ? $$self[22] : ($$self[22]=shift) }
+ { my($self)=shift; !@_ ? $$self[26] : ($$self[26]=shift) }
sub courier_recip_index # index of recipient within control file
- { my($self)=shift; !@_ ? $$self[23] : ($$self[23]=shift) }
+ { my($self)=shift; !@_ ? $$self[27] : ($$self[27]=shift) }
sub recip_final_addr { # return recip_addr_modified if set, else recip_addr
my($self)=shift;
@@ -4581,14 +5131,6 @@ sub add_contents_category {
$self->contents_category($aref);
}
-# get the most relevant contents category (max number from the sorted array)
-sub main_contents_category {
- my($self) = shift; my($major,$minor);
- my($aref) = $self->contents_category; # first element has the largest value
- ($major,$minor) = split(/,/, $aref->[0], -1) if defined $aref;
- !wantarray ? $major : ($major,$minor);
-}
-
# is the "$major,$minor" category in the list?
sub is_in_contents_category {
my($self) = shift; my($major,$minor) = @_;
@@ -4600,24 +5142,36 @@ sub is_in_contents_category {
# get a setting corresponding to the most important contents category;
# i.e. the highest entry from the category list for which a corresponding entry
# in the associative array of settings exists determines returned setting;
-sub setting_by_contents_category($$) {
- my($self) = shift; my($settings_href) = @_;
- return undef if !defined($settings_href);
+sub setting_by_main_contents_category($@) {
+ my($self) = shift; my(@settings_href_list) = @_;
+ return undef if !@settings_href_list;
my($aref) = $self->contents_category;
- setting_by_given_contents_category($settings_href,
- !defined($aref) ? () : @$aref);
+ setting_by_given_contents_category($aref, at settings_href_list);
}
# get a list of settings corresponding to all relevant contents categories,
# sorted from the most important to the least important entry; entries which
-# have no corresponding setting are not included in the list; in scalar context
-# works like setting_by_contents_category, i.e. returns only the first element
-sub setting_by_contents_category_all($$) {
- my($self) = shift; my($settings_href) = @_;
- return undef if !defined($settings_href);
+# have no corresponding setting are not included in the list
+sub setting_by_main_contents_category_all($@) {
+ my($self) = shift; my(@settings_href_list) = @_;
+ return undef if !@settings_href_list;
my($aref) = $self->contents_category;
- setting_by_given_contents_category_all($settings_href,
- !defined($aref) ? () : @$aref);
+ setting_by_given_contents_category_all($aref, at settings_href_list);
+}
+
+sub setting_by_blocking_contents_category($@) {
+ my($self) = shift; my(@settings_href_list) = @_;
+ my($blocking_ccat) = $self->blocking_ccat;
+ !defined($blocking_ccat) ? undef
+ : setting_by_given_contents_category($blocking_ccat, @settings_href_list);
+}
+
+sub setting_by_contents_category($@) {
+ my($self) = shift; my(@settings_href_list) = @_;
+ my($blocking_ccat) = $self->blocking_ccat;
+ !defined($blocking_ccat)
+ ? $self->setting_by_main_contents_category(@settings_href_list)
+ : setting_by_given_contents_category($blocking_ccat, @settings_href_list);
}
1;
@@ -4632,14 +5186,15 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
BEGIN {
import Amavis::Conf qw(:platform);
- import Amavis::rfc2821_2822_Tools qw(rfc2822_timestamp quote_rfc2821_local);
- import Amavis::Util qw(xtext_encode);
+ import Amavis::rfc2821_2822_Tools qw(rfc2822_timestamp quote_rfc2821_local
+ qquote_rfc2821_local);
+ import Amavis::Util qw(orcpt_encode);
import Amavis::In::Message::PerRecip;
}
@@ -4661,6 +5216,8 @@ sub client_os_fingerprint # SMTP client
{ my($self)=shift; !@_ ? $self->{cli_p0f} : ($self->{cli_p0f}=shift) }
sub queue_id # MTA queue ID of message if known (Courier, milter/AM.PDP)
{ my($self)=shift; !@_ ? $self->{queue_id} : ($self->{queue_id}=shift) }
+sub log_id # task id as shown in the log, also known as am_id
+ { my($self)=shift; !@_ ? $self->{log_id} : ($self->{log_id}=shift) }
sub mail_id # long-term unique id of the message on this system
{ my($self)=shift; !@_ ? $self->{mail_id} : ($self->{mail_id}=shift) }
sub secret_id # secret string to grant access to message with mail_id
@@ -4683,11 +5240,17 @@ sub requested_by # Resent-From addr w
{ my($self)=shift; !@_ ? $self->{requested_by}:($self->{requested_by}=shift)}
sub body_type # ESMTP BODY param (rfc1652: 7BIT, 8BITMIME) or BINARYMIME
{ my($self)=shift; !@_ ? $self->{body_type} : ($self->{body_type}=shift) }
-sub sender # envelope sender
+sub header_8bit # true if header contains illegal chars with codes over 255
+ { my($self)=shift; !@_ ? $self->{header_8bit}: ($self->{header_8bit}=shift) }
+sub body_8bit # true if body contains chars with codes over 255
+ { my($self)=shift; !@_ ? $self->{body_8bit}: ($self->{body_8bit}=shift) }
+sub sender # envelope sender, internal form, e.g.: j doe at example.com
{ my($self)=shift; !@_ ? $self->{sender} : ($self->{sender}=shift) }
+sub sender_smtp # env sender, SMTP form in <>, e.g.: <"j doe"@example.com>
+ { my($self)=shift; !@_ ? $self->{sender_smtp}: ($self->{sender_smtp}=shift) }
sub sender_contact # unmangled sender address or undef (e.g. believed faked)
{ my($self)=shift; !@_ ? $self->{sender_c} : ($self->{sender_c}=shift) }
-sub sender_source # unmangled sender address or info from the trace
+sub sender_source # unmangled sender addr. or info from the trace (log/notif)
{ my($self)=shift; !@_ ? $self->{sender_src} : ($self->{sender_src}=shift) }
sub sender_maddr_id # maddr.id field from SQL if logging to SQL is enabled
{ my($self)=shift; !@_ ? $self->{maddr_id} : ($self->{maddr_id}=shift) }
@@ -4703,21 +5266,29 @@ sub mail_tempdir # work directory, un
{ my($self)=shift; !@_ ? $self->{mailtempdir} : ($self->{mailtempdir}=shift)}
sub header_edits # Amavis::Out::EditHeader object or undef
{ my($self)=shift; !@_ ? $self->{hdr_edits} : ($self->{hdr_edits}=shift) }
+sub rfc2822_from #author addresses list (rfc allows one or more), parsed 'From'
+ { my($self)=shift; !@_ ? $self->{hdr_from} : ($self->{hdr_from}=shift) }
+sub rfc2822_sender # sender address (rfc allows none or one), parsed 'Sender'
+ { my($self)=shift; !@_ ? $self->{hdr_sender} : ($self->{hdr_sender}=shift) }
+sub rfc2822_to # parsed 'To' header field: a list of recipients
+ { my($self)=shift; !@_ ? $self->{hdr_to} : ($self->{hdr_to}=shift) }
+sub rfc2822_cc # parsed 'Cc' header field: a list of Cc recipients
+ { my($self)=shift; !@_ ? $self->{hdr_to} : ($self->{hdr_to}=shift) }
sub orig_header_fields # some orig. header fields (first occurence) - a hashref
- { my($self)=shift; !@_ ? $self->{orig_hdr_f}: ($self->{orig_hdr_f}=shift) }
+ { my($self)=shift; !@_ ? $self->{orig_hdr_f} : ($self->{orig_hdr_f}=shift) }
sub orig_header # original header - an arrayref of lines, with trailing LF
{ my($self)=shift; !@_ ? $self->{orig_header}: ($self->{orig_header}=shift) }
sub orig_header_size # size of original header (in bytes)
{ my($self)=shift; !@_ ? $self->{orig_hdr_s} : ($self->{orig_hdr_s}=shift) }
sub orig_body_size # size of original body (in bytes)
{ my($self)=shift; !@_ ? $self->{orig_bdy_s} : ($self->{orig_bdy_s}=shift) }
-sub body_digest # message digest of a message body (e.g. MD5 or SHA1)
+sub body_digest # message digest of a message body (e.g. MD5, SHA1, SHA256)
{ my($self)=shift; !@_ ? $self->{body_digest}: ($self->{body_digest}=shift) }
sub quarantined_to # list of quar mailbox names or addresses if quarantined
{ my($self)=shift; !@_ ? $self->{quarantine} : ($self->{quarantine}=shift) }
sub quar_type # quarantine type: F/Z/B/Q/M (file/zipfile/bsmtp/sql/mailbox)
{ my($self)=shift; !@_ ? $self->{quar_type} : ($self->{quar_type}=shift) }
-sub dsn_sent # delivery status notification was sent(1) or faked(2)
+sub dsn_sent # delivery status notification was sent(1) or suppressed(2)
{ my($self)=shift; !@_ ? $self->{dsn_sent} : ($self->{dsn_sent}=shift) }
sub delivery_method # delivery method, or empty for implicit delivery (milter)
{ my($self)=shift; !@_ ? $self->{deliv_method}:($self->{deliv_method}=shift)}
@@ -4725,6 +5296,10 @@ sub client_delete # don't delete the t
{ my($self)=shift; !@_ ? $self->{client_del} :($self->{client_del}=shift)}
sub contents_category # sorted arrayref CC_VIRUS/CC_BANNED/CC_SPAM../CC_CLEAN
{ my($self)=shift; !@_ ? $self->{category} : ($self->{category}=shift) }
+sub blocking_ccat # category type most responsible for blocking msg, or undef
+ { my($self)=shift; !@_ ? $self->{bl_ccat} : ($self->{bl_ccat}=shift) }
+sub virusnames # a ref to a list of virus names detected, or undef
+ { my($self)=shift; !@_ ? $self->{virusnames} : ($self->{virusnames}=shift) }
sub spam_level
{ my($self)=shift; !@_ ? $self->{spam_level} :($self->{spam_level}=shift)}
sub spam_status # names+score of tests as returned by SA get_tag('TESTSSCORES')
@@ -4733,21 +5308,27 @@ sub spam_report # SA terse report of
{ my($self)=shift; !@_ ? $self->{spam_report} :($self->{spam_report}=shift)}
sub spam_summary # SA summary of tests hit for standard body reports
{ my($self)=shift; !@_ ? $self->{spam_summary}:($self->{spam_summary}=shift)}
-sub autolearn_status
- { my($self)=shift; !@_ ? $self->{a_learn_stat}:($self->{a_learn_stat}=shift)}
+
+# new style of providing additional information from checkers
+sub supplementary_info # holds a hash of tag/value pairs, like from SA get_tag
+ { my($self)=shift; my($key)=shift;
+ !@_ ? $self->{info_tag}{$key} : ($self->{info_tag}{$key}=shift);
+}
# the following methods apply on a per-message level as well, summarizing
# per-recipient information as far as possible
*add_contents_category =
\&Amavis::In::Message::PerRecip::add_contents_category;
-*main_contents_category =
- \&Amavis::In::Message::PerRecip::main_contents_category;
*is_in_contents_category =
\&Amavis::In::Message::PerRecip::is_in_contents_category;
+*setting_by_main_contents_category =
+ \&Amavis::In::Message::PerRecip::setting_by_main_contents_category;
+*setting_by_main_contents_category_all =
+ \&Amavis::In::Message::PerRecip::setting_by_main_contents_category_all;
+*setting_by_blocking_contents_category =
+ \&Amavis::In::Message::PerRecip::setting_by_blocking_contents_category;
*setting_by_contents_category =
\&Amavis::In::Message::PerRecip::setting_by_contents_category;
-*setting_by_contents_category_all =
- \&Amavis::In::Message::PerRecip::setting_by_contents_category_all;
# The order of entries in a per-recipient list is the original order
# in which recipient addresses (e.g. obtained via 'MAIL TO:') were received.
@@ -4771,8 +5352,9 @@ sub recips { # get or set a lis
$self->per_recip_data([ map {
my($per_recip_obj) = Amavis::In::Message::PerRecip->new;
$per_recip_obj->recip_addr($_);
- $per_recip_obj->dsn_orcpt(
- 'rfc822;'.xtext_encode(quote_rfc2821_local($_))) if $set_dsn_orcpt_too;
+ $per_recip_obj->recip_addr_smtp(qquote_rfc2821_local($_));
+ $per_recip_obj->dsn_orcpt(orcpt_encode($per_recip_obj->recip_addr_smtp))
+ if $set_dsn_orcpt_too;
$per_recip_obj->recip_destiny(D_PASS); # default is Pass
$per_recip_obj } @{$recips_list_ref} ]);
}
@@ -4796,7 +5378,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&hdr);
}
@@ -4883,25 +5465,29 @@ sub hdr($$$;$) {
my($field_name, $field_body, $structured, $wrap_char) = @_;
$wrap_char = "\t" if !defined($wrap_char);
local($1);
- if ($field_name =~ /^(X-.*|Subject|Comments)\z/si &&
- $field_body =~ /[^\011\012\040-\176]/ #any nonprintable except TAB and LF
+ if ($field_name =~ /^ (?: Subject\z | Comments\z |
+ X- (?! Envelope- (?:From|To)\z ) )/six &&
+ $field_body !~ /^[\t\n\040-\176]*\z/ # not all printable (or TAB or LF)
) { # encode according to RFC 2047
+ # actually RFC 2047 also allows encoded-words in rfc822 extension
+ # message header fields (now: optional header fields), within comments
+ # in structured header fields, or within 'phrase' (e.g. in From, To, Cc);
+ # we are being sloppy here!
$field_body =~ s/\n([ \t])/$1/g; # unfold
chomp($field_body);
my($field_body_octets);
+ my($chset) = c('hdr_encoding'); my($qb) = c('hdr_encoding_qb');
if (!$unicode_aware) { $field_body_octets = $field_body }
else {
- $field_body_octets = safe_encode(c('hdr_encoding'), $field_body);
+ $field_body_octets = safe_encode($chset, $field_body);
# do_log(5, "hdr - UTF-8 body: %s", $field_body);
# do_log(5, "hdr - body octets: %s", $field_body_octets);
}
- my($qb) = c('hdr_encoding_qb');
- if (uc($qb) eq 'Q') {
- $field_body = q_encode($field_body_octets, $qb, c('hdr_encoding'));
- } else {
- $field_body = MIME::Words::encode_mimeword($field_body_octets,
- $qb, c('hdr_encoding'));
- }
+ my($encoder_func) = uc($qb) eq 'Q' ? \&q_encode
+ : \&MIME::Words::encode_mimeword;
+ $field_body = join("\n", map { /^[\001-\011\013\014\016-\177]*\z/ ? $_
+ : &$encoder_func($_,$qb,$chset) }
+ split(/\n/, $field_body_octets, -1));
} else { # supposed to be in plain ASCII, let's make sure it is
$field_body = safe_encode('ascii', $field_body);
}
@@ -4923,7 +5509,7 @@ sub hdr($$$;$) {
for (@lines)
{ if (length($_) > 998) { $_ = substr($_,0,998-3).'...'; $trunc = 1 } }
if ($trunc) {
- do_log(0, "INFO: truncating long header field (len=%d): %s...",
+ do_log(0, "INFO: truncating long header field (len=%d): %s[...]",
length($str), substr($str,0,100) );
$str = join("\n", at lines);
}
@@ -5022,7 +5608,7 @@ sub write_header($$$$) {
do_log(0, "INFO: unfolded %d illegal all-whitespace ".
"continuation lines", $ill_white_cnt) if $ill_white_cnt;
do_log(0, "INFO: truncated %d header line(s) longer than 998 characters",
- $ill_white_cnt) if $ill_white_cnt;
+ $ill_long_cnt) if $ill_long_cnt;
}
$str = '';
if (c('append_header_fields_to_bottom'))
@@ -5043,7 +5629,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT = qw(&mail_dispatch);
}
@@ -5057,7 +5643,7 @@ sub mail_dispatch($$$$;$) {
my($conn) = shift;
my($msginfo,$initial_submission,$dsn_per_recip_capable,$filter) = @_;
my($via) = $msginfo->delivery_method;
- if ($via =~ /^smtp:/i) {
+ if ($via =~ /^(?:smtp|lmtp):/i) {
Amavis::Out::SMTP::mail_via_smtp(
dynamic_destination($via,$conn,$relayhost_is_client), @_);
} elsif ($via =~ /^pipe:/i) {
@@ -5077,7 +5663,7 @@ sub mail_dispatch($$$$;$) {
sub { shift->recip_final_addr !~ /\@/ ? 1 : 0 });
if (grep { !$_->recip_done } @{$msginfo->per_recip_data}) {
my($nm) = c('notify_method'); # deliver the rest
- if ($nm =~ /^smtp:/i) {
+ if ($nm =~ /^(?:smtp|lmtp):/i) {
Amavis::Out::SMTP::mail_via_smtp(
dynamic_destination($nm,$conn,$relayhost_is_client), at _) }
elsif ($nm =~ /^pipe:/i) { Amavis::Out::Pipe::mail_via_pipe($nm, @_) }
@@ -5102,7 +5688,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&best_try_originator_ip &best_try_originator
&first_received_from);
@@ -5116,39 +5702,6 @@ BEGIN {
split_address parse_received fish_out_ip_from_received);
import Amavis::Lookup qw(lookup);
import Amavis::Lookup::IP qw(lookup_ip_acl);
-}
-# use Mail::Address;
-
-# Returns the envelope sender address, or reconstructs it if there is
-# a good reason to believe the envelope address has been changed or forged,
-# as is common for some varieties of viruses. Returns best guess of the
-# sender address, or undef if it can not be determined.
-#
-sub unmangle_sender($$$) {
- my($sender) = shift; # rfc2821 envelope sender address
- my($from) = shift; # rfc2822 'From:' header, may include comment
- my($virusname_list) = shift; # list ref containing names of detected viruses
- # based on ideas from Furio Ercolessi, Mike Atkinson, Mark Martinec
-# my($localpart,$domain) = split_address($sender);
-# # extract the RFC2822 'from' address, ignoring phrase and comment
-# chomp($from);
-# { local($1,$2,$3,$4); # avoid Perl 5.8.0 & 5.8.2 bug, $1 gets tainted !
-# $from = (Mail::Address->parse($from))[0];
-# }
-# $from = $from->address if $from ne '';
-# # NOTE: rfc2822 allows multiple addresses in the From field!
- my($best_try_originator) = $sender;
- if ($best_try_originator ne '') {
- for my $vn (@$virusname_list) {
- my($result,$matching_key) =
- lookup(0,$vn,@{ca('viruses_that_fake_sender_maps')});
- if ($result) {
- do_log(2,"Virus %s matches %s, sender addr ignored",$vn,$matching_key);
- $best_try_originator = undef; last;
- }
- }
- }
- $best_try_originator;
}
# Given a dotted-quad IPv4 address try reverse DNS resolve, and then
@@ -5187,7 +5740,7 @@ sub first_received_from($) {
sub first_received_from($) {
my($entity) = shift;
my($first_received);
- if (defined($entity)) {
+ if ($entity) {
my($fields) = parse_received($entity->head->get('received', -1));
if (exists $fields->{'from'}) {
my($item, $v1, $v2, $v3, $comment) = @{$fields->{'from'}};
@@ -5213,7 +5766,7 @@ sub best_try_originator_ip($) {
::FFFF:0:0/96 !:: !::1 !FF00::/8 !FE80::/10 !FEC0::/10
::/0)) ) if !@publicnetworks_maps; # rfc3330, rfc3513
my($first_received_from_ip);
- if (defined($entity)) {
+ if ($entity) {
my(@received) = reverse $entity->head->get_all('received');
$#received = 5 if $#received > 5; # first six, chronologically
for my $r (@received) {
@@ -5238,15 +5791,24 @@ sub best_try_originator_ip($) {
# messages (macro %o), NOT to be used as address for sending notifications,
# as it can contain invalid address (but can be more informative).
#
-sub best_try_originator($$) {
- my($msginfo, $virusname_list) = @_;
- my($from) = $msginfo->orig_header_fields->{'from'};
- my($originator) = unmangle_sender($msginfo->sender,$from,$virusname_list);
- return ($originator, $originator) if defined $originator;
- my($first_received_from_ip) = best_try_originator_ip($msginfo->mime_entity);
- $originator = '?@' . ip_addr_to_name($first_received_from_ip)
- if $first_received_from_ip ne '';
- (undef, $originator);
+sub best_try_originator($) {
+ my($msginfo) = @_;
+ my($sender_contact,$sender_source);
+ $sender_contact = $sender_source = $msginfo->sender;
+ my($virusname_list) = $msginfo->virusnames;
+ for my $vn (!defined($virusname_list) ? () : @$virusname_list) {
+ my($result,$match) = lookup(0,$vn,@{ca('viruses_that_fake_sender_maps')});
+ if ($result) { # is a virus known to fake a sender address
+ do_log(2,"Virus %s matches %s, sender addr ignored",$vn,$match);
+ undef $sender_contact; undef $sender_source;
+ # at least try to get some info on sender source from his IP address
+ my($first_rcvd_from_ip) = best_try_originator_ip($msginfo->mime_entity);
+ $sender_source = '?@' . ip_addr_to_name($first_rcvd_from_ip)
+ if $first_rcvd_from_ip ne '';
+ last;
+ }
+ }
+ ($sender_contact, $sender_source);
}
1;
@@ -5259,7 +5821,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&consumed_bytes);
}
@@ -5348,7 +5910,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
@@ -5435,7 +5997,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter MIME::Parser::Filer); # subclass of MIME::Parser::Filer
}
# This package will be used by mime_decode().
@@ -5476,7 +6038,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&check_header_validity &check_for_banned_names);
}
@@ -5492,42 +6054,51 @@ sub check_header_validity($$) {
my($conn, $msginfo) = @_;
local($1,$2,$3); my($curr_head); my(@bad); my($minor_badh_category) = 0;
my(%field_head_counts);
- # minor category: 2: 8-bit char, 3: NUL/CR, 4: empty line, 5: long,
- # 6: syntax, 7: missing, 8: multiple
+ my($allowed_tests) = cr('allowed_header_tests');
+ my(%t) = !ref($allowed_tests) ? () : %$allowed_tests;
+ # minor category: 2: 8-bit char, 3: NUL/CR control, 4: empty line, 5: long,
+ # 6: syntax, 7: missing, 8: multiple
for my $next_head (@{$msginfo->orig_header}, "\n") {
if ($next_head =~ /^[ \t]/) { $curr_head .= $next_head } # folded
else { # new header
if (!defined($curr_head)) { # no previous complete header
} else {
+ my($field_name,$msg1,$msg2); my($pre,$mid,$post);
# obsolete rfc822 syntax allowed whitespace before colon
- my($field_name, $field_body) =
- $curr_head =~ /^([!-9;-\176]+)[ \t]*:(.*)\z/s
- ? ($1, $2) : (undef, $curr_head);
- my($msg1,$msg2);
+ $field_name = $1 if $curr_head =~ /^([!-9;-\176]+)[ \t]*:/s;
$field_head_counts{lc($field_name)}++ if defined $field_name;
- if (!defined($field_name) && $curr_head =~ /^()()(.*)\z/s) {
- $msg1 = "Invalid header field syntax";
- $minor_badh_category = max(6, $minor_badh_category);
- } elsif ($curr_head =~ /^(.*?)([\000\015])(.*)\z/s) {
+ if (!defined($field_name)) {
+ if ($t{'syntax'}) {
+ $msg1 = "Invalid header field syntax";
+ $pre = ''; $mid = ''; $post = $curr_head;
+ $minor_badh_category = max(6, $minor_badh_category);
+ }
+ } elsif ($t{'empty'} && $curr_head =~ /^(.*?)^([ \t]+)(?=\n|\z)/gms) {
+ $msg1 ="Improper folded header field made up entirely of whitespace";
+ $pre = $1; $mid = $2; $post = substr($curr_head,pos($curr_head));
+ # note: using //g and pos to avoid deep recursion in regexp
+ $minor_badh_category = max(4, $minor_badh_category);
+ } elsif ($t{'long'} &&
+ $curr_head =~ /^(.*?)^([^\n]{999,})(?=\n|\z)/gms) {
+ $msg1 = "Header line longer than 998 characters";
+ $pre = $1; $mid = $2; $post = substr($curr_head,pos($curr_head));
+ $minor_badh_category = max(5, $minor_badh_category);
+ } elsif ($t{'control'} && $curr_head =~ /^(.*?)([\000\015])/gs) {
$msg1 = "Improper use of control character";
+ $pre = $1; $mid = $2; $post = substr($curr_head,pos($curr_head));
$minor_badh_category = max(3, $minor_badh_category);
- } elsif ($curr_head =~ /^(.*?)([\200-\377])(.*)\z/s) {
+ } elsif ($t{'8bit'} && $curr_head =~ /^(.*?)([\200-\377])/gs) {
$msg1 = "Non-encoded 8-bit data";
+ $pre = $1; $mid = $2; $post = substr($curr_head,pos($curr_head));
$minor_badh_category = max(2, $minor_badh_category);
- } elsif ($curr_head =~ /^(.*?)([^\000-\377])(.*)\z/s) {
+ } elsif ($t{'8bit'} && $curr_head =~ /^(.*?)([^\000-\377])/gs) {
$msg1 = "Non-encoded Unicode character"; # should not happen
+ $pre = $1; $mid = $2; $post = substr($curr_head,pos($curr_head));
$minor_badh_category = max(2, $minor_badh_category);
- } elsif ($curr_head =~ /^(.*?)^([ \t]+)(\n.*)?\z/ms) {
- $msg1 ="Improper folded header field made up entirely of whitespace";
- $minor_badh_category = max(4, $minor_badh_category);
- } elsif ($curr_head =~ /^(.*?)^([^\n]{999,})(\n.*)?\z/ms) {
- $msg1 = "Header line longer than 998 characters";
- $minor_badh_category = max(5, $minor_badh_category);
}
if (defined $msg1) {
- my($pre, $mid, $post) = ($1, $2, $3);
- if (length($mid) > 20) { $mid = substr($mid,0,15) . "..." }
- if (length($post) > 20) { $post = substr($post,0,15) . "..." }
+ if (length($mid) > 20) { $mid = substr($mid, 0,15) . "..." }
+ if (length($post) > 20) { $post = substr($post,0,15) . "[...]" }
if (length($pre)-length($field_name)-2 > 50-length($post)) {
$pre = "$field_name: ..."
. substr($pre, length($pre) - (45-length($post)));
@@ -5548,10 +6119,10 @@ sub check_header_validity($$) {
for (qw(Date From Sender Reply-To To Cc Bcc Message-ID Subject
In-Reply-To References)) {
my($n) = $field_head_counts{lc($_)};
- if ($n < 1 && /^(?:Date|From)\z/i) {
+ if ($n < 1 && $t{'missing'} && /^(?:Date|From)\z/i) {
push(@bad, "Missing required header field: \"$_\"");
$minor_badh_category = max(7, $minor_badh_category);
- } elsif ($n > 1) {
+ } elsif ($n > 1 && $t{'multiple'}) {
if ($n == 2) {
push(@bad, "Duplicate header field: \"$_\"");
} else {
@@ -5745,7 +6316,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&mime_decode);
}
@@ -5755,7 +6326,7 @@ use MIME::Words;
use MIME::Words;
BEGIN {
- import Amavis::Conf qw(:platform c cr ca);
+ import Amavis::Conf qw(:platform c cr ca $MAXFILES);
import Amavis::Timing qw(section_time);
import Amavis::Util qw(snmp_count ll do_log);
import Amavis::Unpackers::NewFilename qw(consumed_bytes);
@@ -5882,6 +6453,7 @@ sub mime_decode($$$) {
# $parser->extract_nested_messages(0);
$parser->extract_nested_messages("NEST"); # parse embedded message/rfc822
$parser->extract_uuencode(1); # to enable or not to enable ???
+ $parser->max_parts($MAXFILES) if $MAXFILES > 0;
my($entity);
snmp_count('OpsDecByMimeParser');
if (ref($fileh)) { # assume open file handle
@@ -5894,14 +6466,17 @@ sub mime_decode($$$) {
local($1,$2,$3,$4); # avoid Perl 5.8.0 & 5.8.2 bug, $1 gets tainted !
$entity = $parser->parse_open("$tempdir/parts/$fileh");
}
-# my($mime_err) = $parser->last_error; # deprecated
my($mime_err) = $parser->results->errors;
if (defined $mime_err) {
$mime_err=~s/\s+\z//; $mime_err=~s/[ \t\r]*\n+/; /g; $mime_err=~s/\s+/ /g;
- $mime_err = substr($mime_err,0,250) . '...' if length($mime_err) > 250;
+ $mime_err = substr($mime_err,0,250) . '[...]' if length($mime_err) > 250;
do_log(1, "WARN: MIME::Parser %s", $mime_err) if $mime_err ne '';
- }
- mime_traverse($entity, $tempdir, $parent_obj, 0, '1');
+ } elsif (!defined($entity)) {
+ $mime_err = "Unable to parse, perhaps message contains too many parts";
+ do_log(1, "WARN: MIME::Parser %s", $mime_err);
+ $entity = '';
+ }
+ mime_traverse($entity, $tempdir, $parent_obj, 0, '1') if $entity;
section_time('mime_decode');
($entity, $mime_err);
}
@@ -5916,7 +6491,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&delivery_status_notification &delivery_short_report
&string_to_mime_entity &defanged_mime_entity
@@ -5924,8 +6499,8 @@ BEGIN {
}
BEGIN {
- import Amavis::Util qw(ll do_log am_id safe_encode q_encode sanitize_str
- xtext_encode xtext_decode);
+ import Amavis::Util qw(ll do_log safe_encode sanitize_str
+ xtext_decode ccat_split ccat_maj);
import Amavis::Timing qw(section_time);
import Amavis::Conf qw(:platform c cr ca);
import Amavis::Out::EditHeader qw(hdr);
@@ -5974,13 +6549,18 @@ sub string_to_mime_entity($$$$$) {
$mime_type =
$do_multipart ? 'multipart/mixed' : 'text/plain' if !defined($mime_type);
# make sure _our_ source line number is reported in case of failure
- eval {$entity = MIME::Entity->build(
- (defined $nxmh && $nxmh eq '' ? () # leave the MIME::Entity default
- : ('X-Mailer' => $nxmh) ), # X-Mailer hdr or undef
- Type => $mime_type,
- $do_multipart ? (Encoding => '7bit')
- : (Data => $m_body, Encoding => '-SUGGEST', Charset =>c('bdy_encoding')),
- ); 1} or do {chomp($@); die $@};
+ eval {
+ $entity = MIME::Entity->build(
+ (defined $nxmh && $nxmh eq '' ? () # leave the MIME::Entity default
+ : ('X-Mailer' => $nxmh) ), # X-Mailer hdr or undef
+ Type => $mime_type,
+ $do_multipart ? (Encoding => '7bit')
+ : (Data => $m_body, Encoding => '-SUGGEST', Charset=>c('bdy_encoding')),
+ ); 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die $eval_stat;
+ };
# Mail::Header::modify allows all-or-nothing control over automatic header
# folding by Mail::Header, which is too bad - we would prefer to have full
# control on folding of header fields that are explicitly inserted here,
@@ -6000,36 +6580,56 @@ sub string_to_mime_entity($$$$$) {
($fhead,$fbody) = ($1,$2) if $str =~ /^([^:]*):[ \t]*(.*)\z/s;
chomp($fbody);
do_log(5, "string_to_mime_entity %s: %s", $fhead,$fbody);
- # make sure _our_ source line number is reported in case of failure
- if (!eval { $head->replace($fhead,$fbody); 1 }) {
- chomp($@);
+ eval { # make sure _our_ source line number is reported on failure
+ $head->replace($fhead,$fbody); 1;
+ } or do {
+ $@ = "errno=$!" if $@ eq ''; chomp $@;
+ die $@ if $@ =~ /^timed out\b/; # resignal timeout
die sprintf("%s header field '%s: %s'",
($@ eq '' ? "invalid" : "$@, "), $fhead,$fbody);
- }
+ };
}
}
if ($do_multipart) {
- eval {$entity->attach(
- Type => 'text/plain', Data => $m_body,
- Encoding => '-SUGGEST', Charset => c('bdy_encoding'),
- ); 1} or do {chomp($@); die $@};
+ eval { # make sure _our_ source line number is reported on failure
+ $entity->attach(
+ Type => 'text/plain', Data => $m_body,
+ Encoding => '-SUGGEST', Charset => c('bdy_encoding')
+ ); 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die $eval_stat;
+ };
}
if (defined($msginfo) && $attach_orig_message) {
do_log(4, "string_to_mime_entity: attaching entire original message");
- eval {$entity->attach( # rfc2046
- Type => 'message/rfc822; x-spam-type=original',
- Encoding => '8bit', Path => $msginfo->mail_text_fn,
- Disposition => 'attachment', Filename => 'message',
- Description => 'Original message'); 1} or do {chomp($@); die $@};
+ eval { # make sure _our_ source line number is reported on failure
+ $entity->attach( # rfc2046
+ Type => 'message/rfc822; x-spam-type=original',
+ Encoding => ($msginfo->header_8bit || $msginfo->body_8bit) ?
+ '8bit' : '7bit',
+ Path => $msginfo->mail_text_fn,
+ Disposition => 'attachment', Filename => 'message',
+ Description => 'Original message'); 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die $eval_stat;
+ };
} elsif (defined($msginfo) && $attach_orig_headers) {
do_log(4, "string_to_mime_entity: attaching original message headers");
my($rp) = sprintf("Return-Path: %s\n", # fake a local delivery agent
qquote_rfc2821_local($msginfo->sender));
- eval {$entity->attach(
- Type => 'text/rfc822-headers', # rfc3462
- Encoding => '-SUGGEST', Data => [$rp, @{$msginfo->orig_header}],
- Disposition => 'inline', Filename => 'header',
- Description => 'Message headers'); 1} or do {chomp($@); die $@};
+ eval { # make sure _our_ source line number is reported on failure
+ $entity->attach(
+ Type => 'text/rfc822-headers', # rfc3462
+ Encoding => $msginfo->header_8bit ? '8bit' : '7bit',
+ Data => [$rp, @{$msginfo->orig_header}],
+ Disposition => 'inline', Filename => 'header',
+ Description => 'Message headers'); 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die $eval_stat;
+ };
}
$entity; # return the constructed MIME::Entity
}
@@ -6038,23 +6638,38 @@ sub string_to_mime_entity($$$$$) {
# rfc3462 (ex rfc1892), rfc3464 (ex rfc1894) and rfc3461 (ex rfc1891).
# Return a message object containing DSN if DSN is needed.
#
-sub delivery_status_notification($$$$$) {
- my($conn,$msginfo,$dsn_per_recip_capable,$all_rejected,$builtins_ref) = @_;
+sub delivery_status_notification($$$$) {
+ my($conn,$msginfo,$dsn_per_recip_capable,$builtins_ref) = @_;
local($1); my($notification); my($suppressed) = 0;
# my($dsn_time) = time; # time of dsn creation - now
my($dsn_time) = $msginfo->rx_time; # time of dsn creation - same as message
- # reception time for consistency and to be resilient to clock jumps
+ # use reception time for consistency and to be resilient to clock jumps
$dsn_time = time if !defined($dsn_time) || $dsn_time==0; # time: now
my($rfc2822_dsn_time) = rfc2822_timestamp($dsn_time);
my($txt_recip) = ''; # per-recipient part of dsn text according to rfc3464
my($sender) = $msginfo->sender;
my($dsn_passed_on) = $msginfo->dsn_passed_on; # NOTIFY=SUCCESS passed to MTA
my($delivery_method) = $msginfo->delivery_method;
-
+ my($per_recip_data) = $msginfo->per_recip_data;
+ my($all_rejected) = 0;
+ if (@$per_recip_data) {
+ $all_rejected = 1;
+ for my $r (@$per_recip_data) {
+ if ($r->recip_destiny != D_REJECT || $r->recip_smtp_response !~ /^5/)
+ { $all_rejected = 0; last }
+ }
+ }
my($spam_level) = $msginfo->spam_level;
my($os_fingerprint) = $msginfo->client_os_fingerprint;
- my($is_bulk) = $msginfo->orig_header_fields->{'precedence'};
- $is_bulk = $is_bulk=~/^[ \t]*(bulk|list|junk)\b/i ? $1 : undef;
+ my($is_bulk); # mail from a mailing list or some kind of a bounce
+ if ($msginfo->orig_header_fields->{'precedence'} =~
+ /^[ \t]*(bulk|list|junk)\b/si) { $is_bulk = $1 }
+ elsif (defined $msginfo->orig_header_fields->{'list-id'}) # rfc2919
+ { $is_bulk = 'List-Id:' . $msginfo->orig_header_fields->{'list-id'} }
+ elsif ($msginfo->rfc2822_from =~ # won't match if multiple addr., who cares
+ /^ (?: [^@]+-(request|owner|relay|bounces) | owner-[^@]+ |
+ postmaster | mailer-daemon | mailer | uucp ) ( @ | \z )/xsi)
+ { $is_bulk = 'From:' . $msginfo->rfc2822_from }
my($dsn_cutoff_level, $dsn_cutoff_level_bysender);
my($cutoff_level_maps) = ca('spam_dsn_cutoff_level_maps');
my($cutoff_level_bysender_maps) = ca('spam_dsn_cutoff_level_bysender_maps');
@@ -6062,12 +6677,12 @@ sub delivery_status_notification($$$$$)
lookup(0,$sender,@$cutoff_level_bysender_maps) if $sender ne '';
my($any_succ,$any_fail,$any_delayed) = (0,0,0);
- for my $r (@{$msginfo->per_recip_data}) { # prepare per-recip fields first
+ for my $r (@$per_recip_data) { # prepare per-recip fields first
my($recip) = $r->recip_addr;
my($smtp_resp) = $r->recip_smtp_response;
my($recip_done) = $r->recip_done; # 2=relayed to MTA, 1=faked deliv/quarant
- my($ccat_name) =
- $msginfo->setting_by_contents_category(\%ccat_display_names);
+ my($ccat_name) = $r->setting_by_contents_category(\%ccat_display_names);
+ $ccat_name = "NonBlocking:$ccat_name" if !defined($r->blocking_ccat);
my($boost) = $r->recip_score_boost;
if (!$recip_done) {
if ($delivery_method eq '') { # e.g. milter
@@ -6105,10 +6720,10 @@ sub delivery_status_notification($$$$$)
$r->setting_by_contents_category(cr('warnsender_by_ccat'));
ll(5) && do_log(5,
"dsn: %s %s %s <%s> -> <%s>: on_succ=%d, on_dly=%d, on_fail=%d,".
- " never=%d, warn_sender=%s, DSN_passed_on=%s",
+ " never=%d, warn_sender=%s, DSN_passed_on=%s, mta_resp: \"%s\"",
$remote_or_local, $smtp_resp_code, $ccat_name, $sender, $recip,
$notify_on_success, $notify_on_delay, $notify_on_failure,
- $notify_never, $warn_sender, $dsn_passed_on);
+ $notify_never, $warn_sender, $dsn_passed_on, $smtp_resp);
# clearly log common cases to facilitate troubleshooting;
# first look for some standard reasons for not sending a DSN
@@ -6146,7 +6761,7 @@ sub delivery_status_notification($$$$$)
$suppressed = 1;
do_log(4, "DSN: FILTER %s %s %s, destiny=DISCARD: <%s> -> <%s>",
$remote_or_local,$smtp_resp_code,$ccat_name,$sender,$recip);
- } elsif ($msginfo->sender_contact eq '') { # faked sender most likely
+ } elsif (!defined($msginfo->sender_contact)) { # faked sender most likely
$suppressed = 1;
do_log(3, "DSN: FILTER %s %s, <%s> (faked?) -> <%s>",
$smtp_resp_code,$ccat_name,$sender,$recip);
@@ -6170,12 +6785,13 @@ sub delivery_status_notification($$$$$)
"by-recipient cutoff level %s, <%s> -> <%s>",
$smtp_resp_code, $ccat_name,
$spam_level+$boost, $dsn_cutoff_level, $sender, $recip);
- } elsif ($is_bulk && $r->main_contents_category > CC_CLEAN) {
+ } elsif ($is_bulk && ccat_maj($r->contents_category) > CC_CLEAN) {
$suppressed = 1;
do_log(3, "DSN: FILTER %s %s, suppressed, bulk mail (%s), <%s> -> <%s>",
$smtp_resp_code,$ccat_name,$is_bulk,$sender,$recip);
} elsif ($os_fingerprint =~ /^Windows\b/ && # hard-coded limits!
- $spam_level+$boost >= ($os_fingerprint=~/^Windows XP/ ? 5 : 8)) {
+ $spam_level+$boost >=
+ ($os_fingerprint=~/^Windows XP(?![^(]*\b2000 SP)/ ? 5 : 8)) {
$suppressed = 1;
$os_fingerprint =~ /^(\S+\s+\S+)/;
do_log(3, "DSN: FILTER %s %s, suppressed for mail from %s ".
@@ -6253,6 +6869,8 @@ sub delivery_status_notification($$$$$)
$smtp_resp =~ s/\n(?![ \t])/\n /gs;
$txt_recip .= "Diagnostic-Code: smtp; $smtp_resp\n";
$txt_recip .= "Last-Attempt-Date: $rfc2822_dsn_time\n";
+ $txt_recip .= sprintf("Final-Log-ID: %s/%s\n",
+ $msginfo->log_id, $msginfo->mail_id);
do_log(2, "DSN: NOTIFICATION: Action:%s, %s %s %s, <%s> -> <%s>",
$action,
$recip_done==2 && $action ne 'delayed' ? 'RELAYED' : 'LOCAL',
@@ -6295,10 +6913,10 @@ sub delivery_status_notification($$$$$)
# DSN will reach that person.
# Override header fields from the template:
eval { $head->replace('From',expand_variables($hdrfrom_sender)); 1 }
- or do { chomp($@); die $@ };
- eval { $head->replace('To', $to_hdr); 1 } or do { chomp($@); die $@ };
+ or do { chomp $@; die $@ };
+ eval { $head->replace('To', $to_hdr); 1 } or do { chomp $@; die $@ };
eval { $head->replace('Date', $rfc2822_dsn_time); 1 }
- or do { chomp($@); die $@ };
+ or do { chomp $@; die $@ };
my($txt_msg) = ''; # per-message part of dsn text according to rfc3464
my($from_mta) = $conn->smtp_helo; my($client_ip) = $conn->client_ip;
@@ -6312,22 +6930,28 @@ sub delivery_status_notification($$$$$)
if $from_mta ne '';
$txt_msg .= "Arrival-Date: " . rfc2822_timestamp($msginfo->rx_time) . "\n";
- # make sure _our_ source line number is reported in case of failure
- eval { $dsn_entity->add_part(
- MIME::Entity->build(Top => 0,
- Type => 'message/delivery-status', Encoding => '7bit',
- Description => $any_fail ? 'Delivery error report' :
- $any_delayed ? 'Delivery delay report' : 'Delivery report',
- Disposition => 'inline', Filename => 'dsn_status',
- Data => $txt_msg.$txt_recip),
- 1); # insert as second mime part (at offset 1)
- 1} or do {chomp($@); die $@};
+ eval { # make sure _our_ source line number is reported in case of failure
+ $dsn_entity->add_part(
+ MIME::Entity->build(Top => 0,
+ Type => 'message/delivery-status', Encoding => '7bit',
+ Description => $any_fail ? 'Delivery error report' :
+ $any_delayed ? 'Delivery delay report' : 'Delivery report',
+ Disposition => 'inline', Filename => 'dsn_status',
+ Data => $txt_msg.$txt_recip),
+ 1); # insert as a second mime part (at offset 1)
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die $eval_stat;
+ };
$notification = Amavis::In::Message->new;
$notification->rx_time($dsn_time);
+ $notification->log_id($msginfo->log_id);
# $notification->body_type('7BIT');
$notification->mail_text($dsn_entity);
$notification->delivery_method(c('notify_method'));
$notification->sender(''); # DSN envelope sender must be empty!
+ $notification->sender_smtp('<>');
$notification->auth_submitter('<>');
$notification->auth_user(c('amavis_auth_user'));
$notification->auth_pass(c('amavis_auth_pass'));
@@ -6359,7 +6983,7 @@ sub delivery_short_report($) {
} else {
push(@failed_recips, $qrecip_addr);
push(@failed_recips_full, sprintf("%s:%s\n %s", $qrecip_addr,
- (!defined($remote_mta)||$remote_mta eq '' ? '' : " $remote_mta said:"),
+ (!defined($remote_mta)||$remote_mta eq '' ?'' :" [$remote_mta] said:"),
$smtp_resp));
}
}
@@ -6378,13 +7002,16 @@ sub defanged_mime_entity($$$) {
$_ = safe_encode(c('bdy_encoding'), $_)
for (ref $first_part ? @$first_part : $first_part);
my($nxmh) = c('notify_xmailer_header');
- # make sure _our_ source line number is reported in case of failure
- eval {$new_entity = MIME::Entity->build(
- Type => 'multipart/mixed',
- (defined $nxmh && $nxmh eq '' ? () # leave the MIME::Entity default
- : ('X-Mailer' => $nxmh) ), # X-Mailer hdr or undef
- ); 1} or do {chomp($@); die $@};
-
+ eval { # make sure _our_ source line number is reported in case of failure
+ $new_entity = MIME::Entity->build(
+ Type => 'multipart/mixed',
+ (defined $nxmh && $nxmh eq '' ? () # leave the MIME::Entity default
+ : ('X-Mailer' => $nxmh) ) ); # X-Mailer hdr or undef
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die $eval_stat;
+ };
# reinserting some of the original header fields to a new header, sanitized
my($hdr_edits) = $msginfo->header_edits;
if (!$hdr_edits) {
@@ -6423,7 +7050,7 @@ sub defanged_mime_entity($$$) {
# needs to be inserted directly into new header so that it can be
# subjected to header edits, like inserting ***UNCHECKED***
eval { $new_entity->head->add($field_name,$field_body); 1 }
- or do {chomp($@); die $@};
+ or do {chomp $@; die $@};
} else {
$hdr_edits->append_header($field_name,$field_body,2);
}
@@ -6433,15 +7060,28 @@ sub defanged_mime_entity($$$) {
$curr_head = $next_head;
}
}
- eval {$new_entity->attach(
- Type => 'text/plain', Encoding => '-SUGGEST', Charset => c('bdy_encoding'),
- Data => $first_part); 1} or do {chomp($@); die $@};
- eval {$new_entity->attach( # rfc2046
- Type => 'message/rfc822; x-spam-type=original',
- Encoding => '8bit', Path => $msginfo->mail_text_fn,
- Description => 'Original message',
- Filename => 'message', Disposition => 'attachment'); 1}
- or do {chomp($@); die $@};
+ eval {
+ $new_entity->attach(
+ Type => 'text/plain',
+ Encoding => '-SUGGEST', Charset => c('bdy_encoding'),
+ Data => $first_part);
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die $eval_stat;
+ };
+ eval {
+ $new_entity->attach( # rfc2046
+ Type => 'message/rfc822; x-spam-type=original',
+ Encoding =>($msginfo->header_8bit || $msginfo->body_8bit) ?'8bit':'7bit',
+ Path => $msginfo->mail_text_fn,
+ Description => 'Original message',
+ Filename => 'message', Disposition => 'attachment');
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ die $eval_stat;
+ };
$new_entity;
}
@@ -6449,13 +7089,21 @@ sub defanged_mime_entity($$$) {
# Expects $msginfo->mail_text to be a file handle (not Mime::Entity object),
# leaves it positioned at the beginning of a mail body.
# If given a BSMTP file, expects that it contains a single message only.
-sub msg_from_quarantine($$) {
- my($conn,$msginfo) = @_;
+sub msg_from_quarantine($$$) {
+ my($conn,$msginfo,$request_type) = @_;
my($fh) = $msginfo->mail_text;
my($fname) = $msginfo->mail_text_fn;
my($quarantine_id) = $msginfo->mail_id;
- my($release_m) = c('release_method');
- $msginfo->delivery_method($release_m ne '' ? $release_m :c('notify_method'));
+ my($release_m);
+ if ($request_type eq 'requeue') {
+ $release_m = c('requeue_method');
+ $release_m ne '' or die "requeue_method is unspecified";
+ } else {
+ $release_m = c('release_method');
+ $release_m = c('notify_method') if $release_m eq '';
+ $release_m ne '' or die "release_method and notify_method are unspecified";
+ }
+ $msginfo->delivery_method($release_m);
$msginfo->auth_submitter('<>');
$msginfo->auth_user(c('amavis_auth_user'));
$msginfo->auth_pass(c('amavis_auth_pass'));
@@ -6504,9 +7152,9 @@ sub msg_from_quarantine($$) {
$curr_head = $next_head;
}
}
- do_log(0,"Quarantined message release: %s %s -> %s", $quarantine_id,
- qquote_rfc2821_local($sender),
- join(',', qquote_rfc2821_local(@recips)) );
+ my($sender_smtp) = qquote_rfc2821_local($sender);
+ do_log(0,"Quarantined message %s: %s %s -> %s", $request_type,$quarantine_id,
+ $sender_smtp, join(',', qquote_rfc2821_local(@recips)) );
my(@m);
if (!defined $qid) { push(@m, 'missing X-Quarantine-ID') }
elsif ($qid ne $quarantine_id) {
@@ -6515,12 +7163,12 @@ sub msg_from_quarantine($$) {
}
push(@m, 'missing '.($bsmtp?'MAIL FROM':'X-Envelope-From or Return-Path'))
if !defined $sender;
- push(@m, 'missing '.($bsmtp?'RCPT TO' :'X-Envelope-To'))
- if !@recips;
- if (!defined($msginfo->sender)) { $msginfo->sender($sender) }
- else { # sender specified in the request, overrides stored info
+ push(@m, 'missing '.($bsmtp?'RCPT TO' :'X-Envelope-To')) if !@recips;
+ if (!defined($msginfo->sender)) {
+ $msginfo->sender($sender); $msginfo->sender_smtp($sender_smtp);
+ } else { # sender specified in the request, overrides stored info
push(@m, sprintf("overriding sender %s by %s",
- qquote_rfc2821_local($sender, $msginfo->sender) ));
+ $sender_smtp, $msginfo->sender_smtp));
}
if (!defined($msginfo->per_recip_data)) { $msginfo->recips(\@recips) }
else { # recipients specified in the request, overrides stored info
@@ -6528,7 +7176,8 @@ sub msg_from_quarantine($$) {
join(',', qquote_rfc2821_local(@recips)),
join(',', qquote_rfc2821_local(@{$msginfo->recips})) ));
}
- do_log(0, "Quarantine release %s: %s", $quarantine_id, join("; ", at m)) if @m;
+ do_log(0, "Quarantine %s %s: %s",
+ $request_type, $quarantine_id, join("; ", at m)) if @m;
my($hdr_edits) = Amavis::Out::EditHeader->new;
# TODO: should only delete header fields inserted by us during quarantining!
for my $h (qw(Return-Path Delivered-To X-Quarantine-ID
@@ -6578,18 +7227,18 @@ use re 'taint';
use re 'taint';
BEGIN {
- import Amavis::Util qw(ll do_log);
+ import Amavis::Util qw(ll do_log freeze thaw);
}
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.0721';
+ $VERSION = '2.0911';
@ISA = qw(Exporter);
}
# simple local memory-based cache
sub new { # called by each child process
- my($class) = @_;
+ my($class,$keysize) = @_;
do_log(5,"BerkeleyDB-based Amavis::Cache not available, ".
"using memory-based local cache");
bless {}, $class;
@@ -6597,45 +7246,31 @@ sub get { my($self,$key) = @_; thaw($sel
sub get { my($self,$key) = @_; thaw($self->{$key}) }
sub set { my($self,$key,$obj) = @_; $self->{$key} = freeze($obj) }
-# protect % and ~, as well as NUL and \200 for good measure
-sub encode($) {
- my($str) = @_; local($1);
- $str =~ s/([%~\000\200])/sprintf("%%%02X",ord($1))/egs;
- $str;
-}
-
-# simple Storable::freeze lookalike
-sub freeze($); # prototype
-sub freeze($) {
- my($obj) = @_; my($ty) = ref($obj);
- if (!defined($obj)) { 'U' }
- elsif (!$ty) { join('~', '', encode($obj)) } # string
- elsif ($ty eq 'SCALAR') { join('~', 'S', encode(freeze($$obj))) }
- elsif ($ty eq 'REF') { join('~', 'R', encode(freeze($$obj))) }
- elsif ($ty eq 'ARRAY') { join('~', 'A', map {encode(freeze($_))} @$obj) }
- elsif ($ty eq 'HASH') {
- join('~','H',map {(encode($_),encode(freeze($obj->{$_})))} sort keys %$obj)
- } else { die "Can't freeze object type $ty" }
-}
-
-# simple Storable::thaw lookalike
-sub thaw($); # prototype
-sub thaw($) {
- my($str) = @_;
- return undef if !defined $str;
- my($ty, at val) = split(/~/,$str,-1);
- for (@val) { s/%([0-9a-fA-F]{2})/pack("C",hex($1))/eg }
- if ($ty eq 'U') { undef }
- elsif ($ty eq '') { $val[0] }
- elsif ($ty eq 'S') { my($obj)=thaw($val[0]); \$obj }
- elsif ($ty eq 'R') { my($obj)=thaw($val[0]); \$obj }
- elsif ($ty eq 'A') { [map {thaw($_)} @val] }
- elsif ($ty eq 'H') {
- my($hr) = {};
- while (@val) { my($k) = shift @val; $hr->{$k} = thaw(shift @val) }
- $hr;
- } else { die "Can't thaw object type $ty" }
-}
+1;
+
+#
+package Amavis::Custom;
+# MAIL PROCESSING SEQUENCE:
+# child process initialization
+#*custom hook: new()
+# loop for each mail:
+# receive mail
+# mail checking and collecting results
+# *custom hook: checks() - may inspect or modify checking results
+# deciding mail fate (lookup on *_lovers, thresholds, ...)
+# quarantining
+# sending notifications (to admin and recip)
+# *custom hook: before_send() - may send other notif., quarantine, modify mail
+# forwarding (unless blocked)
+# sending delivery status notification (if needed)
+# issue main log entry, manage statistics (timing, counters, nanny)
+# *custom hook: mail_done() - may inspect results
+# endloop after $max_requests or earlier
+
+sub new { my($class,$conn,$msginfo) = @_; undef }
+sub checks { my($self,$conn,$msginfo) = @_; undef }
+sub before_send { my($self,$conn,$msginfo) = @_; undef }
+sub mail_done { my($self,$conn,$msginfo) = @_; undef }
1;
@@ -6645,25 +7280,29 @@ use strict;
use strict;
use re 'taint';
-use Errno qw(ENOENT EACCES);
+use Errno qw(ENOENT EACCES EAGAIN ESRCH);
use POSIX qw(locale_h);
use IO::File qw(O_RDONLY O_WRONLY O_RDWR O_APPEND O_CREAT O_EXCL);
use Time::HiRes ();
-# body digest for caching, either SHA1 or MD5
-#use Digest::SHA1;
+# body digest for caching, either MD5 or SHA1 (or perhaps SHA256)
+#use Digest::SHA;
use Digest::MD5;
use Net::Server 0.87; # need Net::Server::PreForkSimple::done
-use Net::Server::PreForkSimple;
BEGIN {
import Amavis::Conf qw(:platform :sa :confvars c cr ca);
import Amavis::Util qw(untaint min max unique ll do_log sanitize_str
debug_oneshot am_id add_entropy generate_mail_id
- prolong_timer waiting_for_client
+ prolong_timer waiting_for_client
switch_to_my_time switch_to_client_time
snmp_counters_init snmp_count dynamic_destination
- ccat_maj ccat_min cmp_ccat cmp_ccat_maj cloexec xtext_encode);
- import Amavis::Log qw(open_log close_log);
+ run_command exit_status_str proc_status_ok
+ collect_results
+ ccat_split ccat_maj cmp_ccat cmp_ccat_maj
+ setting_by_given_contents_category_all
+ setting_by_given_contents_category
+ cloexec orcpt_encode);
+ import Amavis::Log qw(open_log close_log collect_log_stats);
import Amavis::Timing qw(section_time get_time_so_far);
import Amavis::rfc2821_2822_Tools;
import Amavis::Lookup qw(lookup);
@@ -6682,12 +7321,6 @@ BEGIN {
import Amavis::In::Message;
}
-# Make it a subclass of Net::Server::PreForkSimple
-# to override method &process_request (and others if desired)
-use vars qw(@ISA);
-# @ISA = qw(Net::Server);
- at ISA = qw(Net::Server::PreForkSimple);
-
add_entropy(Time::HiRes::gettimeofday, $$, $], @INC, %ENV);
delete @ENV{'IFS', 'CDPATH', 'ENV', 'BASH_ENV'};
@@ -6712,7 +7345,7 @@ use vars qw($child_invocation_count $chi
# this often runs in sync with $child_invocation_count,
# but with SMTP or LMTP input there may be more than one
# message passed during a single SMTP session
-use vars qw(@config_files);
+use vars qw(@config_files); # configuration files provided by -c or defaulted
use vars qw($CONN $MSGINFO);
use vars qw($av_output @virusname @detecting_scanners
$banned_filename_any $banned_filename_all @bad_headers);
@@ -6726,6 +7359,17 @@ use vars qw($ldap_connection);
use vars qw($ldap_connection); # Amavis::LDAP::Connection object
use vars qw($ldap_policy); # Amavis::Lookup::LDAP object
+use vars qw(@ISA);
+
+sub new {
+ my($class) = shift;
+ # make Amavis a subclass of Net::Server::whatever
+ @ISA = defined($min_servers) ? 'Net::Server::PreFork'
+ : 'Net::Server::PreForkSimple';
+# $class->SUPER::new(@_); # available since Net::Server 0.91
+ bless { server => $_[0] }, $class; # works with all versions
+}
+
# implements macros: T, and SA lookalikes: TESTS, TESTSSCORES
sub macro_tests {
my($name,$sep) = @_;
@@ -6733,30 +7377,6 @@ sub macro_tests {
if (@s > 50) { $#s = 50-1; push(@s,"...") } # sanity limit
@s = map {my($tn,$ts)=split(/=/); $tn} @s if $name eq 'TESTS';
if ($name eq 'T' || !defined($sep)) { \@s } else { join($sep, at s) }
-};
-
-# implements macros: 2, and SA lookalikes: YESNO, YESNOCAPS
-sub macro_yesno {
- my($name) = @_;
- my($any); my($tag2_level);
- my($spam_level) = $MSGINFO->spam_level;
- for my $r (@{$MSGINFO->per_recip_data}) {
- my($recip) = $r->recip_addr;
- my($blacklisted) = $r->recip_blacklisted_sender;
- my($whitelisted) = $r->recip_whitelisted_sender;
- my($boost) = $r->recip_score_boost;
- my($is_local,$tag2_level,$bypassed);
- $is_local = lookup(0,$recip, @{ca('local_domains_maps')});
- $tag2_level = lookup(0,$recip, @{ca('spam_tag2_level_maps')});
- $bypassed = lookup(0,$recip, @{ca('bypass_spam_checks_maps')});
- my($do_tag2) = $is_local && !$bypassed && !$whitelisted &&
- ( $blacklisted ||
- (defined $tag2_level && $spam_level+$boost >= $tag2_level) );
- $any = 1 if $do_tag2; # above tag2 level for any recipient?
- }
- if ($name eq 'YESNOCAPS') { $any ? 'YES' : 'NO' }
- elsif ($name eq 'YESNO') { $any ? 'Yes' : 'No' }
- else { $any ? '1' : '0' }
};
# implements macros: c, and SA lookalikes: SCORE(pad), STARS(*)
@@ -6803,14 +7423,19 @@ sub macro_score {
# implements macros: header_field
# Can only obtain a header field stored in $msginfo->orig_header_fields,
# and only its first occurrence; currently these are: from, to, cc, sender,
-# subject, received, message-id, resent-message-id, precedence, user-agent,
-# x-mailer, dkim-signature, domainkey-signature, authentication-results
+# subject, received, message-id, resent-message-id, in-reply-to, references.
+# precedence, list-id, user-agent, x-mailer,
+# dkim-signature, domainkey-signature, authentication-results
#
sub macro_header_field {
my($name,$header_field_name) = @_;
local($_) = $MSGINFO->orig_header_fields->{lc($header_field_name)};
if (defined $_) { # unfold, trim, protect CR, LF, \000 and \200
chomp; s/\n([ \t])/$1/gs; s/^[ \t]+//; s/[ \t\n]+\z//;
+ if ($header_field_name =~
+ /^(?:Message-ID|Resent-Message-ID|In-Reply-To|References)\z/i) {
+ $_ = join(' ',parse_message_id($_)) if $_ ne ''; # strip CFWS
+ }
s{([\r\n\000\200])}{sprintf("\\%03O",ord($1))}eg;
};
$_;
@@ -6855,13 +7480,8 @@ sub init_builtin_macros() {
y => sub {sprintf("%.0f", 1000*get_time_so_far())}, # elapsed time in ms
h => sub {c('myhostname')}, # fqdn name of this host
HOSTNAME => sub {c('myhostname')},
- l => sub {my($ip) = $MSGINFO->client_addr; my($val);
- $val = $ip ne '' ? $MSGINFO->client_addr_mynets
- : lookup(0,$MSGINFO->sender_source,
- @{ca('local_domains_maps')});
- $val ? 1 : undef}, # sender client IP (if known) from @mynetworks
- # (if IP is known), or sender domain is local
- s => sub {qquote_rfc2821_local($MSGINFO->sender)}, # orig.env. sender in <>
+ l => sub {c('originating') ? 1 : undef}, # client local (mynets or roaming)
+ s => sub {$MSGINFO->sender_smtp}, # orig. unmodified env. sender addr in <>
S => sub { # unmangled sender or sender address to be notified, or empty...
sanitize_str($MSGINFO->sender_contact) }, # ..if sender unknown
o => sub { # best attempt at determining true sender (origin) of the virus,
@@ -6875,18 +7495,65 @@ sub init_builtin_macros() {
r => sub {macro_header_field('header','Resent-Message-ID')},
j => sub {macro_header_field('header','Subject')},
'x-mailer' => sub {macro_header_field('header','X-Mailer')},
- 'useragent'=> sub { my($h) = macro_header_field('header','User-Agent');
- return 'User-Agent: '.$h if defined $h;
- $h = macro_header_field('header','X-Mailer');
- return 'X-Mailer: '.$h if defined $h;
- undef },
header_field => \¯o_header_field,
- ccat_maj => sub {my($maj,$min)=$MSGINFO->main_contents_category;
- $maj ne '' ? "$maj" : "0" },
- ccat_min => sub {my($maj,$min)=$MSGINFO->main_contents_category;
- $min ne '' ? "$min" : "0" },
- ccat_name=> sub {$MSGINFO->setting_by_contents_category(
- \%ccat_display_names)},
+ useragent => # argument: 'name' or 'body', or empty to return entire field
+ sub { my($macro_name,$which_part) = @_; my($head,$body);
+ $body = macro_header_field('header', $head='User-Agent');
+ $body = macro_header_field('header', $head='X-Mailer')
+ if !defined($body);
+ !defined($body) ? undef
+ : lc($which_part) eq 'name' ? $head
+ : lc($which_part) eq 'body' ? $body : "$head: $body";
+ },
+ ccat =>
+ sub {
+ my($name,$attr,$which) = @_;
+ $attr = lc($attr); # name | major | minor | <empty>
+ # | is_blocking | is_nonblocking
+ # | is_blocked_by_nonmain
+ $which = lc($which); # main | blocking | auto
+ my($result) = ''; my($blocking_ccat) = $MSGINFO->blocking_ccat;
+ if ($attr eq 'is_blocking') {
+ $result = defined($blocking_ccat) ? '1' : '';
+ } elsif ($attr eq 'is_nonblocking') {
+ $result = !defined($blocking_ccat) ? '1' : '';
+ } elsif ($attr eq 'is_blocked_by_nonmain') {
+ if (defined($blocking_ccat)) {
+ my($aref) = $MSGINFO->contents_category;
+ $result = '1' if ref($aref) && @$aref > 0
+ && $blocking_ccat ne $aref->[0];
+ }
+ } elsif ($attr eq 'name') {
+ $result =
+ $which eq 'main' ?
+ $MSGINFO->setting_by_main_contents_category(\%ccat_display_names)
+ : $which eq 'blocking' ?
+ $MSGINFO->setting_by_blocking_contents_category(
+ \%ccat_display_names)
+ : $MSGINFO->setting_by_contents_category( \%ccat_display_names);
+ } else { # attr = major, minor, or anything else returns a pair
+ my($maj,$min) = ccat_split(
+ ($which eq 'blocking' ||
+ $which ne 'main' && defined $blocking_ccat)
+ ? $blocking_ccat : $MSGINFO->contents_category);
+ $result = $attr eq 'major' ? $maj
+ : $attr eq 'minor' ? sprintf("%d",$min)
+ : sprintf("(%d,%d)",$maj,$min);
+ }
+ $result;
+ },
+ ccat_maj => # deprecated, use [:ccat|major]
+ sub { my($blocking_ccat) = $MSGINFO->blocking_ccat;
+ (ccat_split(defined $blocking_ccat ? $blocking_ccat
+ : $MSGINFO->contents_category))[0];
+ },
+ ccat_min => # deprecated, use [:ccat|minor]
+ sub { my($blocking_ccat) = $MSGINFO->blocking_ccat;
+ (ccat_split(defined $blocking_ccat ? $blocking_ccat
+ : $MSGINFO->contents_category))[1];
+ },
+ ccat_name => # deprecated, use [:ccat|name]
+ sub { $MSGINFO->setting_by_contents_category(\%ccat_display_names) },
dsn_notify => sub {
return 'NEVER' if $MSGINFO->sender eq '';
my(%merged);
@@ -6897,14 +7564,17 @@ sub init_builtin_macros() {
uc(join(',', sort keys %merged));
},
b => sub {$MSGINFO->body_digest}, # original message body digest
- n => sub {am_id()}, # amavis internal message id (for log entries)
+ n => sub {$MSGINFO->log_id}, # amavis internal task id (in log and nanny)
i => sub {$MSGINFO->mail_id}, # long-term unique mail id on this system
+ LOGID => sub {$MSGINFO->log_id}, # synonym for %n (no equivalent in SA)
+ MAILID => sub {$MSGINFO->mail_id}, # synonym for %i (no equivalent in SA)
q => sub {my($q) = $MSGINFO->quarantined_to;
!defined($q) ? undef :
[map { my($m)=$_; $m=~s{^\Q$QUARANTINEDIR\E/}{}; $m } @$q];
}, # list of quarantine mailboxes
v => sub {[split(/[ \t]*\r?\n/,$av_output)]}, # anti-virus scanner output
- V => sub {unique(@virusname)}, # unique virus names
+ V => sub {my($vn) = $MSGINFO->virusnames; # unique virus names
+ !defined($vn) ? undef : unique(@$vn)},
F => sub { # list of banned file names
my($b) = unique(map { @{$_->banned_parts} }
grep { defined($_->banned_parts) }
@@ -6917,7 +7587,7 @@ sub init_builtin_macros() {
H => sub {[map {my $h=$_; chomp($h); $h} @{$MSGINFO->orig_header}]},
A => sub {[split(/\r?\n/, $MSGINFO->spam_summary)]}, # SA report text
SUMMARY => sub {$MSGINFO->spam_summary},
- REPORT => sub {$MSGINFO->spam_report},
+ REPORT => sub {sanitize_str($MSGINFO->spam_report,1)}, #contains any octet
TESTSSCORES => \¯o_tests, # tests triggered, with scores
TESTS => \¯o_tests, # tests triggered, without scores
z => sub {$MSGINFO->msg_size}, # mail size
@@ -6940,7 +7610,12 @@ sub init_builtin_macros() {
sub {my($ip) = $CONN->client_ip; $ip ne '' ? "[$ip]" : 'localhost'},
# VERSION => Mail::SpamAssassin->Version, # SA version
# SUBVERSION => $Mail::SpamAssassin::SUB_VERSION, # SA sub-version/revision
- AUTOLEARN => sub {$MSGINFO->autolearn_status},
+ AUTOLEARN => sub {$MSGINFO->supplementary_info('AUTOLEARN')},
+ supplementary_info =>
+ sub { my($name,$key,$fmt)=@_;
+ my($info) = $MSGINFO->supplementary_info($key);
+ $info eq '' ? '' : $fmt eq '' ? $info : sprintf($fmt,$info);
+ },
REQD => sub { my($tag2_level);
for (@{$MSGINFO->per_recip_data}) { # get minimal tag2_level
my($tag2_l) = lookup(0,$_->recip_addr,
@@ -6950,29 +7625,21 @@ sub init_builtin_macros() {
}
!defined($tag2_level) ? '-' : 0+sprintf("%.3f",$tag2_level);
},
- k => sub { my($name) = @_; my($kill_level);
- scalar(grep # any recipient declared the message be killed ?
- { !$_->recip_whitelisted_sender &&
- ($_->recip_blacklisted_sender ||
- ($kill_level = lookup(0,$_->recip_addr,
- @{ca('spam_kill_level_maps')}),
- defined $kill_level &&
- $MSGINFO->spam_level + $_->recip_score_boost
- >= $kill_level) )
- } @{$MSGINFO->per_recip_data}) },
- '1'=> sub { my($name) = @_; my($tag_level);
- scalar(grep # above tag level for any recipient?
- { !$_->recip_whitelisted_sender &&
- ($_->recip_blacklisted_sender ||
- ($tag_level=lookup(0,$_->recip_addr,
- @{ca('spam_tag_level_maps')}),
- defined $tag_level && $tag_level ne '' &&
- $MSGINFO->spam_level + $_->recip_score_boost
- >= $tag_level) )
- } @{$MSGINFO->per_recip_data}) },
- '2' => \¯o_yesno,
- YESNO => \¯o_yesno,
- YESNOCAPS => \¯o_yesno,
+ '1'=> sub { # above tag level and not bypassed for any recipient?
+ (grep { $_->is_in_contents_category(CC_CLEAN,1) }
+ @{$MSGINFO->per_recip_data}) ? 'Y' : '0' },
+ '2'=> sub { # above tag2 level and not bypassed for any recipient?
+ (grep { $_->is_in_contents_category(CC_SPAMMY) }
+ @{$MSGINFO->per_recip_data}) ? 'Y' : '0' },
+ YESNO => sub { # like %2, but gives: Yes/No
+ (grep { $_->is_in_contents_category(CC_SPAMMY) }
+ @{$MSGINFO->per_recip_data}) ? 'Yes' : 'No' },
+ YESNOCAPS => sub { # like %2, but gives: YES/NO
+ (grep { $_->is_in_contents_category(CC_SPAMMY) }
+ @{$MSGINFO->per_recip_data}) ? 'YES' : 'NO' },
+ 'k'=> sub { # above kill level and not bypassed for any recipient?
+ (grep { $_->is_in_contents_category(CC_SPAM) }
+ @{$MSGINFO->per_recip_data}) ? 'Y' : '0' },
score_boost => sub {0+sprintf("%.3f",min(map {$_->recip_score_boost}
@{$MSGINFO->per_recip_data}))},
c => sub {macro_score(undef, at _)}, # info on all recipients
@@ -6994,8 +7661,9 @@ sub init_builtin_macros() {
min => sub {my($name, at args) = @_; min(map {/^\s*\z/?undef:$_} @args)},
max => sub {my($name, at args) = @_; max(map {/^\s*\z/?undef:$_} @args)},
sprintf=> sub {my($name,$fmt, at args) = @_; sprintf($fmt, at args)},
- dquote => sub {my($nm)=shift; join('', map { s{"}{\\"}g; '"'.$_.'"' } @_)},
- uquote => sub {my($nm)=shift; join('', map { s{[ \t]+}{_}g; $_ } @_)},
+ join => sub {my($name,$sep, at args) = @_; join($sep, at args)},
+ dquote => sub {my($nm)=shift; join('', map { s{"}{""}g; '"'.$_.'"' } @_)},
+ uquote => sub {my($nm)=shift; join('', map { s{[ \t]+}{_}g; $_ } @_)},
# macros f, T, C, B will be defined for each notification as appropriate
# (representing From:, To:, Cc:, and Bcc: respectively)
# remaining free letters: xEGIJKLMPYZ
@@ -7056,7 +7724,7 @@ sub init_local_delivery_aliases() {
);
}
-# tokenize templates (input to macro expansion), after droping privileges
+# tokenize templates (input to macro expansion), after dropping privileges
sub init_tokenize_templates() {
my(@templ_names) = qw(log_templ log_recip_templ
notify_sender_templ notify_virus_recips_templ
@@ -7090,6 +7758,7 @@ sub after_chroot_init() {
sub after_chroot_init() {
$child_invocation_count = $child_task_count = 0;
%modules_basic = %INC; # helps to track missing modules in chroot
+ do_log(5,"after_chroot_init: EUID: %s (%s); EGID: %s (%s)", $>,$<, $),$( );
my(@msg);
my($euid) = $>; # effective UID
$> = 0; # try to become root
@@ -7103,7 +7772,9 @@ sub after_chroot_init() {
# (non-exhaustive: doesn't test for symlink tricks and higher directories).
# The config file has already been executed by now, so it may be
# too late to feel sorry now, but better late then never.
- for my $config_file (@config_files) {
+ my(@actual_c_f) = Amavis::Conf::get_config_files_read();
+ do_log(2,"config files read: %s", join(", ", at actual_c_f));
+ for my $config_file (@actual_c_f) {
local($1); # IO::Handle::_open_mode_string can taint $1 if mode is '+<'
my($fh) = IO::File->new;
my($errn) = stat($config_file) ? 0 : 0+$!;
@@ -7132,18 +7803,18 @@ sub after_chroot_init() {
}
init_tokenize_templates();
init_preparse_ip_lookups();
+
# report versions of some (more interesting) modules
for my $m ('Amavis::Conf',
sort map { s/\.pm\z//; s[/][::]g; $_ } grep { /\.pm\z/ } keys %INC) {
next if !grep { $_ eq $m } qw(Amavis::Conf
Archive::Tar Archive::Zip Compress::Zlib Convert::TNEF Convert::UUlib
MIME::Entity MIME::Parser MIME::Tools Mail::Header Mail::Internet
- Mail::ClamAV Mail::SpamAssassin Mail::SpamAssassin::SpamCopURI URI
- Digest::MD5 Authen::SASL Razor2::Client::Version
- Mail::DKIM Mail::DomainKeys Mail::SPF::Query
- IO::Socket::INET6 Net::DNS Net::SMTP Net::Cmd Net::Server Net::LDAP
- DBI DBD::mysql DBD::Pg DBD::SQLite BerkeleyDB DB_File
- SAVI Unix::Syslog Time::HiRes);
+ Digest::MD5 Digest::SHA Digest::SHA1 Authen::SASL IO::Socket::INET6
+ Net::Server Mail::ClamAV Mail::SpamAssassin Mail::DKIM Mail::DomainKeys
+ Mail::SPF Mail::SPF::Query NetAddr::IP URI Razor2::Client::Version
+ Net::LDAP DBI DBD::mysql DBD::Pg DBD::SQLite BerkeleyDB DB_File
+ Net::DNS Unix::Syslog Time::HiRes SAVI Anomy::Sanitizer);
do_log(0, "Module %-19s %s", $m, $m->VERSION || '?');
}
do_log(0,"Amavis::DB code %s loaded", $extra_code_db ?'':" NOT");
@@ -7224,14 +7895,14 @@ sub post_configure_hook {
### Net::Server hook
### Occurs in the parent (master) process after binding to sockets,
-### but before chrooting and droping privileges
+### but before chrooting and dropping privileges
sub post_bind_hook {
umask(0027); # restore our preferred umask
}
### Net::Server hook
### This hook occurs in the parent (master) process after chroot,
-### change of user, and change of group has occured. It allows
+### after change of user, and change of group has occured. It allows
### for preparation before looping begins.
sub pre_loop_hook {
my($self) = @_;
@@ -7259,7 +7930,8 @@ sub pre_loop_hook {
elsif ($errn) { die "db_home inaccessible, $!: $name" }
elsif (!-d _) { die "db_home is not a directory : $name" }
elsif (!-w _) { die "db_home directory is not writable: $name" }
- Amavis::DB::init(1);
+ # Amavis::DB::init(1, 15+1+40); # SHA-1
+ Amavis::DB::init(1, 15+1+32); # MD5
}
if (!defined($sql_quarantine_chunksize_max)) {
die "Variable \$sql_quarantine_chunksize_max is undefined\n";
@@ -7280,11 +7952,12 @@ sub pre_loop_hook {
elsif (-d _ && !-w _){ die "QUARANTINEDIR directory not writable: $name"}
}
Amavis::SpamControl::init_pre_fork() if $extra_code_antispam;
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ my($msg) = "TROUBLE in pre_loop_hook: $eval_stat"; do_log(-2,"%s",$msg);
+ die("Suicide (" . am_id() . ") " . $msg . "\n");
};
- if ($@ ne '') {
- chomp($@); my($msg) = "TROUBLE in pre_loop_hook: $@"; do_log(-2,"%s",$msg);
- die("Suicide (" . am_id() . ") " . $msg . "\n");
- }
1;
}
@@ -7299,13 +7972,16 @@ sub write_to_log_hook {
my($self,$level,$msg) = @_;
my($prop) = $self->{server};
local $SIG{CHLD} = 'DEFAULT';
- chomp($msg);
- do_log(1, "Net::Server: %s", $msg); # just call Amavis' traditional logging
+ $level = 0 if $level < 0; $level = 4 if $level > 4;
+# my($ll) = (-2,-1,0,1,3)[$level]; # 0=err, 1=warn, 2=notice, 3=info, 4=debug
+ my($ll) = (-1, 0,1,3,4)[$level]; # 0=err, 1=warn, 2=notice, 3=info, 4=debug
+ chomp($msg); # just call Amavis' traditional logging
+ ll($ll) && do_log($ll, "Net::Server: %s", $msg);
1;
}
### user customizable Net::Server hook (Net::Server 0.88 or later),
-### hook occurs in the master process
+### hook occurs in the master process !!!
sub run_n_children_hook {
Amavis::AV::sophos_savi_reload()
if $extra_code_antivirus && Amavis::AV::sophos_savi_stale();
@@ -7319,21 +7995,24 @@ sub child_init_hook {
sub child_init_hook {
my($self) = @_;
local $SIG{CHLD} = 'DEFAULT';
- $0 = 'amavisd (virgin child)';
+ $my_pid = $$; $0 = 'amavisd (virgin child)';
my($inherited_entropy);
eval {
$db_env = $snmp_db = $body_digest_cache = undef; # just in case
Amavis::Timing::init(); snmp_counters_init();
close_log(); open_log(); # reopen syslog or log file to get per-process fd
if ($extra_code_db) {
+ # Berkeley DB handles should not be shared across process forks,
+ # each forked child should acquire its own Berkeley DB handles
$db_env = Amavis::DB->new; # get access to a bdb environment
$snmp_db = Amavis::DB::SNMP->new($db_env);
- $snmp_db->register_proc('') if defined $snmp_db; # process alive & idle
+ $snmp_db->register_proc(0,1,'') if defined $snmp_db; # alive and idle
my($var_ref) = $snmp_db->read_snmp_variables('entropy');
$inherited_entropy = $var_ref->[0] if $var_ref && @$var_ref;
}
# if $db_env is undef the Amavis::Cache::new creates a memory-based cache
- $body_digest_cache = Amavis::Cache->new($db_env);
+ # $body_digest_cache = Amavis::Cache->new($db_env, 15+1+40); # SHA-1
+ $body_digest_cache = Amavis::Cache->new($db_env, 15+1+32); # MD5
if ($extra_code_db) { # is it worth reporting the timing? (probably not)
section_time('bdb-open');
do_log(2, "%s", Amavis::Timing::report()); # report elapsed times
@@ -7369,11 +8048,12 @@ sub child_init_hook {
'sel_policy') if $sql_dataset_conn_lookups;
$sql_wblist = Amavis::Lookup::SQL->new($sql_dataset_conn_lookups,
'sel_wblist') if $sql_dataset_conn_lookups;
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ do_log(-2, "TROUBLE in child_init_hook: %s", $eval_stat);
+ die "Suicide in child_init_hook: $eval_stat\n";
};
- if ($@ ne '') {
- chomp($@); do_log(-2, "TROUBLE in child_init_hook: %s", $@);
- die "Suicide in child_init_hook: $@\n";
- }
add_entropy($$, Time::HiRes::gettimeofday, $inherited_entropy);
Amavis::Timing::go_idle('vir');
}
@@ -7387,7 +8067,7 @@ sub post_accept_hook {
Amavis::Timing::go_busy('hi ');
# establish initial time right after 'accept'
Amavis::Timing::init(); snmp_counters_init();
- $snmp_db->register_proc('A') if defined $snmp_db; # in 'accept' state
+ $snmp_db->register_proc(1,1,'A') if defined $snmp_db; # enter 'accept' state
load_policy_bank(''); # start with a builtin policy bank
}
@@ -7453,6 +8133,7 @@ sub process_request {
# fine with 0.93.
binmode($sock) or die "Can't set socket to binmode: $!";
local $SIG{ALRM} = sub { die "timed out\n" }; # do not modify the sig text!
+ my($eval_stat);
eval {
# if ($] < 5.006) # Perl older than 5.6.0 did not set FD_CLOEXEC on sockets
# { cloexec($_,1,$_) for @{$prop->{sock}} }
@@ -7568,6 +8249,7 @@ sub process_request {
my($suggested_protocol) = c('protocol'); # suggested by the policy bank
ll(5) && do_log(5,"process_request: suggested_protocol=\"%s\" on %s",
$suggested_protocol,$sock->NS_proto);
+ # $snmp_db->register_proc(2,0,'b') if defined $snmp_db; # begin protocol
if ($sock->NS_proto eq 'UNIX') { # traditional amavis helper program
if ($suggested_protocol eq 'COURIER') {
die "unavailable support for protocol: $suggested_protocol";
@@ -7600,26 +8282,30 @@ sub process_request {
} else {
die("unsupported protocol: $suggested_protocol, " . $sock->NS_proto);
}
- }; # eval
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
alarm(0); # stop the timer
- if ($@ ne '') {
- chomp($@); my($timed_out) = $@ eq "timed out";
+ if ($eval_stat ne '') {
+ chomp $eval_stat; my($timed_out) = $eval_stat =~ /^timed out\b/;
if ($timed_out) {
my($msg) = "Requesting process rundown, task exceeded allowed time";
$msg .= " during waiting for input from client" if waiting_for_client();
do_log(-1, $msg);
} else {
- do_log(-2, "TROUBLE in process_request: %s", $@);
+ do_log(-2, "TROUBLE in process_request: %s", $eval_stat);
$smtp_in_obj->preserve_evidence(1) if $smtp_in_obj;
do_log(-1, "Requesting process rundown after fatal error");
}
undef $smtp_in_obj; undef $amcl_in_obj; undef $courier_in_obj;
$self->done(1);
- } elsif ($child_task_count >= $max_requests) {
+ } elsif ($max_requests > 0 && $child_task_count >= $max_requests) {
# in case of multiple-transaction protocols (e.g. SMTP, LMTP)
# we do not like to keep running indefinitely at the mercy of MTA
- do_log(2, "Requesting process rundown after %d tasks (and %s sessions)",
- $child_task_count,$child_invocation_count);
+ my($have_sawampersand)=Devel::SawAmpersand->UNIVERSAL::can("sawampersand");
+ do_log(2, "Requesting process rundown after %d tasks (and %s sessions)%s",
+ $child_task_count, $child_invocation_count,
+ !$have_sawampersand ? '' : Devel::SawAmpersand::sawampersand() ?
+ ", SawAmpersand is TRUE!" : ", SawAmpersand is false");
undef $smtp_in_obj; undef $amcl_in_obj; undef $courier_in_obj;
$self->done(1);
} elsif ($extra_code_antivirus && Amavis::AV::sophos_savi_stale() ) {
@@ -7629,20 +8315,12 @@ sub process_request {
}
my(@modules_extra) = grep {!exists $modules_basic{$_}} keys %INC;
# do_log(2, "modules loaded: %s", join(", ", sort keys %modules_basic));
- do_log(1, "extra modules loaded: %s",
- join(", ", sort @modules_extra)) if @modules_extra;
+ if (@modules_extra) {
+ do_log(1, "extra modules loaded: %s", join(", ", sort @modules_extra));
+ %modules_basic = %INC;
+ }
do_log(5, "exiting process_request");
}
-
-### override Net::Server::PreForkSimple::done (needed for Net::Server <= 0.87)
-### to be able to rundown the child process prematurely
-#sub done(@) {
-# my($self) = shift;
-# if (@_) { $self->{server}->{done} = shift }
-# elsif (!$self->{server}->{done})
-# { $self->{server}->{done} = $self->SUPER::done }
-# $self->{server}->{done};
-#}
### Net::Server hook
sub post_process_request_hook {
@@ -7654,7 +8332,7 @@ sub post_process_request_hook {
my($remaining_time) = alarm(0);
do_log(5,"post_process_request_hook: %s",
$remaining_time==0 ? "timer was not running" : "timer stopped");
- $snmp_db->register_proc('') if defined $snmp_db; # process is alive and idle
+ $snmp_db->register_proc(1,0,'') if defined $snmp_db; # alive and idle again
Amavis::Timing::go_idle('bye');
if (ll(3)) {
my($load_report) = Amavis::Timing::report_load();
@@ -7679,7 +8357,7 @@ sub child_finish_hook {
undef $sql_storage; undef $sql_wblist; undef $sql_policy; undef $ldap_policy;
undef $sql_dataset_conn_lookups; undef $sql_dataset_conn_storage;
undef $ldap_connection; undef $body_digest_cache;
- eval { $snmp_db->register_proc(undef) } if defined $snmp_db; # going away
+ eval { $snmp_db->register_proc(0,0,undef) } if defined $snmp_db; # unregister
undef $snmp_db; undef $db_env;
}
@@ -7689,7 +8367,7 @@ sub END { # runs before e
undef $sql_storage; undef $sql_wblist; undef $sql_policy; undef $ldap_policy;
undef $sql_dataset_conn_lookups; undef $sql_dataset_conn_storage;
undef $ldap_connection; undef $body_digest_cache;
- eval { $snmp_db->register_proc(undef) } if defined $snmp_db; # going away
+ eval { $snmp_db->register_proc(0,0,undef) } if defined $snmp_db; # unregister
undef $snmp_db; undef $db_env;
}
@@ -7759,93 +8437,170 @@ sub check_mail($$$) {
sub check_mail($$$) {
my($conn, $msginfo, $dsn_per_recip_capable) = @_;
+ my($which_section) = 'check_init'; my(%elapsed,$t0_sect);
+ $elapsed{'TimeElapsedReceiving'} = Time::HiRes::time - $msginfo->rx_time;
my($point_of_no_return) = 0; # past the point where mail or DSN was sent
- my($am_id) = am_id();
- $snmp_db->register_proc($am_id) if defined $snmp_db;
- my($tempdir) = $msginfo->mail_tempdir; my($fh) = $msginfo->mail_text;
- my($sender) = $msginfo->sender; my(@recips) = @{$msginfo->recips};
- # ugly - save in a global to make it accessible to %builtins
- $MSGINFO = $msginfo; $CONN = $conn;
- $msginfo->add_contents_category(CC_CLEAN,0); # CC_CLEAN is always present
- $_->add_contents_category(CC_CLEAN,0) for @{$msginfo->per_recip_data};
- # compute body digest, measure mail size, check for 8-bit data, add entropy
- my($body_digest) = get_body_digest($fh, $msginfo);
-
- my($mail_size) = $msginfo->msg_size; # use corrected ESMTP size if available
- if ($mail_size <= 0) { # not available?
- $mail_size = $msginfo->orig_header_size + 1 + $msginfo->orig_body_size;
- $msginfo->msg_size($mail_size); # store back
- }
- my($file_generator_object) = # maxfiles 0 disables the $MAXFILES limit
- Amavis::Unpackers::NewFilename->new($MAXFILES?$MAXFILES:undef, $mail_size);
- Amavis::Unpackers::Part::init($file_generator_object); # fudge: keep in variable
- my($parts_root) = Amavis::Unpackers::Part->new;
- $msginfo->parts_root($parts_root);
- my($smtp_resp, $exit_code, $preserve_evidence); my($virus_dejavu) = 0;
- my($virus_presence_checked,$spam_presence_checked);
-
- # is any mail component password protected or otherwise non-decodable?
- my($any_undecipherable) = 0;
-
- my($mime_err); # undef, or MIME parsing error string as given by MIME::Parser
-
+ my($am_id) = $msginfo->log_id;
+ if (!defined($am_id)) { $am_id = am_id(); $msginfo->log_id($am_id) }
+ $snmp_db->register_proc(1,0,'=',$am_id) if defined $snmp_db; # check begins
+ my($smtp_resp, $exit_code, $preserve_evidence);
+ my($mail_id, $body_digest, $custom_object);
my($hold); # set to some string causes the message to be placed on hold
# (frozen) by MTA. This can be used in cases when we stumble
# across some permanent problem making us unable to decide
# if the message is to be really delivered.
-
- my($cl_ip) = $msginfo->client_addr;
- add_entropy(Time::HiRes::gettimeofday,
+ # is any mail component password protected or otherwise non-decodable?
+ my($any_undecipherable) = 0;
+ my($mime_err); # undef, or MIME parsing error string as given by MIME::Parser
+
+ # ugly - save in a global to make it accessible to %builtins
+ $MSGINFO = $msginfo; $CONN = $conn;
+ eval {
+ my(@recips); my($sender) = $msginfo->sender;
+ my($cnt_local) = 0; my($cnt_remote) = 0;
+ for my $r (@{$msginfo->per_recip_data}) {
+ my($recip) = $r->recip_addr; push(@recips,$recip);
+ my($is_local) = lookup(0,$recip, @{ca('local_domains_maps')});
+ $is_local ? $cnt_local++ : $cnt_remote++;
+ $r->recip_is_local($is_local); # keep this frequently used information
+ $r->add_contents_category(CC_CLEAN,0); # CC_CLEAN is always present
+ }
+ $msginfo->add_contents_category(CC_CLEAN,0); # CC_CLEAN is always present
+ my($tempdir) = $msginfo->mail_tempdir; my($fh) = $msginfo->mail_text;
+ $msginfo->header_edits(Amavis::Out::EditHeader->new);
+ section_time($which_section);
+ $which_section = 'check_init2';
+ # compute body digest, measure mail size, check for 8-bit data, add entropy
+ # $body_digest = get_body_digest($fh, $msginfo, 'SHA-1');
+ $body_digest = get_body_digest($fh, $msginfo, 'MD5');
+
+ # obtain rfc2822 From and Sender addr. from the mail header, parsed/clean
+ my($rfc2822_sender) = $msginfo->orig_header_fields->{'sender'};
+ my($rfc2822_from_field) = $msginfo->orig_header_fields->{'from'};
+ my(@rfc2822_from); # rfc2822 allows multiple author's addresses
+ if (defined $rfc2822_from_field) {
+ @rfc2822_from = map { unquote_rfc2821_local($_) }
+ parse_address_list($rfc2822_from_field);
+ # rfc2822_from is a ref to a list if there are multiple author addresses!
+ $msginfo->rfc2822_from(@rfc2822_from < 2 ? $rfc2822_from[0]
+ : \@rfc2822_from);
+ }
+ if (defined $rfc2822_sender) {
+ my(@sender_parsed) = map { unquote_rfc2821_local($_) }
+ parse_address_list($rfc2822_sender);
+ $rfc2822_sender = !@sender_parsed ? '' : $sender_parsed[0]; # none or one
+ $msginfo->rfc2822_sender($rfc2822_sender);
+ }
+ if (defined $msginfo->orig_header_fields->{'to'}) {
+ my($rfc2822_to) = $msginfo->orig_header_fields->{'to'};
+ my(@to_parsed) = map { unquote_rfc2821_local($_) }
+ parse_address_list($rfc2822_to);
+ $msginfo->rfc2822_to(@to_parsed<2 ? $to_parsed[0] : \@to_parsed);
+ }
+ if (defined $msginfo->orig_header_fields->{'cc'}) {
+ my($rfc2822_cc) = $msginfo->orig_header_fields->{'cc'};
+ my(@cc_parsed) = map { unquote_rfc2821_local($_) }
+ parse_address_list($rfc2822_cc);
+ $msginfo->rfc2822_cc(@cc_parsed<2 ? $cc_parsed[0] : \@cc_parsed);
+ }
+ my($mail_size) = $msginfo->msg_size; # use corrected ESMTP size if avail.
+ if ($mail_size <= 0) { # not available?
+ $mail_size = $msginfo->orig_header_size + 1 + $msginfo->orig_body_size;
+ $msginfo->msg_size($mail_size); # store back
+ }
+ my($file_generator_object) = # maxfiles 0 disables the $MAXFILES limit
+ Amavis::Unpackers::NewFilename->new($MAXFILES?$MAXFILES:undef,$mail_size);
+ Amavis::Unpackers::Part::init($file_generator_object); # fudge: keep in var
+ my($parts_root) = Amavis::Unpackers::Part->new;
+ $msginfo->parts_root($parts_root);
+ my($virus_dejavu) = 0; my($virus_presence_checked,$spam_presence_checked);
+
+ my($cl_ip) = $msginfo->client_addr;
+ add_entropy(Time::HiRes::gettimeofday,
"$child_task_count $am_id $cl_ip $mail_size", $msginfo->queue_id,
$msginfo->mail_text_fn, $sender, \@recips);
- my($mail_id); my($os_fingerprint_obj,$os_fingerprint);
- my($which_section);
-
- $which_section = 'gen_mail_id';
- # create unique mail_id and save preliminary information to SQL (if enabled)
- for (my($attempt)=5; $attempt>0; ) { # sanity limit on retries
- my($secret_id);
- ($mail_id,$secret_id) = generate_mail_id();
- $msginfo->secret_id($secret_id); $secret_id = '';
- $msginfo->mail_id($mail_id); # assign some long-term unique id to the msg
- if (!$sql_storage) { last } # no need to store and to check for uniqueness
- else { # attempt to save message placeholder to SQL ensuring it is unique
- $which_section = 'sql-enter';
- $sql_storage->save_info_preliminary($conn,$msginfo)
- and last;
- if (--$attempt <= 0) {
- do_log(-2,"ERROR sql_storage: too many retries ".
- "on storing preliminary, info not saved");
- } else {
- do_log(2,"sql_storage: retrying preliminary, %d attempts remain",
- $attempt);
- sleep(int(1+rand(3))); add_entropy(Time::HiRes::gettimeofday,$attempt);
- }
- }
- };
- section_time($which_section);
-
- my($os_fingerprint_method) = c('os_fingerprint_method');
- if (!defined($os_fingerprint_method) || $os_fingerprint_method eq '') {
- # no fingerprinting service configured
- } elsif ($cl_ip eq '' || $cl_ip eq '0.0.0.0' || $cl_ip eq '::') {
- # original client IP address not available, can't query p0f
- } else {
- $which_section = "os_fingerprint";
- $os_fingerprint_obj = Amavis::OS_Fingerprint->new(
+ my($os_fingerprint_obj,$os_fingerprint);
+
+# section_time($which_section);
+ $which_section = 'gen_mail_id';
+ # create unique mail_id and save preliminary info. to SQL (if enabled)
+ for (my($attempt)=5; $attempt>0; ) { # sanity limit on retries
+ my($secret_id);
+ ($mail_id,$secret_id) = generate_mail_id();
+ $msginfo->secret_id($secret_id); $secret_id = '';
+ $msginfo->mail_id($mail_id); # assign some long-term unique id to the msg
+ if (!$sql_storage) { last } #no need to store and to check for uniqueness
+ else { # attempt to save message placeholder to SQL ensuring it is unique
+ $which_section = 'sql-enter';
+ $sql_storage->save_info_preliminary($conn,$msginfo)
+ and last;
+ if (--$attempt <= 0) {
+ do_log(-2,"ERROR sql_storage: too many retries ".
+ "on storing preliminary, info not saved");
+ } else {
+ do_log(2,"sql_storage: retrying preliminary, %d attempts remain",
+ $attempt);
+ sleep(int(1+rand(3)));
+ add_entropy(Time::HiRes::gettimeofday,$attempt);
+ }
+ }
+ };
+ section_time($which_section);
+
+ my($os_fingerprint_method) = c('os_fingerprint_method');
+ if (!defined($os_fingerprint_method) || $os_fingerprint_method eq '') {
+ # no fingerprinting service configured
+ } elsif ($cl_ip eq '' || $cl_ip eq '0.0.0.0' || $cl_ip eq '::') {
+ # original client IP address not available, can't query p0f
+ } else {
+ $which_section = "os_fingerprint";
+ $os_fingerprint_obj = Amavis::OS_Fingerprint->new(
dynamic_destination($os_fingerprint_method,$conn,0),
0.050, $cl_ip, $mail_id);
- }
- my($pbn) = c('policy_bank_path');
- do_log(1,"Checking: %s %s%s%s -> %s", $mail_id,
- $pbn eq '' ? '' : "$pbn ", $cl_ip eq '' ? '' : "[$cl_ip] ",
- qquote_rfc2821_local($sender),
- join(',', qquote_rfc2821_local(@recips)) );
- eval {
+ }
+ my($pbn) = c('policy_bank_path');
+ do_log(1,"Checking: %s %s%s%s -> %s", $mail_id,
+ $pbn eq '' ? '' : "$pbn ", $cl_ip eq '' ? '' : "[$cl_ip] ",
+ qquote_rfc2821_local($sender),
+ join(',', qquote_rfc2821_local(@recips)) );
+ if (ll(3)) {
+ my($envsender) = qquote_rfc2821_local($sender);
+ my($hdrsender) = qquote_rfc2821_local($rfc2822_sender),
+ my($hdrfrom) = qquote_rfc2821_local($rfc2822_from[0]);
+ do_log(3,"2822.From: %s%s%s", $hdrfrom,
+ !defined($rfc2822_sender) ? '' : ", 2822.Sender: $hdrsender",
+ defined $rfc2822_sender && $envsender eq $hdrsender ? ''
+ : $envsender eq $hdrfrom ? '' : ", 2821.Mail_From: $envsender");
+ }
+ $which_section = "custom-new";
+ eval {
+ $custom_object = Amavis::Custom->new($conn,$msginfo); 1;
+ } or do {
+ undef $custom_object;
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ do_log(-1,"custom new err: %s", $eval_stat);
+ };
+ if (ref $custom_object)
+ { do_log(5,"Custom hooks enabled"); section_time($which_section) }
+
+ # update message count and mesage size snmp-like counters
snmp_count('InMsgs');
snmp_count('InMsgsNullRPath') if $sender eq '';
- if (@recips == 1) { snmp_count( 'InMsgsRecips' ) }
- elsif (@recips > 1) { snmp_count( ['InMsgsRecips',scalar(@recips)] ) }
+ snmp_count( ['InMsgsRecips', $cnt_local+$cnt_remote]); # recipients count
+ snmp_count( ['InMsgsSize', $mail_size, 'C64'] );
+ if ($cnt_remote > 0) {
+ snmp_count('InMsgsOutbound');
+ snmp_count( ['InMsgsRecipsOutbound', $cnt_remote]);
+ snmp_count( ['InMsgsSizeOutbound', $mail_size, 'C64'] );
+ }
+ if ($cnt_local > 0) {
+ snmp_count('InMsgsInboundOrInt'); # non_outbound = internal+inbound
+ snmp_count( ['InMsgsRecipsInboundOrInt', $cnt_local]);
+ my($d) = c('originating') ? 'Internal' : 'Inbound';
+ snmp_count( 'InMsgs'.$d);
+ snmp_count( ['InMsgsRecips'.$d, $cnt_local] );
+ snmp_count( ['InMsgsSize'.$d, $mail_size, 'C64'] );
+ }
# mkdir can be a costly operation (must be atomic, flushes buffers).
# If we can re-use directory 'parts' from the previous invocation it saves
@@ -7881,7 +8636,7 @@ sub check_mail($$$) {
$cache_entry->{'ctime'} = $now_utc_iso8601; # create a new cache record
} else {
snmp_count('CacheHits');
- $virus_presence_checked = defined $cache_entry->{'VN'} ? 1 : 0;
+ $virus_presence_checked = defined $cache_entry->{'VN'} ? 1 : 0;
# spam level and spam report may be influenced by mail header, not only
# by mail body, so caching based on body is only a close approximation;
@@ -7927,11 +8682,11 @@ sub check_mail($$$) {
snmp_count('CacheHitsVirusCheck') if $virus_presence_checked;
snmp_count('CacheHitsVirusMsgs') if @virusname;
snmp_count('CacheHitsSpamCheck') if $spam_presence_checked;
- snmp_count('CacheHitsSpamMsgs') if $msginfo->spam_level >= 6; # a hack
+ snmp_count('CacheHitsSpamMsgs') if $msginfo->spam_level >= 5; # a hack
ll(5) && do_log(5,"cache entry age: %s c=%s a=%s",
- (@virusname ? 'V' : $msginfo->spam_level > 5 ? 'S' : '.'),
+ (@virusname ? 'V' : $msginfo->spam_level >= 5 ? 'S' : '.'),
$cache_entry->{'ctime'}, $cache_entry->{'atime'} );
- } # if defined $cache_entry
+ } # end if defined $cache_entry
my($will_do_virus_scanning, $all_bypass_virus_checks);
if ($extra_code_antivirus) {
@@ -7948,12 +8703,14 @@ sub check_mail($$$) {
!c('bypass_decode_parts') &&
($will_do_virus_scanning || $will_do_banned_checking);
- $which_section = "mime_decode-1";
- my($ent); ($ent,$mime_err) = mime_decode($fh, $tempdir, $parts_root);
- $msginfo->mime_entity($ent);
- prolong_timer($which_section);
-
if ($will_do_parts_decoding) { # decoding parts can take a lot of time
+ $which_section = "mime_decode-1";
+ $snmp_db->register_proc(2,0,'D',$am_id) if defined $snmp_db; # decoding
+ $t0_sect = Time::HiRes::time;
+ $mime_err = ensure_mime_entity($msginfo, $fh, $tempdir, $parts_root)
+ if !defined($msginfo->mime_entity);
+ prolong_timer($which_section);
+
$which_section = "parts_decode_ext";
snmp_count('OpsDec');
($hold,$any_undecipherable) =
@@ -7963,12 +8720,16 @@ sub check_mail($$$) {
$_->add_contents_category(CC_UNCHECKED,0)
for @{$msginfo->per_recip_data};
}
+ $elapsed{'TimeElapsedDecoding'} = Time::HiRes::time - $t0_sect;
}
my($bphcm) = ca('bypass_header_checks_maps');
if (grep {!lookup(0,$_->recip_addr,@$bphcm)} @{$msginfo->per_recip_data}) {
+ $which_section = "check_header";
+ my($allowed_tests) = cr('allowed_header_tests');
+ my($allowed_mime_test) = $allowed_tests && $allowed_tests->{'mime'};
# check for bad headers and for bad MIME subheaders / bad MIME structure
- if (defined $mime_err && $mime_err ne '') {
+ if ($allowed_mime_test && defined $mime_err && $mime_err ne '') {
push(@bad_headers, "MIME error: ".$mime_err);
$msginfo->add_contents_category(CC_BADH,1);
}
@@ -7979,11 +8740,13 @@ sub check_mail($$$) {
}
for my $r (@{$msginfo->per_recip_data}) {
my($bypassed) = lookup(0,$r->recip_addr,@$bphcm);
- if (!$bypassed && defined $mime_err && $mime_err ne '')
+ if (!$bypassed && $allowed_mime_test &&
+ defined $mime_err && $mime_err ne '')
{ $r->add_contents_category(CC_BADH,1) } # CC_BADH min: 1=broken mime
if (!$bypassed && @$badh_ref)
{ $r->add_contents_category(CC_BADH,$minor_badh_cc) }
}
+ section_time($which_section);
}
if ($will_do_banned_checking) { # check for banned file contents
@@ -8023,14 +8786,11 @@ sub check_mail($$$) {
} else {
if (!$will_do_virus_scanning)
{ do_log(-1, "NOTICE: will_do_virus_scanning is false???") }
- if (!defined($msginfo->mime_entity)) {
- $which_section = "mime_decode-3";
- my($ent); ($ent,$mime_err) = mime_decode($fh, $tempdir, $parts_root);
- $msginfo->mime_entity($ent);
- prolong_timer($which_section);
- }
+ $mime_err = ensure_mime_entity($msginfo, $fh, $tempdir, $parts_root)
+ if !defined($msginfo->mime_entity) && !c('bypass_decode_parts');
# special case to make available a complete mail file for inspection
if ((defined($mime_err) && $mime_err ne '') ||
+ !defined($msginfo->mime_entity) ||
lookup(0,'MAIL', at keep_decoded_original_maps) ||
$any_undecipherable && lookup(0,'MAIL-UNDECIPHERABLE',
@keep_decoded_original_maps)) {
@@ -8039,7 +8799,7 @@ sub check_mail($$$) {
my($newpart_obj) =
Amavis::Unpackers::Part->new("$tempdir/parts",$parts_root,1);
my($newpart) = $newpart_obj->full_name;
- do_log(2, "presenting full original message to scanners as %s%s%s",
+ do_log(3, "presenting full original message to scanners as %s%s%s",
$newpart,
!$any_undecipherable ? '' :", $any_undecipherable undecipherable",
$mime_err eq '' ? '' : ", MIME error: $mime_err");
@@ -8050,16 +8810,17 @@ sub check_mail($$$) {
$newpart_obj->type_declared('message/rfc822');
}
$which_section = "virus_scan";
- my($av_ret);
+ $snmp_db->register_proc(2,0,'V',$am_id) if defined $snmp_db; # virus scan
+ my($av_ret); $t0_sect = Time::HiRes::time;
eval {
my($vn, $ds);
- ($av_ret, $av_output, $vn, $ds) =
- Amavis::AV::virus_scan($tempdir, $child_task_count==1, $parts_root);
+ ($av_ret, $av_output, $vn, $ds) = Amavis::AV::virus_scan(
+ $conn,$msginfo,$tempdir, $child_task_count==1, $parts_root);
@virusname = @$vn; @detecting_scanners = @$ds; # copy
- };
- if ($@ ne '') {
- chomp($@);
- if ($@ eq "timed out") { # can't happen, timer is stopped
+ 1;
+ } or do {
+ $@ = "errno=$!" if $@ eq ''; chomp $@;
+ if ($@ =~ /^timed out\b/) { # not supposed to happen
@virusname = (); $av_ret = 0; # assume not a virus!
do_log(-1, "virus_scan TIMED OUT, ASSUME NOT A VIRUS !!!");
} else {
@@ -8067,11 +8828,17 @@ sub check_mail($$$) {
$av_ret = 0; # pretend it was ok (msg should be held)
die "$hold\n"; # die, TEMPFAIL is preferred to HOLD
}
- }
+ };
+ $elapsed{'TimeElapsedVirusCheck'} = Time::HiRes::time - $t0_sect;
snmp_count('OpsVirusCheck');
defined($av_ret) or die "All virus scanners failed!";
@$cache_entry{'Vt','VO','VN','VD'} =
($now, $av_output, \@virusname, \@detecting_scanners);
+ if (defined($msginfo->spam_level)) { #also spam results if provided by av
+ @$cache_entry{'St','SL','SS','SR','SY'} =
+ ($now, $msginfo->spam_level, $msginfo->spam_status,
+ $msginfo->spam_report, $msginfo->spam_summary);
+ }
$virus_presence_checked = 1;
if (defined $snmp_db && @virusname) {
$which_section = "read_snmp_variables";
@@ -8091,15 +8858,16 @@ sub check_mail($$$) {
}
}
$msginfo->add_contents_category(CC_VIRUS,0) if @virusname;
- my($sender_contact,$sender_source);
- if (!@virusname) { $sender_contact = $sender_source = $sender }
- else {
- ($sender_contact,$sender_source) =
- best_try_originator($msginfo,\@virusname);
- section_time('best_try_originator');
- }
- $msginfo->sender_contact($sender_contact); # save it
- $msginfo->sender_source($sender_source); # save it
+ $msginfo->virusnames([@virusname]) if @virusname; # copy names to object
+ { my($sender_contact,$sender_source);
+ if (!@virusname) { $sender_contact = $sender_source = $sender }
+ else {
+ ($sender_contact,$sender_source) = best_try_originator($msginfo);
+ section_time('best_try_originator');
+ }
+ $msginfo->sender_contact($sender_contact); # save it
+ $msginfo->sender_source($sender_source); # save it
+ }
if (defined($os_fingerprint_obj)) {
$which_section = "fingerprint_collect";
@@ -8116,8 +8884,8 @@ sub check_mail($$$) {
do_log(5, "no anti-spam code loaded, skipping spam_scan");
} elsif (@virusname) {
do_log(5, "infected contents, skipping spam_scan");
- } elsif ($banned_filename_all) {
- do_log(5, "banned contents, skipping spam_scan");
+ # } elsif ($banned_filename_all) {
+ # do_log(5, "banned contents, skipping spam_scan");
} elsif (!grep {!lookup(0,$_,@{ca('bypass_spam_checks_maps')})} @recips) {
do_log(5, "bypassing of spam checks requested");
} else {
@@ -8131,10 +8899,13 @@ sub check_mail($$$) {
do_log(5, "spam_presence cached, skipping spam_scan");
} else {
$which_section = "spam_scan";
+ $snmp_db->register_proc(2,0,'S',$am_id) if defined $snmp_db; #spamcheck
+ $t0_sect = Time::HiRes::time;
# sets $msginfo->spam_level, spam_status,
- # spam_report, spam_summary, autolearn_status
+ # spam_report, spam_summary, supplementary_info
Amavis::SpamControl::spam_scan($conn,$msginfo);
prolong_timer($which_section);
+ $elapsed{'TimeElapsedSpamCheck'} = Time::HiRes::time - $t0_sect;
snmp_count('OpsSpamCheck');
@$cache_entry{'St','SL','SS','SR','SY'} =
($now, $msginfo->spam_level, $msginfo->spam_status,
@@ -8142,7 +8913,16 @@ sub check_mail($$$) {
$spam_presence_checked = 1;
}
}
-
+ if (ref $custom_object) {
+ $which_section = "custom-checks";
+ eval {
+ $custom_object->checks($conn,$msginfo); 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ do_log(-1,"custom checks error: %s", $eval_stat);
+ };
+ section_time($which_section);
+ }
# store to cache
$which_section = 'update_cache';
$cache_entry->{'atime'} = $now_utc_iso8601; # update accessed timestamp
@@ -8159,7 +8939,7 @@ sub check_mail($$$) {
my($spam_level) = $msginfo->spam_level;
$which_section = "penpals_check";
- if (defined($sql_storage) && $sender ne '' && !@virusname) {
+ if (defined($sql_storage) && !@virusname) {
my($pp_bonus) = c('penpals_bonus_score'); # score points
my($pp_halflife) = c('penpals_halflife'); # seconds
my(@boost_list);
@@ -8174,21 +8954,26 @@ sub check_mail($$$) {
$spam_level + min(@boost_list) - $pp_bonus
> $penpals_threshold_high) {}
# spam, can't get below threshold_high even under best circumstances
- elsif (!$msginfo->client_addr_mynets &&
+ elsif (!c('originating') && $sender ne '' &&
lookup(0,$sender,@{ca('local_domains_maps')})) {}
- # don't trust senders from outside using a local domain address
+ # no bonus to senders from outside using local domain, can't trust them
else {
+ $t0_sect = Time::HiRes::time;
+ $snmp_db->register_proc(2,0,'P',$am_id) if defined $snmp_db; # penpals
for my $r (@{$msginfo->per_recip_data}) {
next if $r->recip_done; # already dealt with
my($recip) = $r->recip_addr;
my($sid,$rid) = ($msginfo->sender_maddr_id, $r->recip_maddr_id);
- if (defined($sid) && defined($rid) && $sid != $rid
- && lookup(0,$recip,@{ca('local_domains_maps')}) ) {
+ if (defined($rid) && $sid ne $rid && $r->recip_is_local) {
+ my($refs_str) = $msginfo->orig_header_fields->{'in-reply-to'} .
+ $msginfo->orig_header_fields->{'references'};
+ my(@refs) = $refs_str eq '' ? () : parse_message_id($refs_str);
+ do_log(4,"penpals: references: %s", join(", ", at refs)) if @refs;
# NOTE: swap $rid and $sid as args here, as we are now checking
# for a potential reply mail - whether the current recipient has
# recently sent any mail to the sender of the current mail:
my($pp_age,$pp_mail_id,$pp_subj) =
- $sql_storage->penpals_find($rid,$sid,$msginfo->rx_time);
+ $sql_storage->penpals_find($rid,$sid,\@refs,$msginfo->rx_time);
if (defined($pp_age)) { # found info about previous correspondence
$r->recip_penpals_age($pp_age); # save the information
my($weight) = exp(-($pp_age/$pp_halflife) * log(2));
@@ -8198,6 +8983,7 @@ sub check_mail($$$) {
my($boost) = $r->recip_score_boost;
my($adj) = $weight * $pp_bonus; $boost -= $adj;
$r->recip_score_boost($boost); # save adjusted result to object
+ $r->recip_penpals_score(-$adj);
if (ll(2)) {
do_log(2,"penpals: bonus %.3f, age %s (%d), ".
"SA score %.3f, <%s> replying to <%s>, ref mail_id: %s",
@@ -8212,12 +8998,21 @@ sub check_mail($$$) {
}
}
section_time($which_section);
+ $elapsed{'TimeElapsedPenPals'} = Time::HiRes::time - $t0_sect;
}
}
$which_section = "decide_mail_destiny";
- my($is_bulk) = $msginfo->orig_header_fields->{'precedence'};
- $is_bulk = $is_bulk=~/^[ \t]*(bulk|list|junk)\b/i ? $1 : undef;
+ $snmp_db->register_proc(2,0,'r',$am_id) if defined $snmp_db; # results...
+ my($is_bulk); # mail from a mailing list or some kind of a bounce
+ if ($msginfo->orig_header_fields->{'precedence'} =~
+ /^[ \t]*(bulk|list|junk)\b/si) { $is_bulk = $1 }
+ elsif (defined $msginfo->orig_header_fields->{'list-id'}) # rfc2919
+ { $is_bulk = 'List-Id:' . $msginfo->orig_header_fields->{'list-id'} }
+ elsif ($msginfo->rfc2822_from =~ # won't match if multiple addr, who cares
+ /^ (?: [^@]+-(request|owner|relay|bounces) | owner-[^@]+ |
+ postmaster | mailer-daemon | mailer | uucp ) ( @ | \z )/xsi)
+ { $is_bulk = 'From:' . $msginfo->rfc2822_from }
my($considered_oversize_by_some_recips);
my($mslm) = ca('message_size_limit_maps');
for my $r (@{$msginfo->per_recip_data}) {
@@ -8237,6 +9032,9 @@ sub check_mail($$$) {
my($whitelisted) = $r->recip_whitelisted_sender;
my($boost) = $r->recip_score_boost;
$boost = 0 if !defined($boost); # avoid uninitialized value warning
+ # penpals_score is already accounted for in recip_score_boost,
+ # it is provided here separately for informational/logging purposes
+ my($penpals_score) = $r->recip_penpals_score; # is zero or negative!
my($do_tag) = $blacklisted || !defined $tag_level || $tag_level eq '' ||
($spam_level+$boost + ($whitelisted?-10:0) >= $tag_level);
my($do_tag2,$do_tag3,$do_kill) =
@@ -8250,15 +9048,15 @@ sub check_mail($$$) {
}
if ($do_tag2) { # spaminess is at or above tag2 level
$msginfo->add_contents_category(CC_SPAMMY);
- $r->add_contents_category(CC_SPAMMY) if !$bypassed;
+ $r->add_contents_category(CC_SPAMMY) if !$bypassed;
}
if ($do_tag3) { # spaminess is at or above tag3 level
$msginfo->add_contents_category(CC_SPAMMY,1);
- $r->add_contents_category(CC_SPAMMY,1) if !$bypassed;
+ $r->add_contents_category(CC_SPAMMY,1) if !$bypassed;
}
if ($do_kill) { # spaminess is at or above kill level
$msginfo->add_contents_category(CC_SPAM,0);
- $r->add_contents_category(CC_SPAM,0) if !$bypassed;
+ $r->add_contents_category(CC_SPAM,0) if !$bypassed;
}
# consider adding CC_OVERSIZED to the contents_category list;
if (@$mslm) { # checking of mail size required?
@@ -8268,8 +9066,7 @@ sub check_mail($$$) {
{ $size_limit = 65536 } # rfc2821 requires at least 64k
if ($size_limit && $mail_size > $size_limit) {
do_log(1,"OVERSIZED from %s to %s: size %s B, limit %s B",
- qquote_rfc2821_local($sender),
- qquote_rfc2821_local($r->recip_addr),
+ $msginfo->sender_smtp, $r->recip_addr_smtp,
$mail_size, $size_limit)
if !$considered_oversize_by_some_recips;
$considered_oversize_by_some_recips = 1;
@@ -8278,33 +9075,111 @@ sub check_mail($$$) {
}
}
- my($final_destiny) = $r->setting_by_contents_category(
- cr('final_destiny_by_ccat'));
-# if ($final_destiny != D_PASS && lookup(0,$sender,
-# [new_RE(qr'bugtraq-return-.*@securityfocus\.com')] )) {
-# $final_destiny = D_PASS;
-# do_log(0, "malware accepted from sender %s", $sender);
-# }
+ # determine true reason for blocking,considering lovers and final_destiny
+ my($blocking_ccat); my($final_destiny) = D_PASS; my($to_be_mangled);
+ my(@fd_tuples) = $r->setting_by_main_contents_category_all(
+ cr('final_destiny_by_ccat'), cr('lovers_maps_by_ccat'),
+ cr('defang_maps_by_ccat') );
+ for my $tuple (@fd_tuples) {
+ my($cc, $fd, $lovers_map_ref, $mangle_map_ref) = @$tuple;
+ if (!defined($fd) || $fd == D_PASS) {
+ do_log(5, "final_destiny (ccat=%s) is PASS, recip %s", $cc,$recip);
+ } elsif (defined($lovers_map_ref) &&
+ lookup(0,$recip, @$lovers_map_ref)) {
+ do_log(5, "contents lover (ccat=%s) %s", $cc,$recip);
+ } elsif ($fd == D_BOUNCE && (defined $is_bulk || $sender eq '') &&
+ ccat_maj($cc) == CC_BADH) {
+ # have mercy on bad headers from mailing lists and in DSN: since
+ # a bounce for such mail will be suppressed, it is probably better
+ # to just let a mail with a bad header pass, it is rather innocent
+ do_log(1, "allow bad header from %s<%s> -> <%s>: %s",
+ $is_bulk eq '' ?'' :"($is_bulk) ", $sender,$recip,$bad_headers[0]);
+ } else {
+ $blocking_ccat = $cc; $final_destiny = $fd;
+ my($cc_main) = $r->contents_category;
+ $cc_main = $cc_main->[0] if $cc_main;
+ if ($blocking_ccat eq $cc_main) {
+ do_log(3, "blocking contents category is (%s) for %s",
+ $blocking_ccat,$recip);
+ } else {
+ do_log(2, "INFO: blocking ccat (%s) differs from ccat_maj=%s, %s",
+ $blocking_ccat,$cc_main,$recip);
+ }
+ last; # first blocking wins, also skips turning on mangling
+ }
+ # topmost mangling reason wins
+ if (!defined($to_be_mangled) && defined($mangle_map_ref)) {
+ my($mangle_type) = !ref($mangle_map_ref) ? $mangle_map_ref #compatib
+ : lookup(0,$recip, @$mangle_map_ref);
+ $to_be_mangled = $mangle_type if $mangle_type ne '';
+ }
+ }
+ $r->recip_destiny($final_destiny);
+ if (defined $blocking_ccat) { # save a blocking contents category
+ $r->blocking_ccat($blocking_ccat);
+ my($msg_bl_ccat) = $msginfo->blocking_ccat;
+ if (!defined($msg_bl_ccat) || cmp_ccat($blocking_ccat,$msg_bl_ccat)>0)
+ { $msginfo->blocking_ccat($blocking_ccat) }
+ } else { # defanging/mangling only has effect on passed mail
+ # defang_all serves mostly for testing purposes and compatibility
+ $to_be_mangled = 1 if !$to_be_mangled && c('defang_all');
+ if ($to_be_mangled) {
+ my($orig_to_be_mangled) = $to_be_mangled;
+ if ($to_be_mangled =~ /^(?:disclaimer|nulldisclaimer)\z/i) {
+ # disclaimers can only go to mail originating from internal
+ # networks - the 'allow_disclaimers' should (only) be enabled
+ # by an appropriate policy bank, e.g. MYNETS and/or ORIGINATING
+ if (!c('allow_disclaimers'))
+ { $to_be_mangled = 0 } # not for remote or unauthorized clients
+ # disclaimers should only go to mail with 2822.Sender or 2822.From
+ # or 2821.mail_from address matching local domains:
+ elsif (!grep {$_ ne '' && lookup(0,$_,@{ca('local_domains_maps')})}
+ @{unique($rfc2822_sender, @rfc2822_from, $sender)}) {
+ $to_be_mangled = 0; # not for foreign 'Sender:' or 'From:'
+ do_log(5,"will not add disclaimer, sender/author not local");
+ }
+ } else { # defanging (not disclaiming)
+ # defanging and other mail mangling/munging only applies to
+ # incoming mail, i.e. for recipients matching local_domains_maps
+ $to_be_mangled = 0 if !$r->recip_is_local;
+ }
+ # store a boolean or a mangling name (defang, disclaimer, ...)
+ $r->mail_body_mangle($to_be_mangled) if $to_be_mangled;
+ ll(2) && do_log(2, "mangling %s: %s (orig: %s), ".
+ "discl_allowed=%d, <%s> -> <%s>", $to_be_mangled ? 'YES' : 'NO',
+ $to_be_mangled, $orig_to_be_mangled, c('allow_disclaimers'),
+ $sender, $recip);
+ }
+ }
+
+ if ($penpals_score < 0) {
+ # only for logging and statistics purposes
+ my($do_tag2_nopp,$do_tag3_nopp,$do_kill_nopp) =
+ map { !$whitelisted &&
+ ($blacklisted ||
+ (defined($_) && $spam_level+$boost-$penpals_score >= $_) ) }
+ ($tag2_level,$tag3_level,$kill_level);
+ $do_tag2_nopp = $do_tag2_nopp || $do_tag3_nopp;
+ my($which) = $do_kill_nopp && !$do_kill ? 'kill'
+ : $do_tag3_nopp && !$do_tag3 ? 'tag3'
+ : $do_tag2_nopp && !$do_tag2 ? 'tag2' : '';
+ if ($which ne '') {
+ snmp_count("PenPalsSavedFrom\u$which") if $final_destiny==D_PASS;
+ do_log(2, "PenPalsSavedFrom%s %.3f%.3f%s, <%s> -> <%s>", "\u$which",
+ $spam_level+$boost-$penpals_score, $penpals_score,
+ ($final_destiny==D_PASS ? '' : ', but mail still blocked'),
+ $sender, $recip);
+ }
+ }
+
if ($final_destiny == D_PASS) {
# recipient wants this message, malicious or not
- do_log(4, "final_destiny PASS, recip %s", $recip);
- } else {
- my($lovers_map_ref) = $r->setting_by_contents_category(
- cr('lovers_maps_by_ccat'));
- if (defined($lovers_map_ref) && lookup(0,$recip, @$lovers_map_ref)) {
- do_log(4, "malware lover %s", $recip);
- } elsif ($final_destiny == D_BOUNCE &&
- (defined $is_bulk || $sender eq '') &&
- $r->main_contents_category == CC_BADH) {
- # have mercy on mailing lists and DSN: since a bounce for such mail
- # will be suppressed, it is probably better to just let a mail pass
- $final_destiny = D_PASS;
- do_log(1,"allow bad header from %s<%s> -> <%s>: %s",
- $is_bulk eq '' ?'' :"($is_bulk) ", $sender,$recip,$bad_headers[0]);
- } else { # recipient does not want this content
- $r->recip_destiny($final_destiny); # supply RFC 3463 enhanced codes
- my($status_and_reason) = $r->setting_by_contents_category({
- CC_VIRUS,
+ do_log(5, "final_destiny PASS, recip %s", $recip);
+ } else { # recipient does not want this content
+ # supply RFC 3463 enhanced codes
+ my($status_and_reason) = setting_by_given_contents_category(
+ $blocking_ccat,
+ { CC_VIRUS,
["554 5.7.0", "VIRUS: ".join(", ", @virusname)],
CC_BANNED,
["554 5.7.0", "BANNED: ".join(", ",@{$r->banned_parts || []})],
@@ -8316,7 +9191,7 @@ sub check_mail($$$) {
CC_SPAMMY.",1",
["554 5.7.0", "SPAMMY (tag3)"],
CC_SPAMMY,
- ["554 5.7.0", "SPAMMY"],
+ ["554 5.7.0", "SPAMMY"], # tag2
CC_BADH.",2", # nonencoded 8-bit character
["554 5.6.3", "BAD_HEADER: ".(split(/\n/,$bad_headers[0]))[0]],
CC_BADH,
@@ -8326,39 +9201,31 @@ sub check_mail($$$) {
"exceeds recipient's size limit"],
CC_CATCHALL, ["554 5.7.0", "CLEAN"],
});
- my($status,$reason);
- ($status,$reason) = @$status_and_reason if $status_and_reason;
- $final_destiny!=D_PASS or die "Assert failed: $final_destiny==pass";
- if ($final_destiny == D_DISCARD) {
- local($1,$2);
- $status =~ s{^5(\d\d) 5(\.\d\.\d)\z}{2$1 2$2}; # 5xx -> 2xx
- }
- $reason = substr($reason,0,100)."..." if length($reason) > 100+3;
- if (ll(3) && $r->main_contents_category == CC_SPAM) { #compatible log
- my($sl) = !defined($spam_level) ? 'x'
- : 0+sprintf("%.3f",$spam_level); # trim fraction
- do_log(3, "SPAM-KILL, %s -> %s, score=%s, kill=%s%s",
- qquote_rfc2821_local($sender, $recip),
- (!defined($boost) || $boost==0 ? $sl
- : $boost >= 0 ? $sl.'+'.$boost : $sl.$boost),
- !defined($kill_level) ? 'x' : 0+sprintf("%.3f",$kill_level),
- $r->recip_blacklisted_sender ? ', BLACKLISTED' : '');
- }
- $r->recip_smtp_response( $status . ' ' .
- ($final_destiny == D_PASS ? "Ok" :
- $final_destiny == D_DISCARD ?
- "Ok, discarded" : "Reject") .
- ", id=$am_id - $reason");
- $r->recip_done(1);
- # note that 5xx status rejects may later be converted to bounces
+ my($status,$reason);
+ ($status,$reason) = @$status_and_reason if $status_and_reason;
+ $final_destiny!=D_PASS or die "Assert failed: $final_destiny==pass";
+ $reason = substr($reason,0,100)."..." if length($reason) > 100+3;
+ if ($final_destiny == D_DISCARD) {
+ local($1,$2);
+ # $status =~ s{^5(\d\d) 5(\.\d\.\d)\z}{2$1 2$2}; # 5xx -> 2xx
+ $status =~ s{^5(\d\d) 5(\.\d\.\d)\z}{250 2$2}; # 5xx -> 250
}
+ my($response) = $status . ' ' .
+ ($final_destiny == D_PASS ? "Ok" :
+ $final_destiny == D_DISCARD ? "Ok, discarded" : "Reject") .
+ ", id=$am_id - $reason";
+ ll(4) && do_log(4, "blocking ccat=%s, SMTP response: %s",
+ $blocking_ccat,$response);
+ $r->recip_smtp_response($response);
+ $r->recip_done(1); # fake a delivery (confirm delivery to a bit bucket)
+ # note that 5xx status rejects may later be converted to bounces
}
}
section_time($which_section);
- $which_section = "quar+notif";
+ $which_section = "quar+notif"; $t0_sect = Time::HiRes::time;
+ $snmp_db->register_proc(2,0,'Q',$am_id) if defined $snmp_db; # notify, quar
do_notify_and_quarantine($conn, $msginfo, $virus_dejavu);
-
$which_section = "aux_quarantine";
# do_quarantine($conn, $msginfo, undef,
# ['archive-files'], 'local:archive/%m');
@@ -8368,32 +9235,59 @@ sub check_mail($$$) {
# ['sender-quarantine'], 'local:user-%m'
# ) if lookup(0,$sender, ['user1 at domain','user2 at domain']);
# section_time($which_section);
+ $elapsed{'TimeElapsedQuarantineAndNotify'} = Time::HiRes::time - $t0_sect;
if (defined $hold && $hold ne '')
{ do_log(-1, "NOTICE: HOLD reason: %s", $hold) }
# THIRD: now that we know what to do with it, do it! (deliver or bounce)
- my($ccat_name) =
- $msginfo->setting_by_contents_category(\%ccat_display_names);
- snmp_count('Content'.$ccat_name.'Msgs');
+ { # update Content*Msgs* counters
+ my($ccat_name) =
+ $msginfo->setting_by_contents_category(\%ccat_display_names);
+ my($counter_name) = 'Content'.$ccat_name.'Msgs';
+ snmp_count($counter_name);
+ snmp_count($counter_name.'Outbound') if $cnt_remote > 0;
+ if ($cnt_local > 0) {
+ snmp_count($counter_name.'InboundOrInt');
+ snmp_count($counter_name .(c('originating') ? 'Internal' : 'Inbound'));
+ }
+ }
+ if (ref $custom_object) {
+ $which_section = "custom-before_send";
+ eval {
+ $custom_object->before_send($conn,$msginfo); 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ do_log(-1,"custom before_send error: %s", $eval_stat);
+ };
+ section_time($which_section);
+ }
+ my($bcc)= $msginfo->setting_by_contents_category(cr('always_bcc_by_ccat'));
+ if (defined $bcc && $bcc ne '') {
+ my($recip_obj) = Amavis::In::Message::PerRecip->new;
+ # leave recip_addr and recip_addr_smtp undefined!
+ $recip_obj->recip_addr_modified($bcc);
+ $recip_obj->recip_destiny(D_PASS);
+ $recip_obj->dsn_notify(['NEVER']);
+ $recip_obj->contents_category($msginfo->contents_category);
+ # $recip_obj->contents_category(CC_CLEAN);
+ $msginfo->per_recip_data([@{$msginfo->per_recip_data}, $recip_obj]);
+ do_log(2,"adding recipient - always_bcc: %s", $bcc);
+ }
my($hdr_edits) = $msginfo->header_edits;
- if (!$hdr_edits) {
- $hdr_edits = Amavis::Out::EditHeader->new;
- $msginfo->header_edits($hdr_edits);
- }
if ($msginfo->delivery_method eq '') { # AM.PDP or AM.CL (milter)
$which_section = "AM.PDP headers";
- ensure_mime_entity($msginfo, $fh, $tempdir, \@virusname, $parts_root);
+ # ensure_mime_entity($msginfo, $fh, $tempdir, $parts_root);
$hdr_edits = add_forwarding_header_edits_common(
$conn, $msginfo, $hdr_edits, $hold, $any_undecipherable,
- $virus_presence_checked, $spam_presence_checked, undef);
+ $virus_presence_checked, $spam_presence_checked);
my($done_all);
my($recip_cl); # ref to a list of similar recip objects
($hdr_edits, $recip_cl, $done_all) =
add_forwarding_header_edits_per_recip(
$conn, $msginfo, $hdr_edits, $hold, $any_undecipherable,
- $virus_presence_checked, $spam_presence_checked, undef, undef);
+ $virus_presence_checked, $spam_presence_checked, undef);
$msginfo->header_edits($hdr_edits); # store edits (redundant)
if (@$recip_cl && !$done_all) {
do_log(-1, "AM.PDP: CLIENTS REQUIRE DIFFERENT HEADERS");
@@ -8401,89 +9295,84 @@ sub check_mail($$$) {
} elsif (grep { !$_->recip_done } @{$msginfo->per_recip_data}) { # forward
# To be delivered explicitly - only to those recipients not yet marked
# as 'done' by the above content filtering sections.
- $which_section = "forwarding";
- ensure_mime_entity($msginfo, $fh, $tempdir, \@virusname, $parts_root);
- # an ad-hoc solution - poor-man's defanging of dangerous contents
- my($mail_defanged); # nonempty indicates mail body is replaced
- my(@explanation);
- my(@df_pairs) =
- $msginfo->setting_by_contents_category_all(cr('defang_by_ccat'));
- for my $pair (@df_pairs) {
- my($df,$cc) = @$pair; do_log(4,'defang? ccat "%s": %s', $cc,$df);
- next if !$df;
- push(@explanation, 'WARNING: contains virus '.join(' ', at virusname))
- if cmp_ccat_maj($cc,CC_VIRUS)==0;
- push(@explanation, 'WARNING: contains banned part')
- if cmp_ccat_maj($cc,CC_BANNED)==0;
- if (cmp_ccat_maj($cc,CC_UNCHECKED)==0) {
- if ($hold ne '') {
- push(@explanation,
- "WARNING: NOT CHECKED FOR VIRUSES (mail bomb?):\n $hold");
- } elsif ($any_undecipherable) {
- push(@explanation, "WARNING: contains undecipherable part");
- }
- }
- push(@explanation,
- split(/\n/, wrap_string(
- 'WARNING: bad headers - '.join(' ', at bad_headers), 78,'',' ')
- )) if cmp_ccat_maj($cc,CC_BADH)==0;
- push(@explanation, 'WARNING: oversized')
- if cmp_ccat_maj($cc,CC_OVERSIZED)==0;
- push(@explanation, split(/\n/, $msginfo->spam_summary))
- if cmp_ccat_maj($cc,CC_SPAM)==0 || cmp_ccat_maj($cc,CC_SPAMMY)==0
- }
- if (@explanation) { # mail needs to be defanged
- my($s) = join(' ', at explanation);
- do_log(1, "DEFANGING MAIL: %s",
- length($s) <= 150 ? $s : substr($s,0,150-3)."...");
- for (@explanation)
- { if (length($_) > 100) { $_ = substr($_,0,100-3) . "..." } }
- $_ .= "\n" for (@explanation); # append newlines
- my($d) = defanged_mime_entity($conn,$msginfo,\@explanation);
- $msginfo->mail_text($d); # substitute mail with rewritten version
- $msginfo->mail_text_fn(undef); # remove filename information
- $mail_defanged = 'Original mail moved to attachment (defanged)';
- # defanged_mime_entity have probably modifed header edits, refetch
- $hdr_edits = $msginfo->header_edits;
- my($resend_m) = c('resend_method');
- $msginfo->delivery_method($resend_m) if $resend_m ne '';
- section_time('defang');
- }
+ $which_section = "forwarding"; $t0_sect = Time::HiRes::time;
+ $snmp_db->register_proc(2,0,'F',$am_id) if defined $snmp_db; # forwardng
+ # ensure_mime_entity($msginfo, $fh, $tempdir, $parts_root);
$hdr_edits = add_forwarding_header_edits_common(
$conn, $msginfo, $hdr_edits, $hold, $any_undecipherable,
- $virus_presence_checked, $spam_presence_checked, $mail_defanged);
+ $virus_presence_checked, $spam_presence_checked);
for (;;) { # do the delivery
my($r_hdr_edits) = Amavis::Out::EditHeader->new; # per-recip edits set
$r_hdr_edits->inherit_header_edits($hdr_edits);
my($done_all);
- my($recip_cl); # ref to a list of similar recip objects
+ my($recip_cl); # ref to a list of recip objects needing same mail edits
+
+ # prepare header edits, clusterize
($r_hdr_edits, $recip_cl, $done_all) =
add_forwarding_header_edits_per_recip(
$conn, $msginfo, $r_hdr_edits, $hold, $any_undecipherable,
- $virus_presence_checked, $spam_presence_checked,
- $mail_defanged, undef);
+ $virus_presence_checked, $spam_presence_checked, undef);
last if !@$recip_cl;
$msginfo->header_edits($r_hdr_edits); # store edits
+
+ # preserve information that may be changed by prepare_modified_mail()
+ my($m_t,$m_tfn) = ($msginfo->mail_text, $msginfo->mail_text_fn);
+ my($m_dm) = $msginfo->delivery_method;
+ # mail body mangling/defanging/sanitizing
+ my($body_modified) = prepare_modified_mail($conn,$msginfo,
+ $hold,$any_undecipherable,$recip_cl);
+ # defanged_mime_entity have modifed header edits, refetch just in case
+ $r_hdr_edits = $msginfo->header_edits;
+ if ($body_modified) {
+ my($resend_m) = c('resend_method');
+ do_log(3, "mail body mangling in effect, %s", $resend_m);
+ $msginfo->delivery_method($resend_m) if $resend_m ne '';
+ }
mail_dispatch($conn, $msginfo, 0, $dsn_per_recip_capable,
sub { my($r) = @_; grep { $_ eq $r } @$recip_cl });
+ # close and delete replacement file, if any
+ my($tmp_fh) = $msginfo->mail_text; # replacement file, to be removed
+ if ($tmp_fh && !$tmp_fh->isa('MIME::Entity') && $tmp_fh ne $m_t) {
+ $tmp_fh->close or do_log(-1,"Can't close replacement: %s", $!);
+ unlink($msginfo->mail_text_fn)
+ or do_log(-1,"Can't remove %s: %s", $msginfo->mail_text_fn, $!);
+ }
+ # restore temporarily modified settings
+ $msginfo->mail_text($m_t); $msginfo->mail_text_fn($m_tfn);
+ $msginfo->delivery_method($m_dm);
snmp_count('OutForwMsgs');
snmp_count('OutForwHoldMsgs') if $hold ne '';
$point_of_no_return = 1; # now past the point where mail was sent
last if $done_all;
}
+ # turn on CC_MTA in case of MTA trouble (e.g, rejected by MTA on fwding)
+ for my $r (@{$msginfo->per_recip_data}) {
+ my($smtp_resp) = $r->recip_smtp_response;
+ # skip successful deliveries and non- MTA-generated status codes
+ next if $smtp_resp =~ /^2/ || $r->recip_done != 2;
+ my($min_ccat) = $smtp_resp =~ /^5/ ? 2 : $smtp_resp =~ /^4/ ? 1 : 0;
+ $r->add_contents_category(CC_MTA,$min_ccat);
+ $msginfo->add_contents_category(CC_MTA,$min_ccat);
+ my($blocking_ccat) = sprintf("%d,%d", CC_MTA,$min_ccat);
+ $r->blocking_ccat($blocking_ccat) if !defined($r->blocking_ccat);
+ $msginfo->blocking_ccat($blocking_ccat)
+ if !defined($msginfo->blocking_ccat);
+ }
+ $elapsed{'TimeElapsedForwarding'} = Time::HiRes::time - $t0_sect;
}
prolong_timer($which_section);
- $which_section = "delivery-notification";
+ $which_section = "delivery-notification"; $t0_sect = Time::HiRes::time;
+ # generate a delivery status notification according to rfc3462 & rfc3464
+ my($notification,$suppressed) = delivery_status_notification(
+ $conn, $msginfo, $dsn_per_recip_capable, \%builtins);
my($ndn_needed);
($smtp_resp, $exit_code, $ndn_needed) =
- one_response_for_all($msginfo, $dsn_per_recip_capable, $am_id);
- do_log(4, "ndn_needed=%s, exit=%s, %s", $ndn_needed,$exit_code,$smtp_resp);
- $msginfo->add_contents_category(CC_TEMPFAIL,0) if $smtp_resp =~ /^4/;
- # generate delivery status notification according to rfc3462 & rfc3464
- my($notification,$suppressed) = delivery_status_notification(
- $conn, $msginfo, $dsn_per_recip_capable,
- $smtp_resp=~/^5/, \%builtins);
+ one_response_for_all($msginfo, $dsn_per_recip_capable,
+ $suppressed && !defined($notification) );
+ do_log(4, "notif=%s, suppressed=%d, ndn_needed=%s, exit=%s, %s",
+ defined $notification ? 'Y' : 'N', $suppressed,
+ $ndn_needed, $exit_code, $smtp_resp);
section_time('prepare-dsn');
if ($suppressed && !defined($notification)) {
$msginfo->dsn_sent(2); # would-be-bounced, but bounce was suppressed
@@ -8491,14 +9380,14 @@ sub check_mail($$$) {
mail_dispatch($conn, $notification, 1, 0);
snmp_count('OutDsnMsgs');
my($n_smtp_resp, $n_exit_code, $n_dsn_needed) =
- one_response_for_all($notification, 0, $am_id); # check status
- if ($n_smtp_resp =~ /^2/ && !$n_dsn_needed) { # dsn successful?
- $msginfo->dsn_sent(1); # mark the message as bounced
+ one_response_for_all($notification, 0); # check status
+ if ($n_smtp_resp =~ /^2/ && !$n_dsn_needed) { # dsn successful?
+ $msginfo->dsn_sent(1); # mark the message as bounced
$point_of_no_return = 2; # now past the point where DSN was sent
} elsif ($n_smtp_resp =~ /^4/) {
snmp_count('OutDsnTempFails');
die sprintf("temporarily unable to send DSN to <%s>: %s",
- $msginfo->sender_contact, $n_smtp_resp);
+ $msginfo->sender, $n_smtp_resp);
} else {
snmp_count('OutDsnRejects');
do_log(-1,"NOTICE: UNABLE TO SEND DSN to <%s>: %s",
@@ -8511,6 +9400,7 @@ sub check_mail($$$) {
# $notification->purge;
}
prolong_timer($which_section);
+ $elapsed{'TimeElapsedDSN'} = Time::HiRes::time - $t0_sect;
# generate customized log report at log level 0 - this is usually the
# only log entry interesting to administrators during normal operation
@@ -8521,9 +9411,13 @@ sub check_mail($$$) {
$mybuiltins{'T'} = $mybuiltins{'TESTSSCORES'};
my($y,$n,$f) = delivery_short_report($msginfo);
@mybuiltins{'D','O','N'} = ($y,$n,$f);
- my($strr) = expand(cr('log_templ'), \%mybuiltins);
- for my $logline (split(/[ \t]*\n/, $$strr)) {
- do_log(0, "%s", $logline) if $logline ne '';
+ my($ll) = 0; # log level for the main log entry
+# $ll = 1 if !@$n; # tame down the log level if all passed
+ if (ll($ll)) {
+ my($strr) = expand(cr('log_templ'), \%mybuiltins);
+ for my $logline (split(/[ \t]*\n/, $$strr)) {
+ do_log($ll, "%s", $logline) if $logline ne '';
+ }
}
}
# if (@virusname || $spam_level > 10) {
@@ -8563,17 +9457,12 @@ sub check_mail($$$) {
} else {
$mybuiltins{'O'} = $qrecip_addr;
$mybuiltins{'N'} = sprintf("%s:%s\n %s", $qrecip_addr,
- ($remote_mta eq '' ? '' : " $remote_mta said:"), $smtp_resp);
+ ($remote_mta eq '' ?'' :" [$remote_mta] said:"), $smtp_resp);
}
my(@b); @b = @{$r->banned_parts} if defined $r->banned_parts;
my($b_chopped) = @b > 2; @b = (@b[0,1],'...') if $b_chopped;
s/[ \t]{6,}/ ... /g for @b;
$mybuiltins{'F'} = \@b; # list of banned file names
- my($ccat_maj,$ccat_min) = $r->main_contents_category;
- $mybuiltins{'ccat_maj'} = $ccat_maj ne '' ? "$ccat_maj" : "0";
- $mybuiltins{'ccat_min'} = $ccat_min ne '' ? "$ccat_min" : "0";
- $mybuiltins{'ccat_name'} = $r->setting_by_contents_category(
- \%ccat_display_names);
my($dn) = $r->dsn_notify;
$mybuiltins{'dsn_notify'} =
uc(join(',', $sender eq '' ? 'NEVER' : !$dn ? 'FAILURE' : @$dn));
@@ -8581,23 +9470,14 @@ sub check_mail($$$) {
my($whitelisted) = $r->recip_whitelisted_sender;
my($boost) = $r->recip_score_boost;
$mybuiltins{'score_boost'} = 0+sprintf("%.3f",0+$boost);
- my($is_local,$tag_level,$tag2_level,$kill_level,$bypassed);
- $is_local = lookup(0,$recip, @{ca('local_domains_maps')});
+ my($tag_level,$tag2_level,$kill_level);
$tag_level = lookup(0,$recip, @{ca('spam_tag_level_maps')});
$tag2_level = lookup(0,$recip, @{ca('spam_tag2_level_maps')});
$kill_level = lookup(0,$recip, @{ca('spam_kill_level_maps')});
- $bypassed = lookup(0,$recip, @{ca('bypass_spam_checks_maps')});
- my($do_tag) = $is_local && !$bypassed &&
- ( $blacklisted || !defined $tag_level || $tag_level eq '' ||
- ($spam_level+$boost + ($whitelisted?-10:0) >= $tag_level) );
- # my($do_tag2) = $is_local && !$bypassed && !$whitelisted &&
- # ( $blacklisted ||
- # (defined $tag2_level && $spam_level+$boost >= $tag2_level) );
- # my($do_kill) = !$bypassed && !$whitelisted &&
- # ( $blacklisted ||
- # (defined $kill_level && $spam_level+$boost >= $kill_level) );
- my($do_tag2) = $r->is_in_contents_category(CC_SPAMMY) && $is_local;
- my($do_kill) = $r->is_in_contents_category(CC_SPAM);
+ my($is_local) = $r->recip_is_local;
+ my($do_tag) = $r->is_in_contents_category(CC_CLEAN,1);
+ my($do_tag2) = $r->is_in_contents_category(CC_SPAMMY);
+ my($do_kill) = $r->is_in_contents_category(CC_SPAM);
for ($do_tag,$do_tag2,$do_kill) { $_ = $_ ? 'Y' : '0' } # normalize
for ($is_local) { $_ = $_ ? 'L' : '0' } # normalize
for ($tag_level,$tag2_level,$kill_level) { $_ = 'x' if !defined($_) }
@@ -8613,6 +9493,45 @@ sub check_mail($$$) {
@mybuiltins{('0','1','2','k')} = ($is_local,$do_tag,$do_tag2,$do_kill);
# macros %3, %4, %5 are deprecated, replaced by tag/tag2/kill_level
@mybuiltins{('3','4','5')} = ($tag_level,$tag2_level,$kill_level);
+
+ $mybuiltins{'ccat'} =
+ sub {
+ my($name,$attr,$which) = @_;
+ $attr = lc($attr); # name | major | minor | <empty>
+ # | is_blocking | is_nonblocking
+ # | is_blocked_by_nonmain
+ $which = lc($which); # main | blocking | auto
+ my($result) = ''; my($blocking_ccat) = $r->blocking_ccat;
+ if ($attr eq 'is_blocking') {
+ $result = defined($blocking_ccat) ? '1' : '';
+ } elsif ($attr eq 'is_nonblocking') {
+ $result = !defined($blocking_ccat) ? '1' : '';
+ } elsif ($attr eq 'is_blocked_by_nonmain') {
+ if (defined($blocking_ccat)) {
+ my($aref) = $r->contents_category;
+ $result = '1' if ref($aref) && @$aref > 0
+ && $blocking_ccat ne $aref->[0];
+ }
+ } elsif ($attr eq 'name') {
+ $result =
+ $which eq 'main' ?
+ $r->setting_by_main_contents_category(\%ccat_display_names)
+ : $which eq 'blocking' ?
+ $r->setting_by_blocking_contents_category(
+ \%ccat_display_names)
+ : $r->setting_by_contents_category( \%ccat_display_names);
+ } else { # attr = major, minor, or anything else returns a pair
+ my($maj,$min) = ccat_split(
+ ($which eq 'blocking' ||
+ $which ne 'main' && defined $blocking_ccat)
+ ? $blocking_ccat : $r->contents_category);
+ $result = $attr eq 'major' ? $maj
+ : $attr eq 'minor' ? sprintf("%d",$min)
+ : sprintf("(%d,%d)",$maj,$min);
+ }
+ $result;
+ };
+
my($strr) = expand(cr('log_recip_templ'), \%mybuiltins);
for my $logline (split(/[ \t]*\n/, $$strr)) {
do_log(0, "%s", $logline) if $logline ne '';
@@ -8631,15 +9550,16 @@ sub check_mail($$$) {
$os_short = $1 if $os_short =~ /^(Windows [^ ]+|[^ ]+)/; # drop vers.
$os_short =~ s{[^0-9A-Za-z:./_+-]}{-}g; $os_short =~ s{\.}{,}g;
my($snmp_counter_name) = $msginfo->setting_by_contents_category(
- { CC_VIRUS,'virus', CC_BANNED,'banned',
- CC_SPAM,'spam', CC_SPAMMY,'spammy', CC_CATCHALL,'clean' });
+ { CC_VIRUS,'virus', CC_BANNED,'banned',
+ CC_SPAM,'spam', CC_SPAMMY,'spammy', CC_CATCHALL,'clean' });
if ($snmp_counter_name eq 'clean')
{ $snmp_counter_name = $spam_level<=$spam_ham_level ? 'ham' : undef }
if (defined $snmp_counter_name) {
snmp_count("$snmp_counter_name.byOS.$os_short");
do_log(3, 'Ham from Windows XP? Most weird! %s [%s] score=%.3f',
$mail_id, $cl_ip, $spam_level)
- if $snmp_counter_name eq 'ham' && $os_fingerprint =~ /^Windows XP/;
+ if $snmp_counter_name eq 'ham' &&
+ $os_fingerprint =~ /^Windows XP(?![^(]*\b2000 SP)/;
}
}
}
@@ -8663,16 +9583,38 @@ sub check_mail($$$) {
}
if (defined $snmp_db) {
$which_section = 'update_snmp';
+ my($log_lines, $log_warnings, $log_status_counts) = collect_log_stats();
+ snmp_count( ['LogLines', $log_lines, 'C64'] );
+ if ($log_warnings > 0) {
+ snmp_count( ['LogWarnings', $log_warnings] );
+ do_log(3,"Syslog warnings: %d x %s",
+ $log_status_counts->{$_}, $_) for (keys %$log_status_counts);
+ }
+ $elapsed{'TimeElapsedSending'} += # merge similar timing entries
+ delete $elapsed{$_} for ('TimeElapsedQuarantineAndNotify',
+ 'TimeElapsedForwarding', 'TimeElapsedDSN');
snmp_count( ['entropy',0,'STR'] );
+ $elapsed{'TimeElapsedTotal'} = Time::HiRes::time - $msginfo->rx_time;
+ snmp_count([$_, int(1000*$elapsed{$_}+0.5), 'C64']) for (keys %elapsed);
$snmp_db->update_snmp_variables;
section_time($which_section);
}
+ if (ref $custom_object) {
+ $which_section = "custom-mail_done";
+ eval {
+ $custom_object->mail_done($conn,$msginfo); 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ do_log(-1,"custom mail_done error: %s", $eval_stat);
+ };
+ section_time($which_section);
+ }
$which_section = 'finishing';
- }; # end eval
- if ($@ ne '') {
- chomp($@);
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
$preserve_evidence = 1;
- my($msg) = "$which_section FAILED: $@";
+ my($msg) = "$which_section FAILED: $eval_stat";
if ($point_of_no_return) {
do_log(-2, "TROUBLE in check_mail, but must continue (%s): %s",
$point_of_no_return,$msg);
@@ -8683,7 +9625,8 @@ sub check_mail($$$) {
for my $r (@{$msginfo->per_recip_data})
{ $r->recip_smtp_response($smtp_resp); $r->recip_done(1) }
}
- }
+ };
+
# if ($hold ne '') {
# do_log(-1, "NOTICE: Evidence is to be preserved: %s", $hold);
# $preserve_evidence = 1;
@@ -8704,7 +9647,7 @@ sub check_mail($$$) {
elsif ($dsn_sent==2) { $which_counter = 'InDiscards' }
}
snmp_count($which_counter);
- $snmp_db->register_proc('.') if defined $snmp_db; # content checking done
+ $snmp_db->register_proc(1,0,'.') if defined $snmp_db; # content checking done
undef $MSGINFO; undef $CONN; # release global references
($smtp_resp, $exit_code, $preserve_evidence);
}
@@ -8713,20 +9656,21 @@ sub check_mail($$$) {
# e.g. to construct notifications. While at it, also get us some additional
# information on sender from the header.
#
-sub ensure_mime_entity($$$$$) {
- my($msginfo, $fh, $tempdir, $virusname_list, $parts_root) = @_;
+sub ensure_mime_entity($$$$) {
+ my($msginfo, $fh, $tempdir, $parts_root) = @_;
+ my($ent,$mime_err);
if (!defined($msginfo->mime_entity)) {
# header may not have been parsed yet, e.g. if the result was cached
- my($ent,$mime_err) = mime_decode($fh, $tempdir, $parts_root);
+ ($ent,$mime_err) = mime_decode($fh, $tempdir, $parts_root);
$msginfo->mime_entity($ent);
- prolong_timer("ensure_mime_entity");
- }
-}
-
-sub add_forwarding_header_edits_common($$$$$$$$) {
+ prolong_timer("mime_decode");
+ }
+ $mime_err;
+}
+
+sub add_forwarding_header_edits_common($$$$$$$) {
my($conn, $msginfo, $hdr_edits, $hold, $any_undecipherable,
- $virus_presence_checked, $spam_presence_checked,
- $mail_defanged) = @_;
+ $virus_presence_checked, $spam_presence_checked) = @_;
my($allowed_hdrs) = cr('allowed_added_header_fields');
$hdr_edits->add_header('X-Quarantine-ID', '<'.$msginfo->mail_id.'>')
if defined($msginfo->quarantined_to) &&
@@ -8740,14 +9684,8 @@ sub add_forwarding_header_edits_common($
# example on how to remove subject tag inserted by some other MTA:
# $hdr_edits->edit_header('Subject',
# sub { my($h,$s)=@_; $s=~s/^\s*\*\*\* SPAM \*\*\*(.*)/$1/s; $s });
- if ($mail_defanged ne '' &&
- $allowed_hdrs && $allowed_hdrs->{lc('X-Amavis-Modified')}) {
- my($msg) = "$mail_defanged by " . c('myhostname');
- $hdr_edits->add_header('X-Amavis-Modified', $msg);
- do_log(1, "Inserting header field: X-Amavis-Modified: %s", $msg);
- }
if ($extra_code_antivirus) {
- $hdr_edits->delete_header('X-Amavis-Alert');
+ # $hdr_edits->delete_header('X-Amavis-Alert'); # it does not hurt to keep it
$hdr_edits->delete_header(c('X_HEADER_TAG'))
if c('remove_existing_x_scanned_headers') &&
(c('X_HEADER_LINE') ne '' && c('X_HEADER_TAG') =~ /^[!-9;-\176]+\z/);
@@ -8767,25 +9705,6 @@ sub add_forwarding_header_edits_common($
$Mail::SpamAssassin::SUB_VERSION, c('myhostname')))
if $allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Checker-Version')};
}
- if ($mail_defanged ne '') {
- # prepend Resent-* header fields, they must precede corresponding Received
- my($hdrfrom_recip) = $msginfo->setting_by_contents_category(
- cr('hdrfrom_notify_recip_by_ccat'));
- $hdrfrom_recip = expand_variables($hdrfrom_recip);
- $hdr_edits->append_header_above_received('Resent-From',$hdrfrom_recip);
- $hdr_edits->append_header_above_received('Resent-Date',
- rfc2822_timestamp($msginfo->rx_time));
- $hdr_edits->append_header_above_received('Resent-Message-ID',
- sprintf('<RE%s@%s>', $msginfo->mail_id, c('myhostname')) );
- # make DKIM/DomainKeys happy, add Sender, existing ones were removed
- $hdr_edits->append_header_above_received('Sender',$hdrfrom_recip)
- if $hdrfrom_recip ne '';
- }
- # misnomer, this _is_ the Received line
- $hdr_edits->append_header_above_received('Received',
- received_line($conn,$msginfo,$msginfo->mail_id,1), 1)
- if c('insert_received_line') && $msginfo->delivery_method ne '' &&
- $allowed_hdrs && $allowed_hdrs->{lc('Received')};
$hdr_edits;
}
@@ -8794,10 +9713,9 @@ sub add_forwarding_header_edits_common($
# that are receiving the same set of header edits (so the message may be
# delivered to them in one SMTP transaction).
#
-sub add_forwarding_header_edits_per_recip($$$$$$$$$) {
+sub add_forwarding_header_edits_per_recip($$$$$$$$) {
my($conn, $msginfo, $hdr_edits, $hold, $any_undecipherable,
- $virus_presence_checked, $spam_presence_checked,
- $mail_defanged, $filter) = @_;
+ $virus_presence_checked, $spam_presence_checked, $filter) = @_;
my(@recip_cluster);
my(@per_recip_data) = grep { !$_->recip_done && (!$filter || &$filter($_)) }
@{$msginfo->per_recip_data};
@@ -8813,65 +9731,65 @@ sub add_forwarding_header_edits_per_reci
if ($os_fp ne '' && $msginfo->client_addr ne '')
{ $os_fp .= ', [' . $msginfo->client_addr . ']' }
for my $r (@per_recip_data) {
- my($recip) = $r->recip_addr;
- my($is_local, $blacklisted, $whitelisted, $boost,
- $tag_level, $tag2_level, $do_tag, $do_tag2, $do_kill,
- $do_tag_virus_checked, $do_tag_virus, $do_tag_banned, $do_tag_badh,
- $subject_tag, $do_p0f, $pp_age);
- $is_local = lookup(0,$recip, @{ca('local_domains_maps')});
- $blacklisted = $r->recip_blacklisted_sender;
- $whitelisted = $r->recip_whitelisted_sender;
- $boost = $r->recip_score_boost;
- $do_tag2 = $r->is_in_contents_category(CC_SPAMMY) && $is_local;
- $do_kill = $r->is_in_contents_category(CC_SPAM);
- $do_tag_badh = $r->is_in_contents_category(CC_BADH);
- $do_tag_banned= $r->is_in_contents_category(CC_BANNED);
- $do_tag_virus = $r->is_in_contents_category(CC_VIRUS);
- $do_tag_virus_checked = $adding_x_header_tag && defined($r->infected);
- $do_p0f = $is_local && $os_fp ne '' &&
- $allowed_hdrs && $allowed_hdrs->{lc('X-Amavis-OS-Fingerprint')};
+ my($recip) = $r->recip_addr;
+ my($is_local) = $r->recip_is_local;
+ my($blacklisted) = $r->recip_blacklisted_sender;
+ my($whitelisted) = $r->recip_whitelisted_sender;
+ my($boost) = $r->recip_score_boost;
+ my($do_tag) = $r->is_in_contents_category(CC_CLEAN,1);
+ my($do_tag2) = $r->is_in_contents_category(CC_SPAMMY);
+ my($do_kill) = $r->is_in_contents_category(CC_SPAM);
+ my($do_tag_badh) = $r->is_in_contents_category(CC_BADH);
+ my($do_tag_banned)= $r->is_in_contents_category(CC_BANNED);
+ my($do_tag_virus) = $r->is_in_contents_category(CC_VIRUS);
+ my($mail_mangle) = $r->mail_body_mangle;
+ my($do_tag_virus_checked) = $adding_x_header_tag && defined($r->infected);
+ my($do_p0f) = $is_local && $os_fp ne '' &&
+ $allowed_hdrs && $allowed_hdrs->{lc('X-Amavis-OS-Fingerprint')};
+ my($tag_level, $tag2_level, $subject_tag, $pp_age);
if ($allowed_hdrs && $allowed_hdrs->{lc('X-Amavis-PenPals')}) {
$pp_age = $r->recip_penpals_age;
$pp_age = format_time_interval($pp_age) if defined $pp_age;
}
if ($extra_code_antispam) {
- my($bypassed);
- $bypassed = lookup(0,$recip, @{ca('bypass_spam_checks_maps')});
- $tag_level = lookup(0,$recip, @{ca('spam_tag_level_maps')});
- $tag2_level = lookup(0,$recip, @{ca('spam_tag2_level_maps')});
- $do_tag = $is_local && !$bypassed &&
- ( $blacklisted || !defined $tag_level || $tag_level eq '' ||
- ($spam_level+$boost + ($whitelisted?-10:0) >= $tag_level) );
+ $tag_level = lookup(0,$recip, @{ca('spam_tag_level_maps')});
+ $tag2_level = lookup(0,$recip, @{ca('spam_tag2_level_maps')});
}
if ($is_local) {
- my(@subj_maps_pairs) =
- $r->setting_by_contents_category_all(cr('subject_tag_maps_by_ccat'));
+ my(@subj_maps_pairs) = $r->setting_by_main_contents_category_all(
+ cr('subject_tag_maps_by_ccat'));
my($spam_may_modify) = lookup(0,$recip,@{ca('spam_modifies_subj_maps')});
for my $pair (@subj_maps_pairs) {
- my($map_ref,$cc) = @$pair;
+ my($cc,$map_ref) = @$pair;
next if !ref($map_ref);
$subject_tag = lookup(0,$recip,@$map_ref)
if $spam_may_modify ||
- !(cmp_ccat_maj($cc,CC_SPAM)==0 ||
- cmp_ccat_maj($cc,CC_SPAMMY)==0 ||
+ !(ccat_maj($cc)==CC_SPAM || ccat_maj($cc)==CC_SPAMMY ||
cmp_ccat($cc, CC_CLEAN.",1")==0);
last if $subject_tag ne ''; # take the first nonempty string
}
}
if ($subject_tag ne '') { # expand subject template
- $subject_tag =~ s{_(SCORE|REQD|YESNO|YESNOCAPS)_}
+ # just implement a small subset of macro-lookalikes, not true macro calls
+ $subject_tag =~
+ s{_(SCORE|REQD|YESNO|YESNOCAPS|HOSTNAME|DATE|U|LOGID|MAILID)_}
{ $1 eq 'SCORE' ? (0+sprintf("%.3f",$spam_level+$boost))
: $1 eq 'REQD' ? (!defined($tag2_level) ? '-' :
0+sprintf("%.3f",$tag2_level))
: $1 eq 'YESNO' ? ($do_tag2 ? 'Yes' : 'No')
: $1 eq 'YESNOCAPS' ? ($do_tag2 ? 'YES' : 'NO')
- : '_'.$1.'_' }egs;
+ : $1 eq 'HOSTNAME' ? c('myhostname')
+ : $1 eq 'DATE' ? rfc2822_timestamp($msginfo->rx_time)
+ : $1 eq 'U' ? iso8601_utc_timestamp($msginfo->rx_time)
+ : $1 eq 'LOGID' ? $msginfo->log_id
+ : $1 eq 'MAILID' ? $msginfo->mail_id
+ : '_'.$1.'_' }egsx;
}
# normalize
$_ = $_?1:0 for ($do_tag_virus_checked, $do_tag_virus, $do_tag_banned,
$do_tag_badh, $do_tag, $do_tag2, $do_p0f, $is_local);
my($spam_level_bar, $full_spam_status);
- if ($do_tag || $do_tag2) { # prepare status and level bar
+ if ($is_local && ($do_tag || $do_tag2)) { # prepare status and level bar
# spam-related headers should _not_ be inserted for:
# - nonlocal recipients (outgoing mail), as a matter of courtesy
# to our users;
@@ -8887,12 +9805,15 @@ sub add_forwarding_header_edits_per_reci
: 0+$spam_level+$boost) if $slc ne '';
my($s) = $msginfo->spam_status;
$s =~ s/,/,\n /g; # allow header field wrapping at any comma
- my($sl) = !defined($spam_level) ? 'x'
- : 0+sprintf("%.3f",$spam_level); # trim fraction
- my($bl) = !defined($boost) ? undef : 0+sprintf("%.3f",$boost);
+ ## some MUAs interpret the score and don't like m+n syntax, so avoid it
+ # my($sl) = !defined($spam_level) ? 'x'
+ # : 0+sprintf("%.3f",$spam_level); # trim fraction
+ # my($bl) = !defined($boost) ? undef : 0+sprintf("%.3f",$boost);
+ # (!defined($boost) || $bl==0 ? $sl : $bl>=0 ? $sl.'+'.$bl : $sl.$bl),
$full_spam_status = sprintf("%s,\n score=%s\n %s%s%stests=[%s]",
$do_tag2 ? 'Yes' : 'No',
- (!defined($boost) || $bl==0 ? $sl : $bl>=0 ? $sl.'+'.$bl : $sl.$bl),
+ !defined($spam_level) && !defined($boost) ? 'x' :
+ 0+sprintf("%.3f",$spam_level+$boost),
!defined $tag_level || $tag_level eq '' ? ''
: sprintf("tagged_above=%s\n ",$tag_level),
!defined $tag2_level ? '' : sprintf("required=%s\n ", $tag2_level),
@@ -8902,36 +9823,64 @@ sub add_forwarding_header_edits_per_reci
}
my($key) = join("\000", map {defined $_ ? $_ : ''} (
$do_tag_virus_checked, $do_tag_virus, $do_tag_banned, $do_tag_badh,
- $do_tag, $do_tag2, $subject_tag, $spam_level_bar, $full_spam_status,
- $do_p0f, $pp_age) );
+ $do_tag && $is_local, $do_tag2 && $is_local, $subject_tag,
+ $spam_level_bar, $full_spam_status, $mail_mangle, $do_p0f, $pp_age) );
if ($first) {
if (ll(4)) {
my($sl) = !defined($spam_level) ? 'x'
: 0+sprintf("%.3f",$spam_level); # trim fraction
do_log(4, "headers CLUSTERING: NEW CLUSTER <%s>: score=%s, ".
- "tag=%s, tag2=%s, local=%s, bl=%s, s=%s",
- $recip,
+ "tag=%s, tag2=%s, local=%s, bl=%s, s=%s, mangle=%s", $recip,
(!defined $boost || $boost==0 ? $sl
: $boost >= 0 ? $sl.'+'.$boost : $sl.$boost),
- $do_tag, $do_tag2, $is_local, $blacklisted, $subject_tag);
+ $do_tag, $do_tag2, $is_local, $blacklisted, $subject_tag,
+ $mail_mangle);
}
$cluster_key = $key; $cluster_full_spam_status = $full_spam_status;
} elsif ($key eq $cluster_key) {
do_log(5,"headers CLUSTERING: <%s> joining cluster", $recip);
} else {
- do_log(5,"headers CLUSTERING: skipping <%s> (t=%s, t2=%s)",
- $recip,$do_tag,$do_tag2);
+ do_log(5,"headers CLUSTERING: skipping <%s> (t=%s, t2=%s, l=%s)",
+ $recip,$do_tag,$do_tag2,$is_local);
next; # this recipient will be handled in some later pass
}
if ($first) { # insert headers required for the new cluster
+ if ($mail_mangle) { # mail body modified, invalidates DKIM signature
+ if ($allowed_hdrs && $allowed_hdrs->{lc('X-Amavis-Modified')}) {
+ $hdr_edits->append_header_above_received('X-Amavis-Modified',
+ sprintf("Mail body modified (%s) by %s",
+ length($mail_mangle) > 1 ? "using $mail_mangle" : "defanged",
+ c('myhostname') ));
+ }
+ if ($msginfo->orig_header_fields->{lc('DKIM-Signature')} ne '' ||
+ $msginfo->orig_header_fields->{lc('DomainKey-Signature')} ne '') {
+ # only bother inserting Resent-* header fields if it is likely
+ # that we are invalidating an existing signature;
+ # prepend Resent-* fields, they must precede corresponding Received
+ my($hdrfrom_recip) = $msginfo->setting_by_contents_category(
+ cr('hdrfrom_notify_recip_by_ccat'));
+ $hdrfrom_recip = expand_variables($hdrfrom_recip);
+ $hdr_edits->append_header_above_received('Resent-From',
+ $hdrfrom_recip);
+ $hdr_edits->append_header_above_received('Resent-Date',
+ rfc2822_timestamp($msginfo->rx_time));
+ $hdr_edits->append_header_above_received('Resent-Message-ID',
+ sprintf('<RE%s@%s>', $msginfo->mail_id, c('myhostname')) );
+ # make DKIM/DomainKeys happy, add Sender, existing ones were removed
+ $hdr_edits->append_header_above_received('Sender',$hdrfrom_recip)
+ if $hdrfrom_recip ne '';
+ }
+ }
if ($do_tag_virus_checked) {
$hdr_edits->add_header(c('X_HEADER_TAG'), c('X_HEADER_LINE'));
}
if ($allowed_hdrs && $allowed_hdrs->{lc('X-Amavis-Alert')}) {
if ($do_tag_virus) {
+ my($virusname_list) = $msginfo->virusnames;
$hdr_edits->add_header('X-Amavis-Alert',
- "INFECTED, message contains virus: " . join(", ", at virusname));
+ "INFECTED, message contains virus: " .
+ (!defined($virusname_list) ? '' : join(", ",@$virusname_list)) );
}
if ($do_tag_banned) {
my(@b); @b = @{$r->banned_parts} if defined $r->banned_parts;
@@ -8946,21 +9895,27 @@ sub add_forwarding_header_edits_per_reci
'BAD HEADER, ' . $bad_headers[0]);
}
}
- if ($do_tag2 && $allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Flag')}) {
- $hdr_edits->add_header('X-Spam-Flag', 'YES');
- }
- if ($do_tag) {
- $hdr_edits->add_header('X-Spam-Score',
- !defined $spam_level ? '-' : 0+sprintf("%.3f",0+$spam_level+$boost))
- if $allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Score')};
+ if ($do_tag && $is_local) {
+ $hdr_edits->add_header('X-Spam-Flag', $do_tag2 ? 'YES' : 'NO')
+ if $allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Flag')};
+ if ($allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Score')}) {
+ my($score) = 0+$spam_level+$boost;
+ $score = max(64,$score) if $blacklisted; # don't go below 64 if bl
+ $score = min( 0,$score) if $whitelisted; # don't go above 0 if wl
+ $hdr_edits->add_header('X-Spam-Score', 0+sprintf("%.3f",$score));
+ }
$hdr_edits->add_header('X-Spam-Level', $spam_level_bar)
if defined $spam_level_bar &&
$allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Level')};
$hdr_edits->add_header('X-Spam-Status', $full_spam_status, 1)
if $allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Status')};
}
- if ($do_tag2) {
- $hdr_edits->add_header('X-Spam-Report', "\n".$msginfo->spam_report, 2)
+ if ($do_tag2 && $is_local) {
+ # SA reports may contain any octet, i.e. 8-bit data from a mail
+ # that is reported by a matching rule; no charset is associated,
+ # so it doesn't make sense to RFC2047-encode it, so just sanitize it
+ $hdr_edits->add_header('X-Spam-Report',
+ "\n".sanitize_str($msginfo->spam_report,1), 2)
if c('sa_spam_report_header') && $msginfo->spam_report ne '' &&
$allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Report')};
}
@@ -8978,20 +9933,52 @@ sub add_forwarding_header_edits_per_reci
sanitize_str($os_fp)) if $do_p0f;
$hdr_edits->add_header('X-Amavis-PenPals',
'age '.$pp_age) if defined $pp_age;
+ # misnomer, this _is_ the Received line
+ $hdr_edits->append_header_above_received('Received',
+ received_line($conn,$msginfo,$msginfo->mail_id,1), 1)
+ if c('insert_received_line') && $msginfo->delivery_method ne '' &&
+ $allowed_hdrs && $allowed_hdrs->{lc('Received')};
}
push(@recip_cluster,$r); $first = 0;
my($delim) = c('recipient_delimiter');
- if ($delim ne '' && $is_local) {
+ if ($is_local) {
+ # rewrite/replace recipient addresses, possibly with multiple recipients
+ my($rewrite_map) = $r->setting_by_contents_category(
+ cr('addr_rewrite_maps_by_ccat'));
+ my($rewrite) = !ref($rewrite_map) ?undef :lookup(0,$recip,@$rewrite_map);
+ if ($rewrite ne '') {
+ my(@replacements) = grep { $_ ne '' }
+ map { /^ [ \t]* (.*?) [ \t]* \z/sx; $1 } split(/,/, $rewrite, -1);
+ if (@replacements) {
+ my($repl_addr) = shift @replacements;
+ my($modif_addr) = replace_addr_fields($recip,$repl_addr,$delim);
+ ll(5) && do_log(5,"addr_rewrite_maps: replacing <%s> by <%s>",
+ $recip,$modif_addr);
+ $r->recip_addr_modified($modif_addr);
+ for my $bcc (@replacements) { # remaining addresses are extra Bcc
+ my($new_addr) = replace_addr_fields($recip,$bcc,$delim);
+ ll(5) && do_log(5,"addr_rewrite_maps: recip <%s>, adding <%s>",
+ $recip,$new_addr);
+ # my($clone) = $r->clone;
+ # $clone->recip_addr_modified($new_addr);
+ }
+ }
+ $r->dsn_orcpt(orcpt_encode($r->recip_addr_smtp))
+ if !defined($r->dsn_orcpt);
+ }
+ }
+ if ($is_local && $delim ne '') {
# append address extensions to mailbox names if desired
my($ext_map) = $r->setting_by_contents_category(
cr('addr_extension_maps_by_ccat'));
my($ext) = !ref($ext_map) ? undef : lookup(0,$recip, @$ext_map);
if ($ext ne '') {
+ $ext = $delim . $ext;
my($orig_extension); my($localpart,$domain) = split_address($recip);
($localpart,$orig_extension) = split_localpart($localpart,$delim)
if c('replace_existing_extension'); # strip existing extension
- my($new_addr) = $localpart.$delim.$ext.$domain;
+ my($new_addr) = $localpart.$ext.$domain;
if (ll(5)) {
if (!defined($orig_extension)) {
do_log(5, "appending addr ext '%s', giving '%s'", $ext,$new_addr);
@@ -9005,7 +9992,7 @@ sub add_forwarding_header_edits_per_reci
# RCPT command when the message is relayed. If an ORCPT parameter is
# added by the relaying MTA, it MUST contain the recipient address
# from the RCPT command used when the message was received by that MTA.
- $r->dsn_orcpt('rfc822;'.xtext_encode(quote_rfc2821_local($recip)))
+ $r->dsn_orcpt(orcpt_encode($r->recip_addr_smtp))
if !defined($r->dsn_orcpt);
$r->recip_addr_modified($new_addr);
}
@@ -9019,15 +10006,201 @@ sub add_forwarding_header_edits_per_reci
} else {
ll(4) && do_log(4, "headers CLUSTERING: got %d recips out of %d: %s",
scalar(@recip_cluster), $per_recip_data_len,
- join(", ", map { "<" . $_->recip_addr . ">" } @recip_cluster) );
+ join(", ", map { $_->recip_addr_smtp } @recip_cluster));
}
if (ll(2) && defined($cluster_full_spam_status) && @recip_cluster) {
my($s) = $cluster_full_spam_status; $s =~ s/\n[ \t]/ /g;
- do_log(2, "SPAM-TAG, %s -> %s, %s", qquote_rfc2821_local($msginfo->sender),
- join(',', qquote_rfc2821_local(
- map { $_->recip_addr } @recip_cluster)), $s);
+ do_log(2, "SPAM-TAG, %s -> %s, %s", $msginfo->sender_smtp,
+ join(',', map { $_->recip_addr_smtp } @recip_cluster), $s);
}
($hdr_edits, \@recip_cluster, $done_all);
+}
+
+# Mail body mangling (defanging, sanitizing or adding disclaimers);
+# Prepare mail body replacement for the first recipient
+# in the @$per_recip_data list (which contains a subset or recipients
+# with the same mail edits, to be dispatched next as a single message)
+#
+sub prepare_modified_mail($$$$$) {
+ my($conn, $msginfo, $hold, $any_undecipherable, $per_recip_data) = @_;
+ my($body_modified) = 0;
+ for my $r (@$per_recip_data) { # a subset of recipients!
+ my($recip) = $r->recip_addr;
+ my($mail_mangle) = $r->mail_body_mangle; my($actual_mail_mangle);
+ if (!$mail_mangle) {
+ # skip
+ } elsif ($mail_mangle =~ /^(?:null|nulldisclaimer)\z/i) { # for testing
+ $body_modified = 1; # pretend mail was modified while actually it was not
+ section_time('mangle-'.$mail_mangle);
+ } elsif (( lc($mail_mangle) ne 'attach' &&
+ ($enable_anomy_sanitizer || $altermime ne '') )
+ || $mail_mangle =~ /^(?:anomy|altermime|disclaimer)\z/i) {
+ do_log(2,"mangling by: %s, <%s>", $mail_mangle,$recip);
+ my($orig_fn) = $msginfo->mail_text_fn;
+ my($repl_fn) = $msginfo->mail_tempdir . '/email-repl.txt';
+ my($inp_fh) = $msginfo->mail_text; my($out_fh);
+ my($repl_size);
+ eval {
+ $inp_fh->seek(0,0) or die "Can't rewind mail file: $!";
+ $out_fh = IO::File->new;
+ $out_fh->open($repl_fn, O_CREAT|O_EXCL|O_RDWR, 0640)
+ or die "Can't create file $repl_fn: $!";
+ binmode($out_fh,":bytes") or die "Can't cancel :utf8 mode: $!"
+ if $unicode_aware;
+ if ($enable_anomy_sanitizer &&
+ $mail_mangle !~ /^(?:altermime|disclaimer)\z/) {
+ $actual_mail_mangle = 'anomy';
+ $enable_anomy_sanitizer or die "Anomy not available: $mail_mangle";
+ my(@scanner_conf); my($e); my($engine) = Anomy::Sanitizer->new;
+ if ($e = $engine->error) { die $e }
+ $engine->configure(@scanner_conf, @{ca('anomy_sanitizer_args')});
+ if ($e = $engine->error) { die $e }
+ my($ret) = $engine->sanitize($msginfo->mail_text, $out_fh);
+ if ($e = $engine->error) { die $e }
+ } else { # use altermime for adding disclaimers or defanging
+ $actual_mail_mangle = 'altermime';
+ $altermime ne '' or die "altermime not available: $mail_mangle";
+ # prepare arguments to altermime
+ my(@altermime_args); my($disclaimer_options);
+ if (lc($mail_mangle) ne 'disclaimer') { # defang: no by-sender opts.
+ @altermime_args = @{ca('altermime_args_defang')};
+ } else { # disclaimer
+ @altermime_args = @{ca('altermime_args_disclaimer')};
+ my($opt_maps) = ca('disclaimer_options_bysender_maps');
+ if (defined($opt_maps) && @$opt_maps && # by sender options?
+ grep(/_OPTION_/, at altermime_args)) {
+ # find disclaimer options pertaining to the
+ # most appropriate sender/author/return-path address
+ my($f) = $msginfo->rfc2822_from;
+ my(@rfc2822_from) = ref $f ? @$f : $f;
+ for my $pair ( [$msginfo->rfc2822_sender, '2822.Sender'],
+ (map { [$_, '2822.From'] } @rfc2822_from),
+ [$msginfo->sender, '2821.mail_from']) {
+ my($addr,$addr_type) = @$pair;
+ do_log(5,"disclaimer options lookup (%s) %s",$addr_type,$addr);
+ next if !defined($addr) || $addr eq '';
+ next if !lookup(0,$addr,@{ca('local_domains_maps')});
+ my($opt,$matchingkey) = lookup(0,$addr,@$opt_maps);
+ if (defined $opt) {
+ $disclaimer_options = $opt;
+ do_log(3,"disclaimer options pertaining to (%s) %s: %s",
+ $addr_type, $addr, $disclaimer_options);
+ last;
+ }
+ }
+ s/_OPTION_/$disclaimer_options/gs for @altermime_args;
+ }
+ }
+ ### copy original mail to $repl_fn, altermime can't handle stdin well
+ my($nbytes,$buff);
+ while (($nbytes=$inp_fh->read($buff,65536)) > 0)
+ { $out_fh->print($buff) or die "Error writing to $repl_fn: $!" }
+ defined $nbytes or die "Error reading mail file: $!";
+ $out_fh->close or die "Can't close file $repl_fn: $!";
+ undef $out_fh;
+ my($proc_fh,$pid) = run_command(undef, "&1", $altermime,
+ "--input=$repl_fn", @altermime_args);
+ my($r,$status) = collect_results($proc_fh,$pid,$altermime,16384,[0]);
+ undef $proc_fh; undef $pid;
+ do_log(2,"program $altermime said: %s",$$r) if ref $r && $$r ne '';
+ $status == 0 or die "Program $altermime failed: $status, $$r";
+ $out_fh = IO::File->new;
+ $out_fh->open($repl_fn,'<') or die "Can't open file $repl_fn: $!";
+ binmode($out_fh,":bytes") or die "Can't cancel :utf8 mode: $!"
+ if $unicode_aware;
+ }
+ my($errn) = stat($repl_fn) ? 0 : 0+$!;
+ if ($errn) { die "Replacement $repl_fn inaccessible: $!" }
+ else { $repl_size = 0 + (-s _) }
+ 1;
+ } or do { $@ = "errno=$!" if $@ eq '' };
+ if ($@ ne '' || $repl_size <= 0) { # handle failure
+ my($msg) = $@ ne '' ? $@ : sprintf("replacement size %d",$repl_size);
+ chomp($msg);
+ do_log(-1,"mangling by %s failed: %s, mail will pass unmodified",
+ $actual_mail_mangle, $msg);
+ if (defined $out_fh) {
+ $out_fh->close or do_log(-1,"Can't close %s: %s", $repl_fn,$!);
+ undef $out_fh;
+ }
+ unlink($repl_fn) or do_log(-1,"Can't remove %s: %s", $repl_fn,$!);
+ if ($actual_mail_mangle eq 'altermime') { # check for leftover files
+ my($repl_tmp_fn) = $repl_fn . '.tmp'; # altermime's temporary file
+ my($errn) = lstat($repl_tmp_fn) ? 0 : 0+$!;
+ if ($errn == ENOENT) {} # fine, does not exist
+ elsif ($errn) {
+ do_log(-1,"Temporary file %s is inaccessible: %s",$repl_tmp_fn,$!);
+ } else { # cleanup after failing altermime
+ unlink($repl_tmp_fn)
+ or do_log(-1,"Can't remove %s: %s",$repl_tmp_fn,$!);
+ }
+ }
+ } else {
+ do_log(1,"mangling by %s (%s) done, new size: %d, orig %d bytes",
+ $actual_mail_mangle, $mail_mangle,
+ $repl_size, $msginfo->msg_size);
+ # don't close or delete the original file, we'll still need it
+ $msginfo->mail_text($out_fh); $msginfo->mail_text_fn($repl_fn);
+ $body_modified = 1;
+ }
+ section_time('mangle-'.$actual_mail_mangle);
+ } else { # 'attach' (default) - poor-man's defanging of dangerous contents
+ do_log(2,"mangling by built-in defanger: %s, <%s>", $mail_mangle,$recip);
+ $actual_mail_mangle = 'attach';
+ my(@explanation); my($spam_summary_inserted) = 0;
+ my(@df_pairs) =
+ $r->setting_by_main_contents_category_all(cr('defang_maps_by_ccat'));
+ for my $pair (@df_pairs) { # collect all defanging reasons that apply
+ my($cc,$mangle_map_ref) = @$pair;
+ my($df) = !defined($mangle_map_ref) ? undef
+ : !ref($mangle_map_ref) ? $mangle_map_ref # compatibility
+ : lookup(0,$recip, @$mangle_map_ref);
+ # the $r->mail_body_mangle happens to be the first noteworthy $df
+ do_log(4,'defang? ccat "%s": %s', $cc,$df);
+ next if !$df;
+ my($ccm) = ccat_maj($cc);
+ if ($ccm==CC_VIRUS) {
+ my($virusname_list) = $msginfo->virusnames;
+ push(@explanation, 'WARNING: contains virus ' .
+ (!defined($virusname_list) ? '' : join(", ",@$virusname_list)));
+ }
+ push(@explanation, 'WARNING: contains banned part') if $ccm==CC_BANNED;
+ if ($ccm==CC_UNCHECKED) {
+ if ($hold ne '') {
+ push(@explanation,
+ "WARNING: NOT CHECKED FOR VIRUSES (mail bomb?):\n $hold");
+ } elsif ($any_undecipherable) {
+ push(@explanation, "WARNING: contains undecipherable part");
+ }
+ }
+ push(@explanation,
+ split(/\n/, wrap_string(
+ 'WARNING: bad headers - '.join(' ', at bad_headers), 78,'',' ')
+ )) if $ccm==CC_BADH;
+ push(@explanation, 'WARNING: oversized') if $ccm==CC_OVERSIZED;
+ if (!$spam_summary_inserted && # can be both CC_SPAMMY and CC_SPAM
+ ($ccm==CC_SPAM || $ccm==CC_SPAMMY)) {
+ push(@explanation, split(/\n/, $msginfo->spam_summary));
+ $spam_summary_inserted = 1;
+ }
+ }
+ my($s) = join(' ', at explanation);
+ do_log(1, "DEFANGING MAIL: %s",
+ length($s) <= 150 ? $s : substr($s,0,150-3)."[...]");
+ for (@explanation)
+ { if (length($_) > 100) { $_ = substr($_,0,100-3) . "..." } }
+ $_ .= "\n" for (@explanation); # append newlines
+ my($d) = defanged_mime_entity($conn,$msginfo,\@explanation);
+ $msginfo->mail_text($d); # substitute mail with a rewritten version
+ $msginfo->mail_text_fn(undef); # remove filename information
+ $body_modified = 1; section_time('defang');
+ }
+ # actually the 'for' loop is bogus, all recipients listed in the argument
+ # are known to be using the same setting for $r->mail_body_mangle,
+ # ensured by add_forwarding_header_edits_per_recip; just exit the loop
+ last;
+ }
+ $body_modified;
}
sub do_quarantine($$$$$;$) {
@@ -9038,8 +10211,11 @@ sub do_quarantine($$$$$;$) {
my($sender) = $msginfo->sender;
my($quar_msg) = Amavis::In::Message->new;
$quar_msg->rx_time($msginfo->rx_time); # copy the reception time
+ $quar_msg->log_id($msginfo->log_id); # use the same log_id
+ $quar_msg->mail_id($msginfo->mail_id); # use the same mail_id
$quar_msg->body_type($msginfo->body_type); # use the same BODY= type
- $quar_msg->mail_id($msginfo->mail_id); # use the same mail_id
+ $quar_msg->header_8bit($msginfo->header_8bit);
+ $quar_msg->body_8bit($msginfo->body_8bit);
$quar_msg->body_digest($msginfo->body_digest); # copy original digest
$quar_msg->delivery_method($quarantine_method);
$quar_msg->dsn_ret($msginfo->dsn_ret);
@@ -9053,12 +10229,14 @@ sub do_quarantine($$$$$;$) {
for my $r (@{$msginfo->per_recip_data}) {
my($recip_obj) = Amavis::In::Message::PerRecip->new;
$recip_obj->recip_addr($r->recip_addr);
+ $recip_obj->recip_addr_smtp($r->recip_addr_smtp);
$recip_obj->dsn_notify($r->dsn_notify);
$recip_obj->dsn_orcpt($r->dsn_orcpt);
$recip_obj->recip_destiny(D_PASS);
push(@recips,$recip_obj);
}
$quar_msg->sender($sender); # original sender & recipients
+ $quar_msg->sender_smtp(qquote_rfc2821_local($sender));
$quar_msg->per_recip_data(\@recips); # original recipients
} else {
# with these quarantine methods the envelope information is used to
@@ -9066,6 +10244,7 @@ sub do_quarantine($$$$$;$) {
# only depend on stashed envelope info into a quarantined mail header
my($mftq) = c('mailfrom_to_quarantine');
$quar_msg->sender(defined $mftq ? $mftq : $sender);
+ $quar_msg->sender_smtp(qquote_rfc2821_local($quar_msg->sender));
$quar_msg->recips($recips_ref); # e.g. per-recip quarantine
}
my($hdr_edits) = Amavis::Out::EditHeader->new;
@@ -9081,6 +10260,11 @@ sub do_quarantine($$$$$;$) {
$hdr_edits->prepend_header('X-Envelope-From',
qquote_rfc2821_local($sender));
}
+# $hdr_edits->append_header_above_received('X-Original-To',
+# $msginfo->orig_header_fields->{'to'});
+# $hdr_edits->delete_header('To');
+# $hdr_edits->append_header_above_received('To',
+# join(",\n ", qquote_rfc2821_local(@{$msginfo->recips})), 1);
$hdr_edits->append_header_above_received('Received',
received_line($conn,$msginfo,$msginfo->mail_id,1), 1);
$quar_msg->auth_submitter(quote_rfc2821_local($quar_msg->sender));
@@ -9088,12 +10272,12 @@ sub do_quarantine($$$$$;$) {
$quar_msg->auth_pass(c('amavis_auth_pass'));
$quar_msg->header_edits($hdr_edits);
$quar_msg->mail_text($msginfo->mail_text); # use the same mail contents
- do_log(5, "DO_QUARANTINE, sender: <%s>", $quar_msg->sender);
+ do_log(5, "DO_QUARANTINE, sender: %s", $quar_msg->sender_smtp);
snmp_count('QuarMsgs');
mail_dispatch($conn, $quar_msg, 1, 0);
my($n_smtp_resp, $n_exit_code, $n_dsn_needed) =
- one_response_for_all($quar_msg, 0, am_id()); # check status
+ one_response_for_all($quar_msg, 0); # check status
if ($n_smtp_resp =~ /^2/ && !$n_dsn_needed) { # ok
snmp_count($snmp_id eq '' ? 'QuarOther' : $snmp_id);
} elsif ($n_smtp_resp =~ /^4/) {
@@ -9111,7 +10295,8 @@ sub do_quarantine($$$$$;$) {
my($mbxname) = $r->recip_mbxname;
if ($mbxname ne '' && !$seen{$mbxname}++) {
push(@qa,$mbxname);
- $quar_type = /^bsmtp:/ ? 'B' : /^smtp:/ ? 'M' : /^sql:/ ? 'Q' :
+ $quar_type = /^smtp:/ ? 'M' : /^lmtp:/ ? 'L' :
+ /^bsmtp:/ ? 'B' : /^sql:/ ? 'Q' :
/^local:/ ? ($mbxname=~/\@/ ? 'M' : $mbxname=~/\.gz\z/ ? 'Z' : 'F')
: '?' for (lc($quarantine_method));
}
@@ -9130,7 +10315,7 @@ sub do_notify_and_quarantine($$$) {
my($q_method, $quarantine_to_maps_ref, $admin_maps_ref,
$mailfrom_admin, $hdrfrom_admin, $mailfrom_recip, $hdrfrom_recip,
$notify_admin_templ_ref, $notify_recips_templ_ref, $warnrecip_maps_ref) =
- map { $msginfo->setting_by_contents_category(cr($_)) }
+ map { scalar($msginfo->setting_by_contents_category(cr($_))) }
qw(quarantine_method_by_ccat
quarantine_to_maps_by_ccat admin_maps_by_ccat
mailfrom_notify_admin_by_ccat hdrfrom_notify_admin_by_ccat
@@ -9139,41 +10324,47 @@ sub do_notify_and_quarantine($$$) {
warnrecip_maps_by_ccat);
my($qar_method) = c('archive_quarantine_method');
my(@ccat_names_pairs) =
- $msginfo->setting_by_contents_category_all(\%ccat_display_names);
- my($ccat_name) = ref $ccat_names_pairs[0] ? $ccat_names_pairs[0][0] : undef;
- my($ccat,$ccat_min) = $msginfo->main_contents_category;
- do_log(3,"do_notify_and_quar: ccat=%s (%d,%d) (%s), q_mth=%s, qar_mth=%s",
- $ccat_name, $ccat, $ccat_min,
- join(', ', map{sprintf('"%s":%s', $_->[1], $_->[0])}
- @ccat_names_pairs),
- $q_method, $qar_method);
+ $msginfo->setting_by_main_contents_category_all(\%ccat_display_names);
+ my($ccat_name) = ref $ccat_names_pairs[0] ? $ccat_names_pairs[0][1] : undef;
+ my($ccat,$ccat_min) = ccat_split($msginfo->contents_category);
+ ll(3) && do_log(3,"do_notify_and_quar: ccat=%s (%d,%d) (%s) ccat_block=(%s)".
+ ", q_mth=%s, qar_mth=%s",
+ $ccat_name, $ccat, $ccat_min,
+ join(', ', map{sprintf('"%s":%s', $_->[0], $_->[1])}
+ @ccat_names_pairs),
+ $msginfo->blocking_ccat, $q_method, $qar_method);
+ my($virusname_list) = $msginfo->virusnames;
my($newvirus_admin_maps_ref) =
- @virusname && !$virus_dejavu ? ca('newvirus_admin_maps') : undef;
- my($blacklisted_any,$whitelisted_any,$do_tag2_any,$do_kill_any) = (0,0,0,0);
+ defined($virusname_list) && @$virusname_list && !$virus_dejavu ?
+ ca('newvirus_admin_maps') : undef;
+ my($blacklisted_any,$whitelisted_any) = (0,0);
+ my($do_tag_any,$do_tag2_any,$do_kill_any) = (0,0,0);
my($tag_level_min,$tag2_level_min,$kill_level_min,$boost_max);
my($spam_level) = $msginfo->spam_level;
my(@q_addr, at qar_addr, at a_addr); # per-recip quarantine address(es) and admins
for my $r (@{$msginfo->per_recip_data}) {
my($rec) = $r->recip_addr;
- my($rec_ccat,$rec_ccat_min) = $r->main_contents_category;
- my($bypassed,$tag_level,$tag2_level,$kill_level,$do_tag2,$do_kill);
+ my($blocking_ccat) = $r->blocking_ccat;
+ my($rec_ccat_maj,$rec_ccat_min) = ccat_split(
+ defined $blocking_ccat ? $blocking_ccat : $r->contents_category);
+ my($tag_level,$tag2_level,$kill_level,$do_tag,$do_tag2,$do_kill);
my($blacklisted) = $r->recip_blacklisted_sender;
my($whitelisted) = $r->recip_whitelisted_sender;
my($boost) = $r->recip_score_boost;
my($spam_level_boosted) = (!defined($spam_level) ? 0 : $spam_level) +
(!defined($boost) ? 0 : $boost);
- do_log(2,"do_notify_and_quarantine: rec_ccat=(%d,%d), ccat=(%d,%s), %s",
- $rec_ccat, $rec_ccat_min, $ccat, $ccat_min, $rec)
- if $rec_ccat != $ccat || $rec_ccat_min != $ccat_min;
- if ($rec_ccat == CC_SPAM || $rec_ccat == CC_SPAMMY) {
+ do_log(2,"do_notify_and_quarantine: rec_bl_ccat=(%d,%d), ccat=(%d,%d) %s",
+ $rec_ccat_maj, $rec_ccat_min, $ccat, $ccat_min, $rec)
+ if $rec_ccat_maj != $ccat || $rec_ccat_min != $ccat_min;
+ $do_tag = $r->is_in_contents_category(CC_CLEAN,1);
+ $do_tag2 = $r->is_in_contents_category(CC_SPAMMY);
+ $do_kill = $r->is_in_contents_category(CC_SPAM);
+ if ($do_tag || $do_tag2 || $do_kill) {
# do the more expensive lookups only when needed
- $bypassed = lookup(0,$rec, @{ca('bypass_spam_checks_maps')});
$tag_level = lookup(0,$rec, @{ca('spam_tag_level_maps')});
$tag2_level = lookup(0,$rec, @{ca('spam_tag2_level_maps')});
$kill_level = lookup(0,$rec, @{ca('spam_kill_level_maps')});
}
- $do_tag2 = $r->is_in_contents_category(CC_SPAMMY);
- $do_kill = $r->is_in_contents_category(CC_SPAM);
# summarize
$blacklisted_any=1 if $blacklisted;
$whitelisted_any=1 if $whitelisted;
@@ -9185,7 +10376,7 @@ sub do_notify_and_quarantine($$$) {
(!defined($kill_level_min) || $kill_level < $kill_level_min);
$boost_max = $boost if defined($boost) &&
(!defined($boost_max) || $boost > $boost_max);
-# $do_tag_any = 1 if $do_tag;
+ $do_tag_any = 1 if $do_tag;
$do_tag2_any = 1 if $do_tag2;
$do_kill_any = 1 if $do_kill;
# get per-recipient quarantine address(es) and admins
@@ -9193,20 +10384,18 @@ sub do_notify_and_quarantine($$$) {
if (defined($q_method) && $q_method ne '' && $quarantine_to_maps_ref) {
my($q); # quarantine (pseudo) address associated with the recipient
($q) = lookup(0,$rec,@$quarantine_to_maps_ref);
- if ($q ne '' && ($rec_ccat == CC_SPAM || $rec_ccat == CC_SPAMMY)) {
+ if ($q ne '' && ($rec_ccat_maj==CC_SPAM || $rec_ccat_maj==CC_SPAMMY)) {
# consider suppressing spam quarantine
- if ($blacklisted && !$whitelisted) {
+ my($cutoff)= lookup(0,$rec,@{ca('spam_quarantine_cutoff_level_maps')});
+ if (!defined $cutoff || $cutoff eq '') {
+ # no cutoff, quarantining all
+ } elsif ($blacklisted && !$whitelisted) {
do_log(2,"do_notify_and_quarantine: cutoff, blacklisted");
$q = ''; # disable quarantine on behalf of this recipient
- } else {
- my($cutoff) =
- lookup(0,$rec,@{ca('spam_quarantine_cutoff_level_maps')});
- if (!defined $cutoff || $cutoff eq '') {}
- elsif ($spam_level_boosted >= $cutoff) {
- do_log(2,"do_notify_and_quarantine: spam level exceeds ".
- "quarantine cutoff level %s", $cutoff);
- $q = ''; # disable quarantine on behalf of this recipient
- }
+ } elsif ($spam_level_boosted >= $cutoff) {
+ do_log(2,"do_notify_and_quarantine: spam level exceeds ".
+ "quarantine cutoff level %s", $cutoff);
+ $q = ''; # disable quarantine on behalf of this recipient
}
}
$q = $rec if $q ne '' && $q_method =~ /^bsmtp:/i; #orig.recip when BSMTP
@@ -9214,7 +10403,7 @@ sub do_notify_and_quarantine($$$) {
}
($a) = lookup(0,$rec,@$admin_maps_ref) if $admin_maps_ref;
push(@a_addr, $a) if defined $a && $a ne '' && !grep {$_ eq $a} @a_addr;
- if ($rec_ccat == CC_VIRUS && $newvirus_admin_maps_ref) {
+ if (ccat_maj($r->contents_category)==CC_VIRUS && $newvirus_admin_maps_ref){
($a) = lookup(0,$rec,@$newvirus_admin_maps_ref);
push(@a_addr, $a) if defined $a && $a ne '' && !grep {$_ eq $a} @a_addr;
}
@@ -9250,12 +10439,17 @@ sub do_notify_and_quarantine($$$) {
$s);
if ($q_method ne '' && @q_addr || $qar_method ne '' && @qar_addr) {
# prepare header edits for the quarantined message and do the quarantining
+ my($allowed_hdrs) = cr('allowed_added_header_fields');
+ my($allowed_hdrs_alert) =
+ $allowed_hdrs && $allowed_hdrs->{lc('X-Amavis-Alert')};
my($hdr_edits) = Amavis::Out::EditHeader->new;
- if ($msginfo->is_in_contents_category(CC_VIRUS)) {
+ if ($msginfo->is_in_contents_category(CC_VIRUS) && $allowed_hdrs_alert) {
+ my($virusname_list) = $msginfo->virusnames;
$hdr_edits->add_header('X-Amavis-Alert',
- "INFECTED, message contains virus: " . join(", ", @virusname));
- }
- if ($msginfo->is_in_contents_category(CC_BANNED)) {
+ "INFECTED, message contains virus: " .
+ (!defined($virusname_list) ? '' : join(", ",@$virusname_list)) );
+ }
+ if ($msginfo->is_in_contents_category(CC_BANNED) && $allowed_hdrs_alert) {
for my $r (@{$msginfo->per_recip_data}) {
my(@b); @b = @{$r->banned_parts} if defined $r->banned_parts;
if (@b) {
@@ -9268,33 +10462,54 @@ sub do_notify_and_quarantine($$$) {
}
}
}
- if ($msginfo->is_in_contents_category(CC_BADH)) {
+ if ($msginfo->is_in_contents_category(CC_BADH) && $allowed_hdrs_alert) {
$hdr_edits->add_header('X-Amavis-Alert', 'BAD HEADER '.$bad_headers[0]);
}
- if ($msginfo->is_in_contents_category(CC_SPAM) ||
- $msginfo->is_in_contents_category(CC_SPAMMY)) {
- $hdr_edits->add_header('X-Spam-Flag',
- $do_tag2_any||$do_kill_any ? 'YES' : 'NO');
- $hdr_edits->add_header('X-Spam-Score',
- 0+sprintf("%.3f",0+$spam_level+$boost_max) );
+ if ($do_tag_any || $do_tag2_any || $do_kill_any) {
+ $hdr_edits->add_header('X-Spam-Flag', $do_tag2_any ? 'YES' : 'NO')
+ if $allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Flag')};
+ if ($allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Score')}) {
+ my($score) = 0+$spam_level+$boost_max;
+ $score = max(64,$score) if $blacklisted_any; # don't go below 64 if bl
+ $score = min( 0,$score) if $whitelisted_any; # don't go above 0 if wl
+ $hdr_edits->add_header('X-Spam-Score', 0+sprintf("%.3f",$score));
+ }
my($slc) = c('sa_spam_level_char');
$hdr_edits->add_header('X-Spam-Level', $spam_level_bar)
- if defined $spam_level_bar;
- $hdr_edits->add_header('X-Spam-Status', $full_spam_status, 1);
- $hdr_edits->add_header('X-Spam-Report', "\n".$msginfo->spam_report, 2)
- if c('sa_spam_report_header') && $msginfo->spam_report ne '';
- }
- do_quarantine($conn,$msginfo,$hdr_edits,\@q_addr,$q_method,
- 'Quar'.$ccat_name.'Msgs') if $q_method ne '' && @q_addr;
- do_quarantine($conn,$msginfo,$hdr_edits,\@qar_addr,$qar_method,
- 'QuarArchMsgs') if $qar_method ne '' && @qar_addr;
- }
- if (ll(2) && $ccat == CC_SPAM) {
+ if defined $spam_level_bar &&
+ $allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Level')};
+ $hdr_edits->add_header('X-Spam-Status', $full_spam_status, 1)
+ if $allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Status')};
+ $hdr_edits->add_header('X-Spam-Report',
+ "\n".sanitize_str($msginfo->spam_report,1), 2)
+ if c('sa_spam_report_header') && $msginfo->spam_report ne '' &&
+ $allowed_hdrs && $allowed_hdrs->{lc('X-Spam-Report')};
+ }
+ my($os_fp) = $msginfo->client_os_fingerprint;
+ $hdr_edits->add_header('X-Amavis-OS-Fingerprint', sanitize_str($os_fp))
+ if $os_fp ne '' &&
+ $allowed_hdrs && $allowed_hdrs->{lc('X-Amavis-OS-Fingerprint')};
+ my($method_used);
+ if ($qar_method ne '' && @qar_addr) { # archival quarantine requested
+ do_quarantine($conn,$msginfo,$hdr_edits,\@qar_addr,$qar_method,
+ 'QuarArchMsgs');
+ $method_used = $qar_method;
+ }
+ if ($q_method ne '' && @q_addr) { # normal quarantine requested
+ # simpleminded collision check, does not take into account quarantine_to
+ if (defined $method_used && $method_used eq $q_method) {
+ do_log(2,"Suppress quarantine, already archived as %s", $method_used);
+ } else {
+ do_quarantine($conn,$msginfo,$hdr_edits,\@q_addr,$q_method,
+ 'Quar'.$ccat_name.'Msgs');
+ }
+ }
+ }
+ if (ll(2) && $msginfo->is_in_contents_category(CC_SPAM)) {
# log entry compatible with older log parsers
- my($autolearn_status) = $msginfo->autolearn_status;
+ my($autolearn_status) = $msginfo->supplementary_info('AUTOLEARN');
$s = $full_spam_status; $s =~ s/\n[ \t]/ /g;
- do_log(2,"SPAM, %s -> %s, %s%s%s",
- qquote_rfc2821_local($msginfo->sender_source),
+ do_log(2,"SPAM, %s -> %s, %s%s%s", $msginfo->sender_smtp,
join(',', qquote_rfc2821_local(@{$msginfo->recips})), $s,
$autolearn_status eq '' ? '' : ", autolearn=$autolearn_status",
!@q_addr ? '' : sprintf(", quarantine %s (%s)",
@@ -9308,11 +10523,14 @@ sub do_notify_and_quarantine($$$) {
do_log(5,"skip admin notifications - empty template");
} else { # notify per-recipient administrators
ll(5) && do_log(5, "Admin notifications to %s; sender: %s",
- join(',',qquote_rfc2821_local(@a_addr)), $msginfo->sender);
+ join(',',qquote_rfc2821_local(@a_addr)),
+ $msginfo->sender_smtp);
my($notification) = Amavis::In::Message->new;
$notification->rx_time($msginfo->rx_time); # copy the reception time
+ $notification->log_id($msginfo->log_id); # copy log id
$notification->delivery_method(c('notify_method'));
$notification->sender($mailfrom_admin);
+ $notification->sender_smtp(qquote_rfc2821_local($mailfrom_admin));
$notification->auth_submitter(quote_rfc2821_local($mailfrom_admin));
$notification->auth_user(c('amavis_auth_user'));
$notification->auth_pass(c('amavis_auth_pass'));
@@ -9330,7 +10548,7 @@ sub do_notify_and_quarantine($$$) {
$notification->header_edits($hdr_edits);
mail_dispatch($conn, $notification, 1, 0);
my($n_smtp_resp, $n_exit_code, $n_dsn_needed) =
- one_response_for_all($notification, 0, am_id()); # check status
+ one_response_for_all($notification, 0); # check status
if ($n_smtp_resp =~ /^2/ && !$n_dsn_needed) { # ok
} elsif ($n_smtp_resp =~ /^4/) {
die "temporarily unable to notify admin: $n_smtp_resp";
@@ -9339,7 +10557,7 @@ sub do_notify_and_quarantine($$$) {
}
# $notification->purge;
}
- # recipient notifications (don't bother for spam and clean)
+ # recipient notifications
for my $r (@{$msginfo->per_recip_data}) {
my($rec) = $r->recip_addr;
my($wr) = lookup(0,$rec,@$warnrecip_maps_ref) if $warnrecip_maps_ref;
@@ -9350,8 +10568,7 @@ sub do_notify_and_quarantine($$$) {
!@$notify_recips_templ_ref : $$notify_recips_templ_ref eq '')){
do_log(5,"skip recipient notifications - empty template");
$wr = 0; # do not send empty notifications
- } elsif (!c('warn_offsite') &&
- !lookup(0,$rec,@{ca('local_domains_maps')})) {
+ } elsif (!c('warn_offsite') && !$r->recip_is_local) {
$wr = 0; # do not notify foreign recipients
# } elsif (! defined($msginfo->sender_contact) ) { # (not general enough)
# do_log(5,"skip recipient notifications for unknown sender");
@@ -9360,8 +10577,10 @@ sub do_notify_and_quarantine($$$) {
if ($wr) { # warn recipient
my($notification) = Amavis::In::Message->new;
$notification->rx_time($msginfo->rx_time); # copy the reception time
+ $notification->log_id($msginfo->log_id); # copy log id
$notification->delivery_method(c('notify_method'));
$notification->sender($mailfrom_recip);
+ $notification->sender_smtp(qquote_rfc2821_local($mailfrom_recip));
$notification->auth_submitter(quote_rfc2821_local($mailfrom_recip));
$notification->auth_user(c('amavis_auth_user'));
$notification->auth_pass(c('amavis_auth_pass'));
@@ -9383,7 +10602,7 @@ sub do_notify_and_quarantine($$$) {
$notification->header_edits($hdr_edits);
mail_dispatch($conn, $notification, 1, 0);
my($n_smtp_resp, $n_exit_code, $n_dsn_needed) =
- one_response_for_all($notification, 0, am_id()); # check status
+ one_response_for_all($notification, 0); # check status
if ($n_smtp_resp =~ /^2/ && !$n_dsn_needed) { # ok
} elsif ($n_smtp_resp =~ /^4/) {
die "temporarily unable to notify recipient rec: $n_smtp_resp";
@@ -9396,79 +10615,93 @@ sub do_notify_and_quarantine($$$) {
do_log(5, "do_notify_and_quarantine - done");
}
-# Calculate message body digest;
+# Calculate a message body digest;
# While at it, also get message size, check for 8-bit data, collect
# entropy, and store original header since we need it for the %H macro,
# and MIME::Tools may modify it.
#
-sub get_body_digest($$) {
- my($fh, $msginfo) = @_;
+sub get_body_digest($$$) {
+ my($fh, $msginfo, $alg) = @_;
$fh->seek(0,0) or die "Can't rewind mail file: $!";
-
- # choose message digest method:
- my($hctx) = Digest::MD5->new; # 128 bits (32 hex digits)
- my($bctx) = Digest::MD5->new; # 128 bits (32 hex digits)
-# my($bctx) = Digest::SHA1->new; # 160 bits (40 hex digits), slightly slower
+ my($hctx,$bctx);
+ # choose a message digest: MD5: 128 bits (32 hex), SHA family: 160..512 bits
+ if (uc($alg) eq 'MD5') { $hctx = Digest::MD5->new; $bctx = Digest::MD5->new }
+ else { $hctx = Digest::SHA->new($alg); $bctx = Digest::SHA->new($alg) }
+# section_time('digest_init');
my(%collectable_header_fields);
- $collectable_header_fields{lc($_)} = 1 for (qw(From To Cc Sender Subject
- Received Message-ID Resent-Message-ID Precedence User-Agent X-Mailer
- DKIM-Signature DomainKey-Signature Authentication-Results));
- my($header_size)=0; my($body_size)=0; my($h_8bit,$b_8bit) = (0,0);
- my(@orig_header,%orig_header_fields); my($ln,$collecting); local($1,$2);
+ $collectable_header_fields{lc($_)} = 1
+ for ( qw(From To Cc Sender Subject Received Message-ID Resent-Message-ID
+ In-Reply-To References Precedence List-Id User-Agent X-Mailer
+ DKIM-Signature DomainKey-Signature Authentication-Results) );
+ my($header_size,$body_size) = (0,0); my($h_8bit,$b_8bit) = (0,0);
+ my($orig_header) = []; my($orig_header_fields) = {};
+ my($ln,$collecting); local($1,$2);
for ($! = 0; defined($ln=<$fh>); $! = 0) { # read mail header
last if $ln eq $eol;
$header_size += length($ln);
- $h_8bit = 1 if !$h_8bit && $ln =~ /^[\000-\177]*\z/;
- $hctx->add($ln); push(@orig_header,$ln); # with trailing EOL
+ $h_8bit = 1 if !$h_8bit && $ln !~ /^[\000-\177]*\z/;
+ $hctx->add($ln); push(@$orig_header,$ln); # with trailing EOL
if ($ln =~ /^[ \t]/) { # header field continuation
- $orig_header_fields{$collecting} .= $ln if defined $collecting;
+ $orig_header_fields->{$collecting} .= $ln if defined $collecting;
} elsif ($ln =~ /^([^ \t]+)[ \t]*:(.*)\z/xsi &&
$collectable_header_fields{lc($1)}) {
$collecting = lc($1);
- if (exists $orig_header_fields{$collecting}) { undef $collecting }
- else { $orig_header_fields{$collecting} = $2 } # keep first occurrence
+ if (exists $orig_header_fields->{$collecting}) { undef $collecting }
+ else { $orig_header_fields->{$collecting} = $2 } # keep first occurrence
} else { undef $collecting }
}
defined $ln || $!==0 or die "Error reading mail header: $!";
- $header_size = untaint($header_size); # silly Perl, length gives tainted
- add_entropy($hctx->digest); # faster than traversing @orig_header again
+ $header_size = untaint($header_size); # silly Perl, length() gives tainted
+ add_entropy($hctx->digest); # faster than traversing @$orig_header again
+ section_time('digest_hdr');
+
my($len); local($_);
- while (($len = read($fh,$_,16384)) > 0) {
+ while (($len = read($fh,$_,65536)) > 0) {
$bctx->add($_); $body_size += $len;
- $b_8bit = 1 if !$b_8bit && /^[\000-\177]*\z/; #faster than !/[^\000-\177]/
+ $b_8bit = 1 if !$b_8bit && !/^[\000-\177]*\z/; #faster than /[^\000-\177]/
}
defined $len or die "Error reading mail body: $!";
my($signature) = $bctx->hexdigest;
# my($signature) = $bctx->b64digest;
add_entropy($signature);
- $signature = untaint($signature) # checked (either 32 or 40 char)
- if $signature =~ /^ [0-9a-fA-F]{32} (?: [0-9a-fA-F]{8} )? \z/x;
+ $signature = untaint($signature) # checked (hex digits, 128..512 bits)
+ if $signature =~ /^ [0-9a-fA-F]{32,128} \z/x;
+# section_time('digest_body');
+
# store information obtained
- $msginfo->orig_header_fields(\%orig_header_fields);
- $msginfo->orig_header(\@orig_header);
+ $msginfo->orig_header_fields($orig_header_fields);
+ $msginfo->orig_header($orig_header);
$msginfo->orig_header_size($header_size);
$msginfo->orig_body_size($body_size);
$msginfo->body_digest($signature);
-
+ $msginfo->header_8bit($h_8bit ? 1 : 0);
+ $msginfo->body_8bit($b_8bit ? 1 : 0);
# check for 8-bit characters and adjust body type if necessary (rfc1652)
my($bt_orig) = $msginfo->body_type;
- my($bt_true) = $h_8bit || $b_8bit ? '8BITMIME' : '7BIT';
- if (!defined($bt_orig) || $bt_orig eq '') {
- do_log(4,"setting body type: %s (h=%s, b=%s)", $bt_true, $h_8bit, $b_8bit);
- $msginfo->body_type($bt_true);
- } elsif ($bt_true eq '8BITMIME' && uc($bt_orig) ne '8BITMIME') {
- do_log(4,"changing body type: %s => %s (h=%s, b=%s)",
- $bt_orig, $bt_true, $h_8bit, $b_8bit);
- $msginfo->body_type($bt_true);
+ $bt_orig = !defined($bt_orig) ? '' : uc($bt_orig);
+ if ($bt_orig eq '') { # unlabeled on reception
+ # keeping 8-bit mail unlabeled might avoid breaking DKIM in transport
+ # (labeled as 8-bit may invoke 8>7 downgrades in MTA, breaking signatures)
+ $msginfo->body_type('7BIT') if !$h_8bit && !$b_8bit; # safe to label
+ }
+ if (ll(4)) {
+ my($msg_fmt) =
+ ($bt_orig eq '' && $b_8bit) ? "%s, but 8-bit body"
+ : ($bt_orig eq '' && $h_8bit) ? "%s, but 8-bit header"
+ : ($bt_orig eq '7BIT' && ($h_8bit || $b_8bit)) ? "%s inappropriately"
+ : ($bt_orig eq '8BITMIME' && !($h_8bit || $b_8bit)) ? "%s unnecessarily"
+ : "%s, good";
+ do_log(4, "body type: $msg_fmt (h=%s, b=%s)",
+ $bt_orig eq '' ? 'unlabeled' : "labeled $bt_orig", $h_8bit,$b_8bit);
}
do_log(3, "body hash: %s", $signature);
- section_time('body_digest');
+ section_time('digest_body');
$signature;
}
-sub find_program_path($$$) {
- my($fv_list, $path_list_ref, $may_log) = @_;
+sub find_program_path($$) {
+ my($fv_list, $path_list_ref) = @_;
$fv_list = [$fv_list] if !ref $fv_list;
my($found);
for my $fv (@$fv_list) {
@@ -9478,8 +10711,7 @@ sub find_program_path($$$) {
my($errn) = stat($fv_cmd[0]) ? 0 : 0+$!;
if ($errn == ENOENT) { }
elsif ($errn) {
- do_log(-1, "find_program_path: %s inaccessible: %s", $fv_cmd[0], $!)
- if $may_log;
+ do_log(-1, "find_program_path: %s inaccessible: %s", $fv_cmd[0], $!);
} elsif (-x _ && !-d _) { $found = join(' ', @fv_cmd) }
} elsif ($fv_cmd[0] =~ /\//) { # relative path
die "find_program_path: relative paths not implemented: @fv_cmd\n";
@@ -9489,8 +10721,7 @@ sub find_program_path($$$) {
if ($errn == ENOENT) { }
elsif ($errn) {
do_log(-1, "find_program_path: %s/%s inaccessible: %s",
- $p, $fv_cmd[0], $!)
- if $may_log;
+ $p, $fv_cmd[0], $!);
} elsif (-x _ && !-d _) {
$found = $p . '/' . join(' ', @fv_cmd);
last;
@@ -9504,9 +10735,9 @@ sub find_program_path($$$) {
sub find_external_programs($) {
my($path_list_ref) = @_;
- for my $f (qw($file $dspam)) {
+ for my $f (qw($file $dspam $altermime)) {
my($g) = $f; $g =~ s/\$/Amavis::Conf::/; my($fv_list) = eval('$' . $g);
- my($found) = find_program_path($fv_list, $path_list_ref, 1);
+ my($found) = find_program_path($fv_list, $path_list_ref);
{ no strict 'refs'; $$g = $found } # NOTE: a symbolic reference
if (!defined $found) { do_log(0,"No %-19s not using it", "$f,") }
else {
@@ -9523,7 +10754,7 @@ sub find_external_programs($) {
for my $d (@$f[2..$#$f]) { # all but the first two elements are programs
# allow one level of indirection
my($dd) = (ref $d eq 'SCALAR' || ref $d eq 'REF') ? $$d : $d;
- my($found) = find_program_path($dd, $path_list_ref, 1);
+ my($found) = find_program_path($dd, $path_list_ref);
if (defined $found) { $any = 1; $d = $dd = $found; push(@found,$dd)}
else {
push(@tried, !ref($dd) ? $dd : join(", ",@$dd)) if $dd ne '';
@@ -9554,9 +10785,9 @@ sub find_external_programs($) {
$tier = 'secondary';
} elsif (!defined $f || !ref $f) { # empty, skip
} elsif (ref($f->[1]) eq 'CODE') {
- do_log(0, "Using internal av scanner code for (%s) %s", $tier,$f->[0]);
+ do_log(0, "Using %s internal av scanner code for %s", $tier,$f->[0]);
} else {
- my($found) = $f->[1] = find_program_path($f->[1], $path_list_ref, 1);
+ my($found) = $f->[1] = find_program_path($f->[1], $path_list_ref);
if (!defined $found) {
do_log(3, "No %s av scanner: %s", $tier, $f->[0]);
$f = undef; # release its storage
@@ -9585,9 +10816,11 @@ sub fetch_modules_extra() {
!grep {exists $policy_bank{$_}{'bypass_decode_parts'} &&
!$policy_bank{$_}{'bypass_decode_parts'} } keys %policy_bank) {
} else {
- push(@modules, qw(Convert::TNEF Convert::UUlib Archive::Zip Archive::Tar));
- }
- push(@modules, 'Authen::SASL') if c('auth_required_out');
+ push(@modules, qw(Convert::TNEF Convert::UUlib Archive::Zip));
+ # push(@modules, qw(Archive::Tar)); # terrible, don't use it!
+ }
+ push(@modules, 'Authen::SASL') if c('auth_required_out');
+ push(@modules, 'Anomy::Sanitizer') if $enable_anomy_sanitizer;
Amavis::Boot::fetch_modules('REQUIRED ADDITIONAL MODULES', 1, @modules);
@modules = (); # now start collecting optional modules
if ($unicode_aware) {
@@ -9596,10 +10829,13 @@ sub fetch_modules_extra() {
Encode Encode::Byte Encode::MIME::Header Encode::Unicode::UTF7
Encode::CN Encode::TW Encode::KR Encode::JP
unicore::Canonical.pl unicore::Exact.pl unicore::PVA.pl
- unicore::To::Fold.pl unicore::To::Title.pl
+ unicore::To::Fold.pl unicore::To::Title.pl
unicore::To::Lower.pl unicore::To::Upper.pl
));
}
+ push(@modules, qw(IO::Socket::INET6));
+ push(@modules, defined($min_servers) ? 'Net::Server::PreFork'
+ : 'Net::Server::PreForkSimple');
push(@modules, @additional_perl_modules);
my($missing);
$missing = Amavis::Boot::fetch_modules('PRE-COMPILE OPTIONAL MODULES', 0,
@@ -9640,8 +10876,8 @@ Usage:
Usage:
$0
[-u user] [-g group]
- [-d log_level] [-m max_servers]
- {-c config_file} {-p listen_port_or_socket}
+ [-i instance_name] {-c config_file}
+ [-d log_level] [-m max_servers] {-p listen_port_or_socket}
[-L lock_file] [-P pid_file] [-H home_dir]
[-D db_home_dir | -D ''] [-Q quarantine_dir | -Q '']
[-R chroot_dir | -R ''] [-S helpers_home_dir] [-T tempbase_dir]
@@ -9692,22 +10928,16 @@ close(\*Amavis::DATA) or die "Error clos
for ($Amavis::Conf::log_templ, $Amavis::Conf::log_recip_templ);
};
+undef $ENV{PATH};
umask(0027); # set our preferred umask
POSIX::setlocale(LC_TIME,"C"); # English dates required in syslog and rfc2822
-# Consider droping privileges early, before reading config file.
+# Consider dropping privileges early, before reading config file.
# This is only possible if running under chroot will not be needed.
#
my($desired_group); # defaults to $desired_user's group
my($desired_user); # username or UID
if ($> != 0) { $desired_user = $> } # use effective UID if not root
-#else {
-# for my $u ('amavis', 'vscan') { # try to guess a good default username
-# my($username,$passwd,$uid,$gid) = getpwnam($u);
-# if (defined $uid && $uid != 0) { $desired_user = $u; last }
-# }
-#}
-
# collect and parse command line options
my($log_level_override, $max_servers_override);
@@ -9715,7 +10945,7 @@ my($quarantinedir_override, $db_home_ove
my($quarantinedir_override, $db_home_override, $daemon_chroot_dir_override);
my($lock_file_override, $pid_file_override);
my(@listen_sockets_override, $listen_sockets_overridden);
-while (@ARGV >= 2 && $ARGV[0] =~ /^-[ugdmcpDHLPQRST]\z/ ||
+while (@ARGV >= 2 && $ARGV[0] =~ /^-[ugdimcpDHLPQRST]\z/ ||
@ARGV >= 1 && $ARGV[0] =~ /^-/) {
my($opt,$val);
$opt = shift @ARGV;
@@ -9730,11 +10960,12 @@ while (@ARGV >= 2 && $ARGV[0] =~ /^-[ugd
if ($> == 0) { $desired_user = $val }
else { print STDERR "Ignoring option -u when not running as root\n" }
} elsif ($opt eq '-g') { # -g group
- if ($val != $desired_group) {
- $desired_group = $val;
- print STDERR "NOTICE: Option -g may not achieve desired result ".
- "when running as non-root\n" if $> != 0;
- }
+ print STDERR "NOTICE: Option -g may not achieve desired result when ".
+ "running as non-root\n" if $> != 0 && $val ne $desired_group;
+ $desired_group = $val;
+ } elsif ($opt eq '-i') { # -i instance_name, may be of use to a .conf file
+ $val =~ /^[a-z0-9._+-]*\z/i or die "Special chars in option -i $val\n";
+ $instance_name = untaint($val); # not used by amavisd directly
} elsif ($opt eq '-d') { # -d log_level or -d SAdbg1,SAdbg2,..,SAdbg3
$log_level_override = untaint($val);
} elsif ($opt eq '-m') { # -m max_servers
@@ -9756,7 +10987,7 @@ while (@ARGV >= 2 && $ARGV[0] =~ /^-[ugd
} elsif ($opt eq '-Q') { # -Q quarantine_dir, empty string disables quarant.
$quarantinedir_override = untaint($val);
} elsif ($opt eq '-R') { # -R chroot_dir, empty string or '/' avoids chroot
- $daemon_chroot_dir_override = untaint($val);
+ $daemon_chroot_dir_override = $val eq '/' ? '' : untaint($val);
} elsif ($opt eq '-S') { # -S helpers_home_dir for SA
$helpers_home_override = untaint($val) if $val ne '';
} elsif ($opt eq '-T') { # -T tempbase_dir
@@ -9765,11 +10996,21 @@ while (@ARGV >= 2 && $ARGV[0] =~ /^-[ugd
die "Error in parsing command line options: $opt\n\n" . usage();
}
}
- at ARGV <= 1 or die sprintf("Only one command line parameter allowed: %s\n",
- join(" ", at ARGV));
my($cmd) = lc($ARGV[0]);
-
-if (defined $desired_user && ($> == 0 || $< == 0)) { # drop privileges early
+if (@ARGV > 1) {
+ die "$myversion: Only one command line parameter allowed: %s\n" .
+ join(" ", at ARGV) . usage();
+} elsif ($cmd !~ /^(?:start|debug|debug-sa|foreground|reload|stop)?\z/) {
+ die "$myversion: Unknown command line parameter: $cmd\n\n" . usage();
+}
+
+if (!defined($desired_user)) {} # early dropping of privs not requested
+elsif ($> != 0 && $< != 0) {} # early dropping of privs not needed
+elsif (defined $daemon_chroot_dir_override &&
+ $daemon_chroot_dir_override ne '') {
+ # early dropping of privs would prevent later chroot and is to be skipped
+} else {
+ # drop privileges early if an uid was specified on a command line, option -u
local($1);
my($username,$passwd,$uid,$gid) =
$desired_user=~/^(\d+)$/ ? (undef,undef,$1,undef) :getpwnam($desired_user);
@@ -9777,17 +11018,17 @@ if (defined $desired_user && ($> == 0 ||
if ($desired_group eq '') { $desired_group = $gid } # for logging purposes
else { $gid = $desired_group=~/^(\d+)$/ ? $1 : getgrnam($desired_group) }
defined $gid or die "No such group: $desired_group\n";
- $( = $gid; # real GID
- $) = "$gid $gid"; # effective GID
+ $( = $gid; $) = "$gid $gid"; # real and effective GID
+ POSIX::setgid($gid) or die "Can't setgid to $gid: $!";
POSIX::setuid($uid) or die "Can't setuid to $uid: $!";
$> = $uid; $< = $uid; # just in case
# print STDERR "desired user=$desired_user ($uid), current: EUID: $> ($<)\n";
-# print STDERR "desired group=$desired_group, current: EGID: $) ($()\n";
+# print STDERR "desired group=$desired_group ($gid), current: EGID: $) ($()\n";
$> != 0 or die "Still running as root, aborting\n";
$< != 0 or die "Effective UID changed, but Real UID is 0\n";
}
-# these settings must be overridden before and after read_config(),
+# these settings must be overridden before and after read_config
# because some other settings in a config file may be derived from them
$Amavis::Conf::MYHOME = $myhome_override if defined $myhome_override;
$Amavis::Conf::TEMPBASE = $tempbase_override if defined $tempbase_override;
@@ -9801,6 +11042,7 @@ if (defined $desired_user && ($> == 0 ||
# but before reading configuration file
init_local_delivery_aliases();
init_builtin_macros();
+$instance_name = '' if !defined($instance_name);
# convert arrayref to Amavis::Lookup::RE object, the Amavis::Lookup::RE module
# was not yet available during BEGIN phase
@@ -9809,18 +11051,19 @@ init_builtin_macros();
# default location of the config file if none specified
push(@config_files, '/etc/amavisd.conf') if !@config_files;
-# Read/execute the config file, which may override default settings
-Amavis::Conf::read_config(@config_files);
+# Read and evaluate config files, which may override default settings
+Amavis::Conf::include_config_files(@config_files);
+Amavis::Conf::supply_after_defaults();
if (defined $desired_user && $daemon_user ne '') {
local($1);
# compare the config file settings to current UID
my($username,$passwd,$uid,$gid) =
$daemon_user=~/^(\d+)$/ ? (undef,undef,$1,undef) : getpwnam($daemon_user);
- $uid == $> or warn sprintf(
- "WARN: running under user '%s' (UID=%s), the config file".
- " specifies \$daemon_user='%s' (UID=%s)\n",
- $desired_user, $>, $daemon_user, defined $uid ? $uid : '?');
+ ($desired_user eq $daemon_user || $desired_user eq $uid)
+ or warn sprintf("WARN: running under user '%s' (UID=%s), ".
+ "the config file specifies \$daemon_user='%s' (UID=%s)\n",
+ $desired_user, $>, $daemon_user, defined $uid ? $uid : '?');
}
# override certain config file options by command line arguments
@@ -9912,7 +11155,8 @@ undef $extra_code_sql_quar if !defined(
{ my(%needed_protocols_out); local($1);
for my $bank_name (keys %policy_bank) {
for my $method_name qw(
- forward_method notify_method resend_method release_method
+ forward_method notify_method resend_method
+ release_method requeue_method
os_fingerprint_method virus_quarantine_method
banned_files_quarantine_method spam_quarantine_method
bad_header_quarantine_method clean_quarantine_method
@@ -9922,7 +11166,8 @@ undef $extra_code_sql_quar if !defined(
$needed_protocols_out{uc($1)} = 1 if $var =~ /^([A-Za-z0-9]*):/;
}
}
- if (!$needed_protocols_out{'SMTP'}) { undef $extra_code_out_smtp }
+ if (!$needed_protocols_out{'SMTP'} &&
+ !$needed_protocols_out{'LMTP'}) { undef $extra_code_out_smtp }
else {
eval $extra_code_out_smtp or die "Problem in Amavis::Out::SMTP code: $@";
$extra_code_out_smtp = 1; # release memory occupied by the source code
@@ -9953,6 +11198,7 @@ undef $extra_code_sql_quar if !defined(
}
}
+#if (0) {} #****
if (!defined($extra_code_sql_log) && !defined($extra_code_sql_quar) &&
!defined($extra_code_sql_lookup)) { undef $extra_code_sql_base }
else {
@@ -9973,7 +11219,7 @@ if (defined $extra_code_sql_lookup) {
}
if (!$enable_ldap) { undef $extra_code_ldap }
else {
- eval $extra_code_ldap or die "Problem in the Lookup::LDAP code: $@";
+ eval $extra_code_ldap or die "Problem in Lookup::LDAP code: $@";
$extra_code_ldap = 1; # release memory occupied by the source code
}
@@ -9984,7 +11230,7 @@ if (!@{ca('av_scanners')} && !@{ca('av_s
# do a simple-minded test to make it easy to turn off virus checks
undef $extra_code_antivirus;
} else {
- eval $extra_code_antivirus or die "Problem in the antivirus code: $@";
+ eval $extra_code_antivirus or die "Problem in antivirus code: $@";
$extra_code_antivirus = 1; # release memory occupied by the source code
}
if (!$extra_code_antivirus) # release storage
@@ -9996,9 +11242,9 @@ if (@$bpscm && !ref($bpscm->[0]) && $bps
undef $extra_code_antispam;
undef $extra_code_antispam_sa;
} else {
- eval $extra_code_antispam or die "Problem in the antispam code: $@";
+ eval $extra_code_antispam or die "Problem in antispam code: $@";
$extra_code_antispam = 1; # release memory occupied by the source code
- eval $extra_code_antispam_sa or die "Problem in the antispam SA code: $@";
+ eval $extra_code_antispam_sa or die "Problem in antispam SA code: $@";
$extra_code_antispam_sa = 1; # release memory occupied by the source code
}
@@ -10007,74 +11253,163 @@ if (c('bypass_decode_parts') &&
!$policy_bank{$_}{'bypass_decode_parts'} } keys %policy_bank) {
undef $extra_code_unpackers;
} else {
- eval $extra_code_unpackers or die "Problem in the Amavis::Unpackers code: $@";
+ eval $extra_code_unpackers or die "Problem in Amavis::Unpackers code: $@";
$extra_code_unpackers = 1; # release memory occupied by the source code
}
-# act on command line parameter in $cmd
-my($killed_amavisd_pid); my($kill_sig_used);
-if ($cmd =~ /^(start|debug|debug-sa|foreground)?\z/) {
- $DEBUG=1 if $cmd eq 'debug';
- $daemonize=0 if $cmd eq 'foreground';
- $daemonize=0, $sa_debug=1 if $cmd eq 'debug-sa';
-} elsif ($cmd !~ /^(reload|stop)\z/) {
- die "$myversion: Unknown argument: $cmd\n\n" . usage();
-} else { # stop or reload
- eval { # first stop a running daemon
- my($pidf) = defined $pid_file_override ? $pid_file_override : $pid_file;
- $pidf ne '' or die "Config parameter \$pid_file not defined";
- my($errn) = stat($pidf) ? 0 : 0+$!;
- $errn != ENOENT or die "No PID file $pidf\n";
- $errn == 0 or die "PID file $pidf inaccessible: $!";
- my($ln); my($amavisd_pid); my($pidf_h) = IO::File->new;
- $pidf_h->open($pidf,'<') or die "Can't open file $pidf: $!";
- for ($! = 0; defined($ln=$pidf_h->getline); $! = 0) {
- chomp($ln);
- $amavisd_pid = $ln if $ln =~ /^\d+\z/ && !defined($amavisd_pid);
- }
- defined $ln || $!==0 or die "Error reading from $pidf: $!";
+Amavis::Log::init($DO_SYSLOG, $LOGFILE); # initialize logging
+Amavis::Log::log_to_stderr($cmd eq 'debug' ? 1 : 0);
+do_log(2, "logging initialized, log level %s, %s", c('log_level'),
+ $DO_SYSLOG ? sprintf("syslog: %s.%s",c('syslog_ident'),c('syslog_facility')):
+ $LOGFILE ne '' ? "logfile: $LOGFILE" : "STDERR");
+
+eval {
+ # is amavisd daemon already running?
+ my($amavisd_pid); # obtain PID of the currently running amavisd daemon
+ my($pidf) = defined $pid_file_override ? $pid_file_override : $pid_file;
+ $pidf ne '' or die "Config parameter \$pid_file not defined";
+ my(@stat_list) = lstat($pidf); my($errn) = @stat_list ? 0 : 0+$!;
+ if ($errn == ENOENT) {
+ die "The amavisd daemon is apparently not running, no PID file $pidf\n"
+ if $cmd =~ /^(?:reload|stop)\z/;
+ } elsif ($errn != 0) {
+ die "PID file $pidf is inaccessible: $!\n";
+ } elsif (!-f _) {
+ die "PID file $pidf is not a regular file\n";
+ } else { # determine PID of the currently running amavisd daemon, validate it
+ my($mtime) = $stat_list[9]; my($ln); my($pidf_h) = IO::File->new;
+ $pidf_h->open($pidf,'<') or die "Can't open PID file $pidf: $!";
+ for ($! = 0; defined($ln=$pidf_h->getline); $! = 0)
+ { chomp($ln); $amavisd_pid = $ln if $ln ne '' && !defined $amavisd_pid }
+ defined $ln || $!==0 or die "Error reading from file $pidf: $!";
$pidf_h->close or die "Error closing file $pidf: $!";
- defined($amavisd_pid) or die "Invalid PID in the $pidf";
+ defined($amavisd_pid) or die "Missing process ID in file $pidf";
+ $amavisd_pid =~ /^\d{1,10}\z/ && $amavisd_pid > 1 && $amavisd_pid != $$
+ or die "Invalid process ID in file $pidf: [$amavisd_pid]";
$amavisd_pid = untaint($amavisd_pid);
- $kill_sig_used = 'TERM';
- kill($kill_sig_used,$amavisd_pid)
- or die "Can't SIGTERM amavisd[$amavisd_pid]: $!";
- my($waited) = 0; my($sigkill_sent) = 0; my($delay) = 1; # seconds
- for (;;) { # wait for the old running daemon to go away
- sleep($delay); $waited += $delay; $delay = 5;
- if (!kill(0,$amavisd_pid)) # is the old daemon still there?
- { $killed_amavisd_pid = $amavisd_pid; last } # old proc is gone, done
- if ($waited < 60 || $sigkill_sent) {
- print STDERR "Waiting for the process [$amavisd_pid] to terminate\n";
- } else { # use stronger hammer
- print STDERR "Sending SIGKILL to amavisd[$amavisd_pid]\n";
- $kill_sig_used = 'KILL';
- kill($kill_sig_used,$amavisd_pid)
- or warn "Can't SIGKILL amavisd[$amavisd_pid]: $!";
- $sigkill_sent = 1;
- }
- }
- };
- if ($@ ne '') { chomp($@); die "$@, can't $cmd the process\n" }
- my($msg) = "Daemon [$killed_amavisd_pid] terminated by SIG$kill_sig_used";
- if ($cmd eq 'stop') { print STDERR "$msg\n"; exit 0 }
- print STDERR "$msg, waiting for dust to settle...\n";
- sleep 5; # wait for the TCP socket to be released
- print STDERR "becoming a new daemon...\n";
-}
+ if (!kill(0,$amavisd_pid)) {
+ $! == ESRCH or die "Can't send SIG 0 to process [$amavisd_pid]: $!";
+ undef $amavisd_pid; # process does not exist
+ };
+ if (defined $amavisd_pid && defined $mtime) { # listed process does exist
+ # Is pid file older than system uptime? If so, it should be disregarded,
+ # it must not prevent starting up amavisd after unclean shutdown.
+ my($now) = time; my($uptime,$uptime_fmt); # sys uptime in seconds
+ my(@prog_args); my(@progs) = ('/usr/bin/uptime','uptime');
+ if (lc($^O) eq 'freebsd')
+ { @progs = ('/sbin/sysctl','sysctl'); @prog_args = 'kern.boottime' }
+ my($prog) = find_program_path(\@progs, [split(/:/,$path,-1)] );
+ if (!defined($prog)) { do_log(1,'No programs: %s',join(", ", at progs)) }
+ else {
+ my($proc_fh,$uppid) = run_command(undef,undef,$prog, at prog_args);
+ for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) {
+ local($1,$2,$3,$4); chomp($ln);
+ if (defined $uptime) {}
+ elsif ($ln =~ /{[^}]*\bsec\s*=\s*(\d+)[^}]*}/) { $uptime = $now-$1 }
+ # amazing how broken reports from uptime(1) soon after boot can be!
+ elsif ($ln =~ /\b up \s+ (?: (\d{1,4}) \s* days? )? [,\s]*
+ (\d{1,2}) : (\d{1,2}) (?: : (\d{1,2}))? (?! \d ) /ix
+ || $ln =~ /\b up (?: \s* \b (\d{1,4}) \s* days? )?
+ (?: [,\s]* \b (\d{1,2}) \s* hrs? )?
+ (?: [,\s]* \b (\d{1,2}) \s* mins? )?
+ (?: [,\s]* \b (\d{1,2}) \s* secs? )? /ix )
+ { $uptime = (($1*24 + $2)*60 + $3)*60 + $4 }
+ elsif ($ln =~ /\b (\d{1,2}) \s* secs?/ix) { $uptime = $1 } # OpenBSD
+ $uptime_fmt = format_time_interval($uptime);
+ do_log(5,"system uptime %s: %s",$uptime_fmt,$ln);
+ }
+ defined $ln || $!==0 || $!==EAGAIN or do_log(1,"Read uptime: %s",$!);
+ my($err)=0; $proc_fh->close or $err = $!; undef $proc_fh; undef $uppid;
+ proc_status_ok($?,$err)
+ or do_log(1,'Error running %s: %s', $prog, exit_status_str($?,$err));
+ }
+ if (!defined($uptime)) {
+ do_log(1,'Unable to determine system uptime, will trust PID file');
+ } elsif ($now-$mtime <= $uptime+70) {
+ do_log(1,'Valid PID file (younger than sys uptime %s)',$uptime_fmt);
+ } else {
+ undef $amavisd_pid;
+ do_log(1,'Ignoring stale PID file %s, older than system uptime %s',
+ $pidf,$uptime_fmt);
+ }
+ }
+ }
+
+ # act on command line parameter in $cmd
+ my($killed_amavisd_pid); my($kill_sig_used);
+ if ($cmd =~ /^(?:start|debug|debug-sa|foreground)?\z/) {
+ !defined($amavisd_pid)
+ or die "The amavisd daemon is already running, PID: [$amavisd_pid]\n";
+ $DEBUG=1 if $cmd eq 'debug';
+ $daemonize=0 if $cmd eq 'foreground';
+ $daemonize=0, $sa_debug=1 if $cmd eq 'debug-sa';
+ } elsif ($cmd !~ /^(?:reload|stop)\z/) {
+ die "$myversion: Unknown command line parameter: $cmd\n\n" . usage();
+ } else { # stop or reload
+ if (!defined($amavisd_pid)) { die "The amavisd daemon is not running\n" }
+ else { # first stop a running daemon
+ eval {
+ $kill_sig_used = 'TERM';
+ kill($kill_sig_used,$amavisd_pid) or $! == ESRCH
+ or die "Can't SIGTERM amavisd[$amavisd_pid]: $!";
+ my($waited) = 0; my($sigkill_sent) = 0; my($delay) = 1; # seconds
+ for (;;) { # wait for the old running daemon to go away
+ sleep($delay); $waited += $delay; $delay = 5;
+ if (!kill(0,$amavisd_pid)) { # is the old daemon still there?
+ $! == ESRCH or die "Can't send SIG 0 to amavisd[$amavisd_pid]: $!";
+ $killed_amavisd_pid = $amavisd_pid; # old process is gone, done
+ last;
+ }
+ if ($waited < 60 || $sigkill_sent) {
+ do_log(2,"Waiting for the process [%s] to terminate",$amavisd_pid);
+ print STDERR
+ "Waiting for the process [$amavisd_pid] to terminate\n";
+ } else { # use stronger hammer
+ do_log(2,"Sending SIGKILL to amavisd[%s]",$amavisd_pid);
+ print STDERR "Sending SIGKILL to amavisd[$amavisd_pid]\n";
+ $kill_sig_used = 'KILL';
+ kill($kill_sig_used,$amavisd_pid) or $! == ESRCH
+ or warn "Can't SIGKILL amavisd[$amavisd_pid]: $!";
+ $sigkill_sent = 1;
+ }
+ }
+ };
+ if ($@ ne '') { chomp $@; die "$@, can't $cmd the process\n" }
+ }
+ my($msg) = !defined($killed_amavisd_pid) ? undef :
+ "Daemon [$killed_amavisd_pid] terminated by SIG$kill_sig_used";
+ if ($cmd eq 'stop') {
+ if (defined $msg) { do_log(2,"%s",$msg); print STDERR "$msg\n" }
+ exit(0);
+ }
+ if (defined $killed_amavisd_pid) {
+ print STDERR "$msg, waiting for dust to settle...\n";
+ sleep 5; # wait for the TCP socket to be released
+ }
+ print STDERR "becoming a new daemon...\n";
+ }
+ 1;
+} or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ do_log(2,"%s", $eval_stat); die "$eval_stat\n";
+};
$daemonize = 0 if $DEBUG;
# Set path, home and term explictly. Don't trust environment
$ENV{PATH} = $path if $path ne '';
$ENV{HOME} = $helpers_home if $helpers_home ne '';
$ENV{TERM} = 'dumb'; $ENV{COLUMNS} = '80'; $ENV{LINES} = '100';
-
-Amavis::Log::init($DEBUG, $DO_SYSLOG, $LOGFILE);
-
+{ my($msg) = '';
+ $msg .= ", instance=$instance_name" if $instance_name ne '';
+ $msg .= ", eol=\"$eol\"" if $eol ne "\n";
+ $msg .= ", Unicode aware" if $unicode_aware;
+ for (qw(PERLIO LC_ALL LC_TYPE LC_CTYPE LANG))
+ { $msg .= sprintf(', %s="%s"', $_,$ENV{$_}) if $ENV{$_} ne '' }
+ do_log(0,"starting. %s at %s %s%s", $0, c('myhostname'), $myversion, $msg);
+}
# report version of Perl and process UID
-do_log(1, "user=%s, EUID: %s (%s); group=%s, EGID: %s (%s); log_level=%s",
- $desired_user, $>, $<, $desired_group, $), $(, c('log_level'));
-#do_log(c('log_level'), "Test log entry at log_level %s", c('log_level'));
+do_log(1, "user=%s, EUID: %s (%s); group=%s, EGID: %s (%s)",
+ $desired_user, $>, $<, $desired_group, $), $();
do_log(0, "Perl version %s", $]);
# insist on a FQDN in $myhostname
my($myhn) = c('myhostname');
@@ -10094,9 +11429,13 @@ fetch_modules_extra(); # bring addition
fetch_modules_extra(); # bring additional modules into memory and compile them
Amavis::SpamControl::init_pre_chroot() if $extra_code_antispam;
+if ($daemonize) { # log warnings and uncaught errors
+ $SIG{'__DIE__' } = sub { do_log(-1,"_DIE: %s", $_[0]) if !$^S };
+ $SIG{'__WARN__'} = sub { do_log( 3,"_WARN: %s",$_[0]) };
+}
+
# set up Net::Server configuration
-my $server = bless {
- server => {
+my($server) = Amavis->new({
# command args to be used after HUP must be untainted, deflt: [$0, at ARGV]
# commandline => ['/usr/local/sbin/amavisd','-c',$config_file[0] ],
commandline => [], # disable
@@ -10106,9 +11445,13 @@ my $server = bless {
: $inet_socket_bind),
max_servers => defined $max_servers_override ? $max_servers_override
: $max_servers, # number of pre-forked children
- max_requests => $max_requests, # restart child after that many accept's
- user => (($> == 0 || $< == 0) ? $daemon_user : undef),
- group => (($> == 0 || $< == 0) ? $daemon_group : undef),
+ !defined($min_servers) ? ()
+ : ( min_servers => $min_servers,
+ min_spare_servers => $min_spare_servers,
+ max_spare_servers => $max_spare_servers),
+ max_requests => $max_requests > 0 ? $max_requests : 2E9, # avoid dflt 1000
+ user => ($> == 0 || $< == 0) ? $daemon_user : undef,
+ group => ($> == 0 || $< == 0) ? $daemon_group : undef,
pid_file => defined $pid_file_override ? $pid_file_override : $pid_file,
# socket serialization lockfile
lock_file => defined $lock_file_override? $lock_file_override: $lock_file,
@@ -10121,10 +11464,9 @@ my $server = bless {
no_client_stdout => (Net::Server->VERSION >= 0.93 ? 1 : 0),
# controls log level for Net::Server internal log messages:
# 0=err, 1=warning, 2=notice, 3=info, 4=debug
- log_level => $DEBUG ? 4 : 2,
+ log_level => ($DEBUG || c('log_level') >= 5) ? 4 : 2,
log_file => undef, # will be overridden to call do_log()
- },
-}, 'Amavis';
+});
$0 = 'amavisd (master)';
$server->run; # transfer control to Net::Server
@@ -10145,24 +11487,26 @@ use warnings FATAL => 'utf8';
use warnings FATAL => 'utf8';
BEGIN {
- import Amavis::Conf qw($myversion $myhostname);
+ import Amavis::Conf qw(:platform $myversion $myhostname
+ $nanny_details_level);
import Amavis::Util qw(ll do_log snmp_counters_get
add_entropy fetch_entropy);
}
use BerkeleyDB;
+use Time::HiRes ();
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
# open existing databases (called by each child process)
sub new {
my($class,$db_env) = @_; $! = 0; my($env) = $db_env->get_db_env;
- defined $env or die "BDB bad db env.: $BerkeleyDB::Error, $!.";
+ defined $env or die "BDB get_db_env (dbS/dbN): $BerkeleyDB::Error, $!.";
$! = 0; my($dbs) = BerkeleyDB::Hash->new(-Filename=>'snmp.db', -Env=>$env);
defined $dbs or die "BDB no dbS: $BerkeleyDB::Error, $!.";
$! = 0; my($dbn) = BerkeleyDB::Hash->new(-Filename=>'nanny.db',-Env=>$env);
@@ -10172,12 +11516,21 @@ sub new {
sub DESTROY {
my($self) = shift;
- eval { do_log(5,"Amavis::DB::SNMP DESTROY called") };
- for my $db ($self->{'db_snmp'}, $self->{'db_nanny'}) {
- if (defined $db) {
- eval { $db->db_close==0 or die "db_close: $BerkeleyDB::Error, $!." };
- if ($@ ne '') { warn "BDB S+N DESTROY INFO $@" }
- undef $db;
+ if (defined($my_pid) && $$ != $my_pid) {
+ eval { do_log(5,"Amavis::DB::SNMP DESTROY skip, clone [%s] (born as [%s])",
+ $$,$my_pid) };
+ } else {
+ eval { do_log(5,"Amavis::DB::SNMP DESTROY called") };
+ for my $db_name ('db_snmp', 'db_nanny') {
+ my($db) = $self->{$db_name};
+ if (defined $db) {
+ eval {
+ $db->db_close==0 or die "db_close: $BerkeleyDB::Error, $!."; 1;
+ } or do { $@ = "errno=$!" if $@ eq '' };
+ if ($@ ne '' && $@ !~ /\bDatabase is already closed\b/)
+ { warn "[$$] BDB S+N DESTROY INFO ($db_name): $@" }
+ undef $db;
+ }
}
}
}
@@ -10225,13 +11578,17 @@ sub update_snmp_variables {
defined $cursor or die "db_cursor: $BerkeleyDB::Error, $!.";
for my $key (@$snmp_var_names_ref) {
my($snmp_var_name,$arg,$type) = ref $key ? @$key : ($key);
- $type = 'C32' if !defined($type) || $type eq '';
- $arg = 1 if !defined($arg) && $type eq 'C32';
+ $type = 'C32' if !defined($type) || $type eq '';
+ if ($type eq 'C32' || $type eq 'C64') {
+ if (!defined($arg)) { $arg = 1 } # by default counter increments by 1
+ elsif ($arg < 0) { $arg = 0 } # counter is supposed to be unsigned
+ }
my($val,$flags); local($1);
my($stat) = $cursor->c_get($snmp_var_name,$val,DB_SET);
- if ($stat==0) { # exists, update it
+ if ($stat==0) { # exists, update it (or replace it)
if ($type eq 'C32' && $val=~/^C32 (\d+)\z/) { $val = $1+$arg }
- elsif ($type eq 'INT' && $val=~/^INT (\d+)\z/) { $val = $arg }
+ elsif ($type eq 'C64' && $val=~/^C64 (\d+)\z/) { $val = $1+$arg }
+ elsif ($type eq 'INT' && $val=~/^INT ([+-]?\d+)\z/) { $val = $arg }
elsif ($type=~/^(STR|OID)\z/ && $val=~/^\Q$type\E (.*)\z/) {
if ($snmp_var_name ne 'entropy') { $val = $arg }
else { # blend-in entropy
@@ -10245,99 +11602,116 @@ sub update_snmp_variables {
$stat==DB_NOTFOUND or die "c_get: $BerkeleyDB::Error, $!.";
$flags = DB_KEYLAST; $val = $arg;
}
- my($str) = $type =~ /^(C32|INT)\z/ ? sprintf("%010d",$val) : $val;
+ my($fmt) = $type eq 'C32' ? "%010d" : $type eq 'C64' ? "%020d"
+ : $type eq 'INT' ? "%010d" : undef;
+ # format for INT should really be %011d, but keep compatibility for now
+ my($str) = defined($fmt) ? sprintf($fmt,$val) : $val;
$cursor->c_put($snmp_var_name, "$type $str", $flags) == 0
or die "c_put: $BerkeleyDB::Error, $!.";
}
$cursor->c_close==0 or die "c_close: $BerkeleyDB::Error, $!.";
- $cursor = undef;
- };
- $eval_stat = $@;
+ undef $cursor; 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
if (defined $db) {
$cursor->c_close if defined $cursor; # unlock, ignoring status
- $cursor = undef;
+ undef $cursor;
# if ($eval_stat eq '') {
# my($stat); $db->db_sync(); # not really needed
-# $stat==0 or warn "BDB S db_sync, status $stat: $BerkeleyDB::Error, $!.";
+# $stat==0 or warn "BDB S db_sync,status $stat: $BerkeleyDB::Error, $!.";
# }
}
- }
+ }; # restore signal handlers
delete $self->{'cnt'};
- if ($interrupt ne '') { kill($interrupt,$$) } # resignal
- elsif ($eval_stat ne '')
- { chomp($eval_stat); die "update_snmp_variables: BDB S $eval_stat\n" }
+ if ($interrupt ne '') { kill($interrupt,$$) } # resignal, ignoring status
+ elsif ($eval_stat ne '') {
+ chomp $eval_stat;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ die "update_snmp_variables: BDB S $eval_stat\n";
+ }
}
sub read_snmp_variables {
my($self, at snmp_var_names) = @_;
my($eval_stat,$interrupt); $interrupt = '';
my($db) = $self->{'db_snmp'}; my($cursor); my(@values);
- my($h1) = sub { $interrupt = $_[0] };
- local(@SIG{qw(INT HUP TERM TSTP QUIT ALRM USR1 USR2)}) = ($h1) x 8;
- eval { # ensure cursor will be unlocked even in case of errors or signals
- $cursor = $db->db_cursor; # obtain read lock
- defined $cursor or die "db_cursor: $BerkeleyDB::Error, $!.";
- for my $cname (@snmp_var_names) {
- my($val); my($stat) = $cursor->c_get($cname,$val,DB_SET);
- push(@values, $stat==0 ? $val : undef);
- $stat==0 || $stat==DB_NOTFOUND or die "c_get: $BerkeleyDB::Error, $!.";
- }
- $cursor->c_close==0 or die "c_close: $BerkeleyDB::Error, $!.";
- $cursor = undef;
- };
- $eval_stat = $@;
- if (defined $db) {
- $cursor->c_close if defined $cursor; # unlock, ignoring status
- $cursor = undef;
- }
- if ($interrupt ne '') { kill($interrupt,$$) } # resignal
- elsif ($eval_stat ne '')
- { chomp($eval_stat); die "read_snmp_variables: BDB S $eval_stat\n" }
+ { my($h1) = sub { $interrupt = $_[0] };
+ local(@SIG{qw(INT HUP TERM TSTP QUIT ALRM USR1 USR2)}) = ($h1) x 8;
+ eval { # ensure cursor will be unlocked even in case of errors or signals
+ $cursor = $db->db_cursor; # obtain read lock
+ defined $cursor or die "db_cursor: $BerkeleyDB::Error, $!.";
+ for my $cname (@snmp_var_names) {
+ my($val); my($stat) = $cursor->c_get($cname,$val,DB_SET);
+ push(@values, $stat==0 ? $val : undef);
+ $stat==0 || $stat==DB_NOTFOUND or die "c_get: $BerkeleyDB::Error, $!.";
+ }
+ $cursor->c_close==0 or die "c_close: $BerkeleyDB::Error, $!.";
+ undef $cursor; 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ if (defined $db) {
+ $cursor->c_close if defined $cursor; # unlock, ignoring status
+ undef $cursor;
+ }
+ }; # restore signal handlers
+ if ($interrupt ne '') { kill($interrupt,$$) } # resignal, ignoring status
+ elsif ($eval_stat ne '') {
+ chomp $eval_stat;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ die "read_snmp_variables: BDB S $eval_stat\n";
+ }
for my $val (@values) {
if (!defined($val)) {} # keep undefined
- elsif ($val =~ /^(?:C32|INT) (\d+)\z/) { $val = 0+$1 }
- elsif ($val =~ /^(?:STR|OID) (.*)\z/) { $val = $1 }
+ elsif ($val =~ /^(?:C32|C64|INT) (\d+)\z/) { $val = 0+$1 }
+ elsif ($val =~ /^(?:STR|OID) (.*)\z/) { $val = $1 }
else { do_log(-2,"WARN: counter syntax? %s", $val); $val = undef }
}
\@values;
}
sub register_proc {
- my($self,$task_id) = @_;
- my($db) = $self->{'db_nanny'}; my($cursor);
- my($val,$new_val); my($key) = sprintf("%05d",$$);
- $new_val = sprintf("%010d %-12s", time, $task_id) if defined $task_id;
- my($eval_stat,$interrupt); $interrupt = '';
- my($h1) = sub { $interrupt = $_[0] };
- local(@SIG{qw(INT HUP TERM TSTP QUIT ALRM USR1 USR2)}) = ($h1) x 8;
- eval { # ensure cursor will be unlocked even in case of errors or signals
- $cursor = $db->db_cursor(DB_WRITECURSOR); # obtain write lock
- defined $cursor or die "db_cursor: $BerkeleyDB::Error, $!.";
- my($stat) = $cursor->c_get($key,$val,DB_SET);
- $stat==0 || $stat==DB_NOTFOUND or die "c_get: $BerkeleyDB::Error, $!.";
- if ($stat==0 && !defined $task_id) { # remove existing entry
- $cursor->c_del==0 or die "c_del: $BerkeleyDB::Error, $!.";
- } elsif (defined $task_id && !($stat==0 && $new_val eq $val)) {
- # add new, or update existing entry if different
- $cursor->c_put($key, $new_val,
- $stat==0 ? DB_CURRENT : DB_KEYLAST ) == 0
- or die "c_put: $BerkeleyDB::Error, $!.";
- }
- $cursor->c_close==0 or die "c_close: $BerkeleyDB::Error, $!.";
- $cursor = undef;
- };
- $eval_stat = $@;
- if (defined $db) {
- $cursor->c_close if defined $cursor; # unlock, ignoring status
- $cursor = undef;
-# if ($eval_stat eq '') {
-# my($stat) = $db->db_sync(); # not really needed
-# $stat==0 or warn "BDB N db_sync, status $stat: $BerkeleyDB::Error, $!.";
-# }
- }
- if ($interrupt ne '') { kill($interrupt,$$) } # resignal
- elsif ($eval_stat ne '')
- { chomp($eval_stat); die "register_proc: BDB N $eval_stat\n" }
+ my($self,$details_level,$reset_timestamp,$state,$task_id) = @_;
+ my($eval_stat) = ''; my($interrupt) = '';
+ if (!defined($state) || $details_level <= $nanny_details_level) {
+ $task_id = '' if !defined $task_id;
+ my($db) = $self->{'db_nanny'}; my($key) = sprintf("%05d",$$);
+ my($cursor); my($val);
+ my($h1) = sub { $interrupt = $_[0] };
+ local(@SIG{qw(INT HUP TERM TSTP QUIT ALRM USR1 USR2)}) = ($h1) x 8;
+ eval { # ensure cursor will be unlocked even in case of errors or signals
+ $cursor = $db->db_cursor(DB_WRITECURSOR); # obtain write lock
+ defined $cursor or die "db_cursor: $BerkeleyDB::Error, $!.";
+ my($stat) = $cursor->c_get($key,$val,DB_SET);
+ $stat==0 || $stat==DB_NOTFOUND or die "c_get: $BerkeleyDB::Error, $!.";
+ if ($stat==0 && !defined $state) { # remove existing entry
+ $cursor->c_del==0 or die "c_del: $BerkeleyDB::Error, $!.";
+ } elsif (defined $state) { # add new, or update existing entry
+ my($timestamp); local($1);
+ # keep its timestamp when updating existing record
+ $timestamp = $1 if $stat==0 && $val=~/^(\d+(?:\.\d*)?) /s;
+ $timestamp = sprintf("%014.3f", Time::HiRes::time)
+ if !defined($timestamp) || $reset_timestamp;
+ my($new_val) = sprintf("%s %-14s", $timestamp, $state.$task_id);
+ $cursor->c_put($key, $new_val,
+ $stat==0 ? DB_CURRENT : DB_KEYLAST ) == 0
+ or die "c_put: $BerkeleyDB::Error, $!.";
+ }
+ $cursor->c_close==0 or die "c_close: $BerkeleyDB::Error, $!.";
+ undef $cursor; 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ if (defined $db) {
+ $cursor->c_close if defined $cursor; # unlock, ignoring status
+ undef $cursor;
+# if ($eval_stat eq '') {
+# my($stat) = $db->db_sync(); # not really needed
+# $stat==0 or warn "BDB N db_sync,status $stat: $BerkeleyDB::Error, $!.";
+# }
+ }
+ }; # restore signal handlers
+ if ($interrupt ne '') { kill($interrupt,$$) } # resignal, ignoring status
+ elsif ($eval_stat ne '') {
+ chomp $eval_stat;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ die "register_proc: BDB N $eval_stat\n";
+ }
}
1;
@@ -10358,17 +11732,17 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
# create new databases, then close them (called by the parent process)
# (called only if $db_home is nonempty)
-sub init($) {
- my($predelete) = @_; # delete existing db files first?
+sub init($$) {
+ my($predelete,$cache_keysize) = @_;
my($name) = $db_home;
$name = "$daemon_chroot_dir $name" if $daemon_chroot_dir ne '';
- if ($predelete) { # delete old database files
+ if ($predelete) { # delete existing db files first?
local(*DIR);
opendir(DIR,$db_home) or die "db_init: Can't open directory $name: $!";
my(@dirfiles) = readdir(DIR); #must avoid modifying dir while traversing it
@@ -10384,7 +11758,7 @@ sub init($) {
$! = 0; my($env) = BerkeleyDB::Env->new(-Home=>$db_home, -Mode=>0640,
-Flags=> DB_CREATE | DB_INIT_CDB | DB_INIT_MPOOL);
defined $env
- or die "db_init: BDB bad db env. at $db_home: $BerkeleyDB::Error, $!.";
+ or die "BDB can't create db env. at $db_home: $BerkeleyDB::Error, $!.";
do_log(0, "Creating db in %s/; BerkeleyDB %s, libdb %s",
$name, BerkeleyDB->VERSION, $BerkeleyDB::db_version);
$! = 0; my($dbc) = BerkeleyDB::Hash->new(
@@ -10392,7 +11766,7 @@ sub init($) {
defined $dbc or die "db_init: BDB no dbC: $BerkeleyDB::Error, $!.";
$! = 0; my($dbq) = BerkeleyDB::Queue->new(
-Filename=>'cache-expiry.db', -Flags=>DB_CREATE, -Env=>$env,
- -Len=>15+1+32 ); # '-ExtentSize' needs DB 3.2.x, e.g. -ExtentSize=>2
+ -Len=>$cache_keysize); # '-ExtentSize' needs DB 3.2.x, e.g. -ExtentSize=>2
defined $dbq or die "db_init: BDB no dbQ: $BerkeleyDB::Error, $!.";
$! = 0; my($dbs) = BerkeleyDB::Hash->new(
-Filename=>'snmp.db', -Flags=>DB_CREATE, -Env=>$env );
@@ -10411,9 +11785,10 @@ sub new {
sub new {
my($class) = @_; my($env);
if (defined $db_home) {
- $env = BerkeleyDB::Env->new(
+ $! = 0; $env = BerkeleyDB::Env->new(
-Home=>$db_home, -Mode=>0640, -Flags=> DB_INIT_CDB | DB_INIT_MPOOL);
- defined $env or die "BDB bad db env. at $db_home: $BerkeleyDB::Error, $!.";
+ defined $env
+ or die "BDB can't connect db env. at $db_home: $BerkeleyDB::Error, $!.";
}
bless \$env, $class;
}
@@ -10432,7 +11807,8 @@ use warnings FATAL => 'utf8';
use warnings FATAL => 'utf8';
BEGIN {
- import Amavis::Util qw(ll do_log);
+ import Amavis::Conf qw(:platform);
+ import Amavis::Util qw(ll do_log freeze thaw);
}
use BerkeleyDB;
@@ -10440,38 +11816,48 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.0722';
+ $VERSION = '2.0912';
@ISA = qw(Exporter);
}
# open existing databases (called by each child process);
# if $db_env is undef a memory-based cache is created, otherwise use BerkeleyDB
sub new {
- my($class,$db_env) = @_;
+ my($class,$db_env,$keysize) = @_;
my($dbc,$dbq,$mem_cache);
if (!defined($db_env)) {
do_log(1,"BerkeleyDB not available, using memory-based local cache");
$mem_cache = {};
} else {
my($env) = $db_env->get_db_env;
- defined $env or die "BDB bad db env.: $BerkeleyDB::Error, $!.";
+ defined $env or die "BDB get_db_env (dbC/dbQ): $BerkeleyDB::Error, $!.";
$dbc = BerkeleyDB::Hash->new(-Filename=>'cache.db', -Env=>$env);
defined $dbc or die "BDB no dbC: $BerkeleyDB::Error, $!.";
$dbq = BerkeleyDB::Queue->new(-Filename=>'cache-expiry.db', -Env=>$env,
- -Len=>15+1+32); # '-ExtentSize' needs DB 3.2.x, e.g. -ExtentSize=>2
+ -Len=>$keysize); # '-ExtentSize' needs DB 3.2.x, e.g. -ExtentSize=>2
defined $dbq or die "BDB no dbQ: $BerkeleyDB::Error, $!.";
}
- bless {'db_cache'=>$dbc, 'db_queue'=>$dbq, 'mem_cache'=>$mem_cache}, $class;
+ bless {'db_cache'=>$dbc, 'db_queue'=>$dbq,
+ 'mem_cache'=>$mem_cache, 'key_size'=>$keysize}, $class;
}
sub DESTROY {
my($self) = shift;
- eval { do_log(5,"Amavis::Cache DESTROY called") };
- for my $db ($self->{'db_cache'}, $self->{'db_queue'}) {
- if (defined $db) {
- eval { $db->db_close==0 or die "db_close: $BerkeleyDB::Error, $!." };
- if ($@ ne '') { warn "BDB C+Q DESTROY INFO $@" }
- undef $db;
+ if (defined($my_pid) && $$ != $my_pid) {
+ eval { do_log(5,"Amavis::Cache DESTROY skip, clone [%s] (born as [%s])",
+ $$,$my_pid) };
+ } else {
+ eval { do_log(5,"Amavis::Cache DESTROY called") };
+ for my $db_name ('db_cache', 'db_queue') {
+ my($db) = $self->{$db_name};
+ if (defined $db) {
+ eval {
+ $db->db_close==0 or die "db_close: $BerkeleyDB::Error, $!."; 1;
+ } or do { $@ = "errno=$!" if $@ eq '' };
+ if ($@ ne '' && $@ !~ /\bDatabase is already closed\b/)
+ { warn "[$$] BDB C+Q DESTROY INFO ($db_name): $@" }
+ undef $db;
+ }
}
}
}
@@ -10531,7 +11917,7 @@ sub get {
} else {
my($stat) = $db->db_get($key,$val);
$stat==0 || $stat==DB_NOTFOUND
- or die "BDB C c_get: $BerkeleyDB::Error, $!.";
+ or die "BDB Cg c_get: $BerkeleyDB::Error, $!.";
local($1,$2);
if ($stat==0 && $val=~/^([^ ]+) (.*)/s) { $val = $2 } else { $val = undef }
}
@@ -10545,16 +11931,16 @@ sub set {
$self->{'mem_cache'}{$key} = freeze($obj);
} else {
my($cursor) = $db->db_cursor(DB_WRITECURSOR);
- defined $cursor or die "BDB C db_cursor: $BerkeleyDB::Error, $!.";
+ defined $cursor or die "BDB Cs db_cursor: $BerkeleyDB::Error, $!.";
my($val); my($stat) = $cursor->c_get($key,$val,DB_SET);
$stat==0 || $stat==DB_NOTFOUND
- or die "BDB C c_get: $BerkeleyDB::Error, $!.";
+ or die "BDB Cs c_get: $BerkeleyDB::Error, $!.";
$cursor->c_put($key, $expires_utc_iso8601.' '.freeze($obj),
$stat==0 ? DB_CURRENT : DB_KEYLAST ) == 0
- or die "BDB C c_put: $BerkeleyDB::Error, $!.";
- $cursor->c_close==0 or die "BDB C c_close: $BerkeleyDB::Error, $!.";
+ or die "BDB Cs c_put: $BerkeleyDB::Error, $!.";
+ $cursor->c_close==0 or die "BDB Cs c_close: $BerkeleyDB::Error, $!.";
# $stat = $db->db_sync(); # only worth doing if cache were persistent
- # $stat==0 or warn "BDB C db_sync, status $stat: $BerkeleyDB::Error, $!.";
+ # $stat==0 or warn "BDB Cs db_sync, status $stat: $BerkeleyDB::Error, $!.";
$self->enqueue($key,$now_utc_iso8601,$expires_utc_iso8601);
}
$obj;
@@ -10573,7 +11959,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
BEGIN {
@@ -10673,11 +12059,11 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
-use DBI;
+use DBI qw(:sql_types);
BEGIN {
import Amavis::Conf qw(:platform :confvars c cr ca);
@@ -10694,7 +12080,7 @@ sub new {
else {
# $clause_name is a key into %sql_clause of the currently selected
# policy bank; one level of indirection is allowed in %sql_clause result,
- # the resulting SQL clause may include %k, to be expanded
+ # the resulting SQL clause may include %k or %a, to be expanded
bless { conn_h => $conn_h, incarnation => 0, clause_name => $clause_name },
$class;
}
@@ -10790,8 +12176,11 @@ sub lookup_sql($$$;$) {
my($n) = sprintf("%d",scalar(@$keys_ref)); # number of keys
my(@extras_tmp) = !ref $extra_args ? () : @$extra_args;
local($1); my(@pos_args); my($sel_taint) = substr($sel,0,0); # taintedness
- $sel =~ s{ ( %k | \? ) } # substitute %k for keys and ? for each extra arg
- { push(@pos_args, $1 eq '%k' ? @$keys_ref : shift @extras_tmp),
+ $sel =~ s{ ( %k | %a | \? ) } # substitute %k for keys, %a for exact mail
+ # address, and ? for each extra arg
+ { push(@pos_args, $1 eq '%k' ? @$keys_ref
+ : $1 eq '%a' ? $keys_ref->[0] # same as first in %k
+ : shift @extras_tmp),
$1 eq '%k' ? join(',', ('?') x $n) : '?' }gxe;
$sel = untaint($sel) . $sel_taint; # keep original clause taintedness
$_ = untaint($_) for @pos_args; # untaint arguments
@@ -10825,12 +12214,13 @@ sub lookup_sql($$$;$) {
last if !$get_all;
}
$conn_h->finish($sel) if defined $a_ref; # only if not all read
- }; # eval
- if ($@ ne '') {
- my($err) = $@; chomp($err);
+ 1;
+ } or do {
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err;
do_log(-1, "lookup_sql: %s, %s, %s", $err, $DBI::err, $DBI::errstr);
+ die $err if $err =~ /^timed out\b/; # resignal timeout
die $err;
- }
+ };
if (!ll(4)) {
# don't bother preparing log report which will not be printed
} elsif (!@result) {
@@ -10859,7 +12249,7 @@ BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION
$ldap_sys_default);
- $VERSION= '2.072';
+ $VERSION= '2.091';
@ISA = qw(Exporter);
import Amavis::Conf qw(:platform :confvars c cr ca);
@@ -10982,9 +12372,10 @@ sub do_search {
deref => $self->{deref},
);
if ($result->code) { die $result->error_name, "\n"; }
- };
- if ($@ ne '') {
- my($err) = $@; chomp $err;
+ 1;
+ } or do {
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err;
+ die $err if $err =~ /^timed out\b/; # resignal timeout
if ($err !~ /^LDAP_/) {
die "do_search: $err";
} else { # LDAP related error
@@ -11002,14 +12393,15 @@ sub do_search {
deref => $self->{deref},
);
if ($result->code) { die $result->error_name, "\n"; }
+ 1;
+ } or do {
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err;
+ $self->disconnect_from_ldap;
+ die $err if $err =~ /^timed out\b/; # resignal timeout
+ die "do_search: failed again, $err";
};
- if ($@ ne '') {
- my($err) = $@; chomp $err;
- $self->disconnect_from_ldap;
- die "do_search: failed again, $err";
- }
- }
- }
+ };
+ };
$result;
}
@@ -11023,7 +12415,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
import Amavis::Util qw(ll do_log);
@@ -11118,7 +12510,7 @@ BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION
$ldap_sys_default @ldap_attrs @mv_ldap_attrs);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
import Amavis::Conf qw(:platform :confvars c cr ca);
@@ -11287,12 +12679,12 @@ sub lookup_ldap($$$) {
push(@tmp_matchingkey, [$pos,$key_str]);
last if !$get_all;
}
- }; # eval
- if ($@ ne '') {
- my($err) = $@; chomp $err;
+ 1;
+ } or do {
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err;
do_log(-1,"lookup_ldap: %s", $err);
die $err;
- }
+ };
@result = map {$_->[1]} sort {$a->[0] <=> $b->[0]} @tmp_result;
@matchingkey = map {$_->[1]} sort {$a->[0] <=> $b->[0]} @tmp_matchingkey;
if (!ll(4)) {
@@ -11322,7 +12714,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
@@ -11334,7 +12726,7 @@ BEGIN {
BEGIN {
import Amavis::Conf qw(:platform :confvars c cr ca);
import Amavis::Util qw(ll do_log debug_oneshot snmp_counters_init
- snmp_count untaint waiting_for_client
+ snmp_count untaint orcpt_encode waiting_for_client
switch_to_my_time switch_to_client_time
am_id new_am_id add_entropy rmdir_recursively);
import Amavis::Lookup qw(lookup);
@@ -11384,14 +12776,17 @@ sub process_policy_request($$$$) {
$attr{'recipient'} = \@recips; $attr{'ldaargs'} = \@ldaargs;
$attr{'delivery_care_of'} = @ldaargs ? 'client' : 'server';
eval {
- my($msginfo) = preprocess_policy_query(\%attr);
+ my($msginfo,$bank_names_ref) = preprocess_policy_query(\%attr);
$response = (map { local($1); /^exit_code=(\d+)\z/ ? $1 : () }
- check_amcl_policy($conn,$msginfo,$check_mail,1))[0];
+ check_amcl_policy($conn,$msginfo,$check_mail,1,
+ $bank_names_ref))[0];
+ 1;
+ } or do {
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err;
+ do_log(-2, "policy_server FAILED: %s", $err);
+ $response = EX_TEMPFAIL;
+ die $err if $err =~ /^timed out\b/; # resignal timeout
};
- if ($@ ne '') {
- chomp($@); do_log(-2, "policy_server FAILED: %s", $@);
- $response = EX_TEMPFAIL;
- }
$state = 4;
} elsif ($state == 2) {
push(@recips, $inbuff);
@@ -11430,20 +12825,24 @@ sub process_policy_request($$$$) {
if ($end_of_request) { # end of request
section_time('got data');
eval {
- my($msginfo) = preprocess_policy_query(\%attr);
- @response = $attr{'request'} eq 'smtpd_access_policy'
+ my($msginfo,$bank_names_ref) = preprocess_policy_query(\%attr);
+ my($req) = lc($attr{'request'});
+ @response = $req eq 'smtpd_access_policy'
? postfix_policy($conn,$msginfo,\%attr)
- : $attr{'request'} eq 'release'
- ? dispatch_from_quarantine($conn,$msginfo)
- : check_amcl_policy($conn,$msginfo,$check_mail,0);
- };
- if ($@ ne '') {
- chomp($@); do_log(-2, "policy_server FAILED: %s", $@);
- @response = (proto_encode('setreply','450','4.5.0',"Failure: $@"),
+ : $req eq 'release' || $req eq 'requeue'
+ ? dispatch_from_quarantine($conn,$msginfo,$req)
+ : check_amcl_policy($conn,$msginfo,$check_mail,0,
+ $bank_names_ref);
+ 1;
+ } or do {
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err;
+ do_log(-2, "policy_server FAILED: %s", $err);
+ @response = (proto_encode('setreply','450','4.5.0',"Failure: $err"),
proto_encode('return_value','tempfail'),
proto_encode('exit_code',sprintf("%d",EX_TEMPFAIL)));
+ die $err if $err =~ /^timed out\b/; # resignal timeout
# last;
- }
+ };
$sock->print( map { $_."\015\012" } (@response,'') )
or die "Can't write response to socket: $!, fileno=".fileno($sock);
%attr = (); @response = ();
@@ -11484,6 +12883,7 @@ sub preprocess_policy_query($) {
my($msginfo) = Amavis::In::Message->new;
$msginfo->rx_time(time); # now
+ $msginfo->log_id(am_id());
$msginfo->add_contents_category(CC_CLEAN,0);
add_entropy(%$attr_ref);
@@ -11506,8 +12906,9 @@ sub preprocess_policy_query($) {
# protocol_name=ESMTP
# helo_name=b.example.com
# client_address=10.2.3.4
- # Required 'release' fields are: request, mail_id
- # request=release
+ # policy_bank=TLS,ORIGINATING,MYNETS,...
+ # Required 'release' or 'requeue' fields are: request, mail_id
+ # request=release (or request=requeue)
# mail_id=xxxxxxxxxxxx
# secret_id=xxxxxxxxxxxx (authorizes a release)
# quar_type=x F/Z/B/Q/M (defaults to Q or F)
@@ -11518,40 +12919,50 @@ sub preprocess_policy_query($) {
# recipient=<bar1 at example.net> (optional: replaces envelope recips)
# recipient=<bar2 at example.net>
# recipient=<bar3 at example.net>
- my($sender, at recips);
+ my(@recips); my(@bank_names);
exists $attr_ref->{'request'} or die "Missing 'request' field";
- my($ampdp) = $attr_ref->{'request'} =~ /^AM\.CL|AM\.PDP|release\z/i;
+ my($ampdp) = $attr_ref->{'request'} =~ /^AM\.CL|AM\.PDP|release|requeue\z/i;
+ @bank_names = grep { $_ ne '' } map { s/^[ \t]+//; s/[ \t]+\z//; $_ }
+ split(/,/, $attr_ref->{'policy_bank'})
+ if exists $attr_ref->{'policy_bank'};
$msginfo->delivery_method(
lc($attr_ref->{'delivery_care_of'}) eq 'server' ? c('forward_method') :'');
$msginfo->client_delete(lc($attr_ref->{'tempdir_removed_by'}) eq 'client'
? 1 : 0);
$msginfo->queue_id($attr_ref->{'queue_id'})
if exists $attr_ref->{'queue_id'};
- if (exists $attr_ref->{'client_address'}) {
- my($cl_ip) = $attr_ref->{'client_address'};
- my($cl_ip_mynets) = $cl_ip eq '' ? undef :
- lookup_ip_acl($cl_ip,@{ca('mynetworks_maps')});
- $msginfo->client_addr($cl_ip); $msginfo->client_addr_mynets($cl_ip_mynets);
- }
+ $msginfo->client_addr($attr_ref->{'client_address'})
+ if exists $attr_ref->{'client_address'};
$msginfo->client_name($attr_ref->{'client_name'})
if exists $attr_ref->{'client_name'};
$msginfo->client_proto($attr_ref->{'protocol_name'})
if exists $attr_ref->{'protocol_name'};
$msginfo->client_helo($attr_ref->{'helo_name'})
if exists $attr_ref->{'helo_name'};
-# $msginfo->body_type('8BITMIME'); # get_body_digest will set this if undef
+# $msginfo->body_type('8BITMIME');
$msginfo->requested_by(unquote_rfc2821_local($attr_ref->{'requested_by'}))
if exists $attr_ref->{'requested_by'};
if (exists $attr_ref->{'sender'}) {
- $sender = $attr_ref->{'sender'};
+ my($sender) = $attr_ref->{'sender'};
+ $sender = '<'.$sender.'>' if $sender !~ /^<.*>\z/;
+ $msginfo->sender_smtp($sender);
$sender = unquote_rfc2821_local($sender);
$msginfo->sender($sender);
}
if (exists $attr_ref->{'recipient'}) {
- my($r) = $attr_ref->{'recipient'};
- @recips = !ref($r) ? $r : @$r;
- $_ = unquote_rfc2821_local($_) for @recips;
- $msginfo->recips(\@recips);
+ my($r) = $attr_ref->{'recipient'}; @recips = ();
+ for my $addr (!ref($r) ? $r : @$r) {
+ my($addr_quo) = $addr;
+ my($addr_unq) = unquote_rfc2821_local($addr);
+ $addr_quo = '<'.$addr_quo.'>' if $addr_quo !~ /^<.*>\z/;
+ my($recip_obj) = Amavis::In::Message::PerRecip->new;
+ $recip_obj->recip_addr($addr_unq);
+ $recip_obj->recip_addr_smtp($addr_quo);
+ $recip_obj->dsn_orcpt(orcpt_encode($addr_quo));
+ $recip_obj->recip_destiny(D_PASS); # default is Pass
+ push(@recips,$recip_obj);
+ }
+ $msginfo->per_recip_data(\@recips);
}
if (!exists $attr_ref->{'tempdir'}) {
$msginfo->mail_tempdir($TEMPBASE); # defaults to $TEMPBASE
@@ -11564,16 +12975,21 @@ sub preprocess_policy_query($) {
}
my($quar_type);
if (!$ampdp) {} # don't bother with filenames
- elsif ($attr_ref->{'request'} eq 'release') {
+ elsif ($attr_ref->{'request'} eq 'release' ||
+ $attr_ref->{'request'} eq 'requeue') {
exists $attr_ref->{'mail_id'} or die "Missing 'mail_id' field";
my($fn) = $attr_ref->{'mail_id'};
- $fn =~ m{^[A-Za-z0-9][A-Za-z0-9/_.+-]*\z}s or die "Invalid mail_id '$fn'";
+ # amavisd almost-base64: 62 +, 63 -
+ # rfc4648 base64: 62 +, 63 /
+ # rfc4648 base64url: 62 -, 63 _
+ $fn =~ m{^ [A-Za-z0-9] [A-Za-z0-9/_.+-]* ={0,2} \z}xs
+ or die "Invalid mail_id '$fn'";
$msginfo->mail_id($fn);
if (!exists($attr_ref->{'secret_id'}) || $attr_ref->{'secret_id'} eq '') {
die "Secret_id is required, but missing" if c('auth_required_release');
} else {
my($id) = Digest::MD5->new->add($attr_ref->{'secret_id'})->b64digest;
- $id = substr($id,0,12); $id =~ tr{/}{-};
+ $id = substr($id,0,12); $id =~ tr{/}{-}; # base64 -> almost-base64
$id eq $fn or die "Result $id of secret_id does not match mail_id $fn";
}
$quar_type = $attr_ref->{'quar_type'};
@@ -11601,8 +13017,9 @@ sub preprocess_policy_query($) {
}
if ($ampdp && $msginfo->mail_text_fn ne '') {
my($fh); my($fname) = $msginfo->mail_text_fn;
- new_am_id('rel-'.$msginfo->mail_id) if $attr_ref->{'request'} eq 'release';
- if ($attr_ref->{'request'} eq 'release' && $quar_type eq 'Q') {
+ my($releasing) = $attr_ref->{'request'} =~ /^(?:release|requeue)\z/i;
+ new_am_id('rel-'.$msginfo->mail_id) if $releasing;
+ if ($releasing && $quar_type eq 'Q') {
do_log(5, "preprocess_policy_query: opening in sql: %s",
$msginfo->mail_id);
my($obj) = $Amavis::sql_storage;
@@ -11617,7 +13034,7 @@ sub preprocess_policy_query($) {
do_log(5, "preprocess_policy_query: opening mail '%s'", $fname);
# set new amavis message id
new_am_id( ($fname =~ m{amavis-(milter-)?([^/ \t]+)}s ? $2 : undef) )
- if $attr_ref->{'request'} ne 'release';
+ if !$releasing;
# file created by amavis helper program or other client, just open it
my(@stat_list) = lstat($fname); my($errn) = @stat_list ? 0 : 0+$!;
if ($errn == ENOENT) { die "File $fname does not exist" }
@@ -11636,21 +13053,23 @@ sub preprocess_policy_query($) {
}
}
$msginfo->mail_text($fh); # save file handle to object
+ $msginfo->log_id(am_id());
}
if ($ampdp) {
- do_log(1, "%s %s %s: <%s> -> %s", $attr_ref->{'request'},
- $attr_ref->{'mail_id'}, $msginfo->mail_tempdir, $sender,
- join(',', qquote_rfc2821_local(@recips)) );
+ do_log(1, "%s %s %s: %s -> %s", $attr_ref->{'request'},
+ $attr_ref->{'mail_id'}, $msginfo->mail_tempdir,
+ $msginfo->sender_smtp,
+ join(',', map { $_->recip_addr_smtp } @recips) );
} else {
do_log(1, "Request: %s(%s): %s %s %s: %s[%s] <%s> -> <%s>",
@$attr_ref{qw(request protocol_state mail_id protocol_name
queue_id client_name client_address sender recipient)});
}
- $msginfo;
-}
-
-sub check_amcl_policy($$$$) {
- my($conn,$msginfo,$check_mail,$old_amcl) = @_;
+ ($msginfo, \@bank_names);
+}
+
+sub check_amcl_policy($$$$$) {
+ my($conn,$msginfo,$check_mail,$old_amcl,$bank_names_ref) = @_;
my($smtp_resp, $exit_code, $preserve_evidence);
my(%baseline_policy_bank); my($policy_changed) = 0;
%baseline_policy_bank = %current_policy_bank;
@@ -11658,8 +13077,27 @@ sub check_amcl_policy($$$$) {
if (!ref($msginfo->per_recip_data) || !defined($msginfo->mail_text)) {
$smtp_resp = '450 4.5.0 Incomplete request'; $exit_code = EX_TEMPFAIL;
} else {
- if ($msginfo->client_addr_mynets && defined($policy_bank{'MYNETS'}))
- { Amavis::load_policy_bank('MYNETS'); $policy_changed = 1 }
+ # loading a policy bank can affect subsequent c(), cr() and ca() results,
+ # so it is necessary to load each policy bank in the right order and soon
+ # after information becomes available; general principle is that policy
+ # banks are loaded in order in which information becomes available:
+ # interface/socket, client IP, SMTP session info, sender, ...
+ my($cl_ip) = $msginfo->client_addr;
+ # treat unknown client IP address as 0.0.0.0, from "This" Network, rfc1700
+ my($cl_ip_mynets) = lookup_ip_acl(
+ !defined($cl_ip) || $cl_ip eq '' ? '0.0.0.0' : $cl_ip,
+ @{ca('mynetworks_maps')});
+ $msginfo->client_addr_mynets($cl_ip_mynets);
+ if (($cl_ip_mynets?1:0) > (c('originating')?1:0)) {
+ $current_policy_bank{'originating'} = $cl_ip_mynets; $policy_changed = 1;
+ }
+ if ($cl_ip_mynets && defined($policy_bank{'MYNETS'})) {
+ Amavis::load_policy_bank('MYNETS'); $policy_changed = 1;
+ }
+ for my $bank_name (@$bank_names_ref) { # additional banks from the request
+ if (defined $policy_bank{$bank_name})
+ { Amavis::load_policy_bank($bank_name); $policy_changed = 1 }
+ }
my($sender) = $msginfo->sender;
if ($sender ne '' && defined $policy_bank{'MYUSERS'}
&& lookup(0,$sender,@{ca('local_domains_maps')})) {
@@ -11691,11 +13129,11 @@ sub check_amcl_policy($$$$) {
# amavisd -> amavis-helper protocol response consists of any number of
# the following lines, the response is terminated by an empty line
# version_server=2
- # delrcpt=recipient
- # addrcpt=recipient
+ # delrcpt=<recipient>
+ # addrcpt=<recipient>
# delheader=hdridx hdr_head
# chgheader=hdridx hdr_head hdr_body
- # insheader=hdridx hdr_head hdr_body (hdridx will always be 0)
+ # insheader=hdridx hdr_head hdr_body
# addheader=hdr_head hdr_body
# replacebody=new_body (not implemented)
# quarantine=reason (currently never used, supposed to call
@@ -11726,15 +13164,15 @@ sub check_amcl_policy($$$$) {
die "Not all recips done, but explicit forwarding"; # just in case
} else { # EX_OK
for my $r (@{$msginfo->per_recip_data}) { # modified recipient addresses?
- my($addr,$newaddr) = ($r->recip_addr, $r->recip_final_addr);
- if ($r->recip_done) { # delete
- push(@response, proto_encode('delrcpt',
- quote_rfc2821_local($addr)));
- } elsif ($newaddr ne $addr) { # modify, e.g. adding extension
- push(@response, proto_encode('delrcpt',
- quote_rfc2821_local($addr)));
+ my($newaddr) = $r->recip_final_addr;
+ if ($r->recip_done) { # delete
+ push(@response, proto_encode('delrcpt', $r->recip_addr_smtp))
+ if defined $r->recip_addr; # if in the original list, not always_bcc
+ } elsif ($newaddr ne $r->recip_addr) { # modify, e.g. adding extension
+ push(@response, proto_encode('delrcpt', $r->recip_addr_smtp))
+ if defined $r->recip_addr; # if in the original list, not always_bcc
push(@response, proto_encode('addrcpt',
- quote_rfc2821_local($newaddr)));
+ qquote_rfc2821_local($newaddr)));
}
}
my($hdr_edits) = $msginfo->header_edits;
@@ -11743,7 +13181,7 @@ sub check_amcl_policy($$$$) {
while ( ($field_name,$edit) = each %{$hdr_edits->{edit}} ) {
$field_body = $msginfo->orig_header_fields->{lc($field_name)}; # quick
$field_body = $msginfo->mime_entity->head->get($field_name,0) # slower
- if !defined($field_body);
+ if !defined($field_body) && defined($msginfo->mime_entity);
if (!defined($field_body)) {
# such header field does not exist, do nothing
} else { # edit the first occurrence
@@ -11761,17 +13199,21 @@ sub check_amcl_policy($$$$) {
}
}
}
+ my($append_to_bottom) = c('append_header_fields_to_bottom');
+ my($hdridx) = c('prepend_header_fields_hdridx'); # milter insertion index
+ $hdridx = 0 if !defined($hdridx) || $hdridx < 0;
+ $hdridx = sprintf("%d",$hdridx); # convert to string
# prepend header fields one at a time, topmost field last
for my $hf (map {ref $hdr_edits->{$_} ? reverse @{$hdr_edits->{$_}} : ()}
- (c('append_header_fields_to_bottom') ?
- qw(prepend addrcvd) : qw(prepend add addrcvd)) ) {
+ ($append_to_bottom ? qw(prepend addrcvd)
+ : qw(prepend add addrcvd)) ) {
if ($hf =~ /^([^:]+):[ \t]*(.*?)$/s)
- { push(@response, proto_encode('insheader','0',$1,$2)) }
+ { push(@response, proto_encode('insheader',$hdridx,$1,$2)) }
}
# append header fields
- for my $hf (map { ref $hdr_edits->{$_} ? @{$hdr_edits->{$_}} : () }
- (c('append_header_fields_to_bottom') ?
- qw(add append) : qw(append)) ) {
+ for my $hf (map {ref $hdr_edits->{$_} ? @{$hdr_edits->{$_}} : ()}
+ ($append_to_bottom ? qw(add append)
+ : qw(append)) ) {
if ($hf =~ /^([^:]+):[ \t]*(.*?)$/s)
{ push(@response, proto_encode('addheader',$1,$2)) }
}
@@ -11790,7 +13232,7 @@ sub check_amcl_policy($$$$) {
}
push(@response, proto_encode('exit_code',sprintf("%d",$exit_code)));
ll(2) && do_log(2, "mail checking ended: %s", join("\n", at response));
- if ($policy_changed) {
+ if ($policy_changed) { # restore bank settings
%current_policy_bank = %baseline_policy_bank; $policy_changed = 0;
}
@response;
@@ -11822,14 +13264,18 @@ sub proto_encode($@) {
$attribute_name . '=' . join(' ', at strings);
}
-sub dispatch_from_quarantine($$) {
- my($conn,$msginfo) = @_;
+sub dispatch_from_quarantine($$$) {
+ my($conn,$msginfo,$request_type) = @_;
+ my($err);
eval {
- msg_from_quarantine($conn,$msginfo); # fill message object information
+ msg_from_quarantine($conn,$msginfo,$request_type); # fill msg object info
mail_dispatch($conn,$msginfo,0,1); # re-send the mail
+ 1;
+ } or do {
+ $err = $@ ne '' ? $@ : "errno=$!"; chomp $err;
+ do_log(0, "WARN: dispatch_from_quarantine failed: %s",$err);
+ die $err if $err =~ /^timed out\b/; # resignal timeout
};
- my($err) = $@; chomp($err);
- if ($@ ne '') { do_log(0, "WARN: dispatch_from_quarantine failed: %s",$err) }
my(@response);
for my $r (@{$msginfo->per_recip_data}) {
local($1,$2,$3); my($smtp_s,$smtp_es,$msg);
@@ -11858,7 +13304,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
use Errno qw(ENOENT EACCES);
@@ -11867,7 +13313,7 @@ BEGIN {
BEGIN {
import Amavis::Conf qw(:platform :confvars c cr ca);
import Amavis::Util qw(ll do_log untaint am_id new_am_id snmp_counters_init
- xtext_encode xtext_decode debug_oneshot
+ orcpt_encode xtext_decode debug_oneshot
prolong_timer waiting_for_client
switch_to_my_time switch_to_client_time
sanitize_str rmdir_recursively add_entropy
@@ -11895,17 +13341,20 @@ sub new($) {
sub DESTROY {
my($self) = shift;
- eval { do_log(5,"Amavis::In::SMTP DESTROY called, sock=%s, normal=%s",
- $self->{sock}, $self->{session_closed_normally}) };
eval {
- if (ref($self->{sock}) && ! $self->{session_closed_normally}) {
+ if (defined($my_pid) && $$ != $my_pid) {
+ do_log(5,"Skip closing SMTP session in a clone [%s] (born as [%s])",
+ $$,$my_pid);
+ } elsif (ref($self->{sock}) && ! $self->{session_closed_normally}) {
my($msg) = "421 4.3.2 Service shutting down, closing channel";
$msg .= ", during waiting for input from client" if waiting_for_client();
$self->smtp_resp(1,$msg);
}
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!";
+ eval { do_log(1,"SMTP shutdown: %s", $eval_stat) };
};
- if ($@ ne '')
- { my($eval_stat) = $@; eval { do_log(1,"SMTP shutdown: %s",$eval_stat) } }
}
sub preserve_evidence { # preserve temporary files etc in case of trouble
@@ -11954,7 +13403,7 @@ sub process_smtp_request($$$$) {
$myheloname = '[' . $conn->socket_ip . ']';
new_am_id(undef, $Amavis::child_invocation_count, undef);
- my($initial_am_id) = 1; my($sender, at recips); my($got_rcpt);
+ my($initial_am_id) = 1; my($sender_unq,$sender_quo, at recips); my($got_rcpt);
my($max_recip_size_limit); # maximum of per-recipient message size limits
my($terminating,$aborting,$eof,$voluntary_exit); my($seq) = 0;
my(%xforward_args); my(%baseline_policy_bank); my($policy_changed);
@@ -11962,8 +13411,8 @@ sub process_smtp_request($$$$) {
$conn->smtp_proto($self->{proto} = $lmtp ? 'LMTP' : 'SMTP');
# system-wide message size limit, if any
- my($final_oversized_destiny) = scalar(setting_by_given_contents_category(
- cr('final_destiny_by_ccat'), CC_OVERSIZED));
+ my($final_oversized_destiny) = setting_by_given_contents_category(
+ CC_OVERSIZED, cr('final_destiny_by_ccat'));
my($message_size_limit) = c('smtpd_message_size_limit');
if ($enforce_smtpd_message_size_limit_64kb_min &&
$message_size_limit && $message_size_limit < 65536)
@@ -11981,6 +13430,7 @@ sub process_smtp_request($$$$) {
'protocol' => $lmtp?'LMTP':'ESMTP' }->{lc($1.$2)}
}egx;
$self->smtp_resp(1, "220 $smtpd_greeting_banner_tmp");
+ section_time('SMTP greeting');
# each call to smtp_resp starts a $smtpd_timeout timeout to tame slow clients
$0 = sprintf("amavisd (ch%d-idle)", $Amavis::child_invocation_count);
@@ -12003,28 +13453,28 @@ sub process_smtp_request($$$$) {
# (causes holdups in Postfix, it doesn't retry immediately; better set max_use)
# $Amavis::child_task_count >= $max_requests # exceeded max_requests
-# && /^(?:HELO|EHLO|LHLO|DATA|NOOP)\z/ && do { # pipelining checkpoints
+# && /^(?:HELO|EHLO|LHLO|DATA|NOOP|QUIT|VRFY|EXPN|TURN)\z/ && do {
+# # pipelining checkpoints;
# # in case of multiple-transaction protocols (e.g. SMTP, LMTP)
# # we do not like to keep running indefinitely at the MTA's mercy
# my($msg) = "Closing transmission channel ".
# "after $Amavis::child_task_count transactions, $_";
-# do_log(2,"%s",$msg); $self->smtp_resp(1,"421 4.3.0 ".$msg);
+# do_log(2,"%s",$msg); $self->smtp_resp(1,"421 4.3.0 ".$msg); #flush!
# $terminating=1; last;
# };
/^(?:RSET|DATA|QUIT)\z/ && $args ne '' && do {
$self->smtp_resp(1,"501 5.5.4 Error: $_ does not accept arguments",
- 1,$cmd);
+ 1,$cmd); #flush for DATA or QUIT (no need for RSET)
last;
};
- /^RSET\z/ && do { $sender = undef; @recips = (); $got_rcpt = 0;
- $max_recip_size_limit = undef; $msginfo = undef;
- if ($policy_changed) {
- %current_policy_bank = %baseline_policy_bank;
- $policy_changed = 0;
- }
- $self->smtp_resp(0,"250 2.0.0 Ok $_"); last;
- };
- /^NOOP\z/ && do { $self->smtp_resp(1,"250 2.0.0 Ok $_"); last };
+ /^RSET\z/ && do {
+ undef $sender_unq; undef $sender_quo; @recips = (); $got_rcpt = 0;
+ $max_recip_size_limit = undef; $msginfo = undef;
+ if ($policy_changed)
+ { %current_policy_bank = %baseline_policy_bank; $policy_changed = 0 }
+ $self->smtp_resp(0,"250 2.0.0 Ok $_"); last;
+ };
+ /^NOOP\z/ && do { $self->smtp_resp(1,"250 2.0.0 Ok $_"); last }; #flush!
/^QUIT\z/ && do {
my($smtpd_quit_banner_tmp) = c('smtpd_quit_banner');
$smtpd_quit_banner_tmp =~
@@ -12038,12 +13488,12 @@ sub process_smtp_request($$$$) {
'product' => $myproduct_name,
'protocol' => $lmtp?'LMTP':'ESMTP' }->{lc($1.$2)}
}egx;
- $self->smtp_resp(1,"221 2.0.0 $smtpd_quit_banner_tmp");
+ $self->smtp_resp(1,"221 2.0.0 $smtpd_quit_banner_tmp"); #flush!
$terminating=1; last;
};
### !$lmtp && /^HELO\z/ && do { # strict
/^HELO\z/ && do {
- $sender = undef; @recips = (); $got_rcpt = 0; # implies RSET
+ undef $sender_unq; undef $sender_quo; @recips = (); $got_rcpt = 0;
$max_recip_size_limit = undef; $msginfo = undef; # forget previous
if ($policy_changed)
{ %current_policy_bank = %baseline_policy_bank; $policy_changed = 0 }
@@ -12053,7 +13503,7 @@ sub process_smtp_request($$$$) {
};
### (!$lmtp && /^EHLO\z/ || $lmtp && /^LHLO\z/) && do { # strict
/^(?:EHLO|LHLO)\z/ && do {
- $sender = undef; @recips = (); $got_rcpt = 0; # implies RSET
+ undef $sender_unq; undef $sender_quo; @recips = (); $got_rcpt = 0;
$max_recip_size_limit = undef; $msginfo = undef; # forget previous
if ($policy_changed)
{ %current_policy_bank = %baseline_policy_bank; $policy_changed = 0 }
@@ -12076,12 +13526,13 @@ sub process_smtp_request($$$$) {
@ehlo_keywords =
grep { /^([A-Za-z0-9]+)/ &&
!$smtpd_discard_ehlo_keywords{uc($1)} } @ehlo_keywords;
- $self->smtp_resp(0,"250 $myheloname\n" . join("\n", at ehlo_keywords));
+ $self->smtp_resp(1,"250 $myheloname\n" .
+ join("\n", at ehlo_keywords)); #flush!
$conn->smtp_helo($args); section_time("SMTP $_");
last;
};
/^XFORWARD\z/ && do { # Postfix extension
- if (defined($sender)) {
+ if (defined($sender_unq)) {
$self->smtp_resp(0,"503 5.5.1 Error: XFORWARD not allowed ".
"within transaction",1,$cmd);
last;
@@ -12126,7 +13577,7 @@ sub process_smtp_request($$$$) {
if ($authenticated) {
$self->smtp_resp(0,"503 5.5.1 Error: session already authenticated",
1,$cmd);
- } elsif (defined($sender)) {
+ } elsif (defined($sender_unq)) {
$self->smtp_resp(0,"503 5.5.1 Error: AUTH not allowed within ".
"transaction",1,$cmd);
} elsif (!grep {uc($_) eq $auth_mech} @{ca('auth_mech_avail')}) {
@@ -12174,14 +13625,14 @@ sub process_smtp_request($$$$) {
};
/^VRFY\z/ && do {
if ($args eq '') {
- $self->smtp_resp(1,"501 5.5.2 Syntax: VRFY address", 1, $cmd);
+ $self->smtp_resp(1,"501 5.5.2 Syntax: VRFY address", 1,$cmd); #flush!
} else { # rfc2505
- $self->smtp_resp(1,"252 2.0.0 Argument not checked", 0, $cmd);
+ $self->smtp_resp(1,"252 2.0.0 Argument not checked", 0,$cmd); #flush!
}
last;
};
/^MAIL\z/ && do { # begin new SMTP transaction
- if (defined($sender)) {
+ if (defined($sender_unq)) {
$self->smtp_resp(0,"503 5.5.1 Error: nested MAIL command", 1, $cmd);
last;
}
@@ -12198,20 +13649,32 @@ sub process_smtp_request($$$$) {
Amavis::Timing::init(); snmp_counters_init();
}
$seq++;
- new_am_id(undef,$Amavis::child_invocation_count,$seq)
- if !$initial_am_id;
+ if (!$initial_am_id) {
+ new_am_id(undef,$Amavis::child_invocation_count,$seq);
+ # enter 'in transaction' state
+ $Amavis::snmp_db->register_proc(
+ 1,1,'m',am_id()) if defined $Amavis::snmp_db;
+ }
$initial_am_id = 0;
Amavis::check_mail_begin_task();
$self->{tempdir}->prepare;
$self->{tempdir}->prepare_file;
- my($cl_ip) = $xforward_args{'ADDR'};
- my($cl_ip_mynets) = $cl_ip eq '' ? undef :
- lookup_ip_acl($cl_ip,@{ca('mynetworks_maps')});
- if ($cl_ip_mynets && defined($policy_bank{'MYNETS'}))
- { Amavis::load_policy_bank('MYNETS'); $policy_changed = 1 }
$msginfo = Amavis::In::Message->new;
$msginfo->rx_time($now);
- # $msginfo->body_type('7bit'); # presumed, unless explicitly declared
+ $msginfo->log_id(am_id());
+ my($cl_ip) = $xforward_args{'ADDR'};
+ # treat unknown client IP addr as 0.0.0.0, from "This" Network, rfc1700
+ my($cl_ip_mynets) = lookup_ip_acl(
+ !defined($cl_ip) || $cl_ip eq '' ? '0.0.0.0' : $cl_ip,
+ @{ca('mynetworks_maps')});
+ if (($cl_ip_mynets?1:0) > (c('originating')?1:0)) {
+ $current_policy_bank{'originating'} = $cl_ip_mynets;
+ $policy_changed = 1;
+ }
+ if ($cl_ip_mynets && defined($policy_bank{'MYNETS'})) {
+ Amavis::load_policy_bank('MYNETS'); $policy_changed = 1;
+ }
+ # $msginfo->body_type('7BIT'); # presumed, unless explicitly declared
$msginfo->delivery_method(c('forward_method'));
my($submitter);
if ($authenticated) {
@@ -12258,13 +13721,21 @@ sub process_smtp_request($$$$) {
} elsif ($name eq 'ENVID') { # rfc3461, value encoded as xtext
if (!defined($dsn_envid)) { $dsn_envid = $val }
else { $msg = "501 5.5.4 Syntax error in MAIL parameter: $name" }
- } elsif ($name eq 'AUTH' && @{ca('auth_mech_avail')} &&
- !defined($submitter) ) { # rfc2554
- $submitter = xtext_decode($val); # encoded as xtext: rfc3461
- do_log(5, "MAIL command, %s, submitter: %s",
- $authenticated,$submitter);
- } elsif ($name eq 'AUTH' && !@{ca('auth_mech_avail')}) {
- $msg = "503 5.7.4 Error: authentication disabled";
+ } elsif ($name eq 'AUTH') { # rfc2554
+ my($s) = xtext_decode($val); # encoded as xtext: rfc3461
+ do_log(5, "MAIL command, %s, submitter: %s", $authenticated,$s);
+ if (defined $submitter) {
+ $msg = "504 5.5.4 MAIL command duplicate param.: $name=$val";
+ } elsif (!@{ca('auth_mech_avail')}) {
+ do_log(3, "MAIL command parameter AUTH supplied, ".
+ "but authentication is disabled, ignored");
+ $submitter = '<>';
+ # mercifully ignore invalid parameter for the benefit of
+ # running amavisd as a Postix pre-queue smtp proxy filter
+ # $msg = "503 5.7.4 Error: authentication disabled";
+ } else {
+ $submitter = $s;
+ }
} else {
$msg = "504 5.5.4 MAIL command parameter error: $name=$val";
}
@@ -12283,17 +13754,18 @@ sub process_smtp_request($$$$) {
do_log(0, "%s REJECT 'MAIL FROM': %s", $self->{proto},$msg);
}
if (!defined($msg)) {
+ $sender_quo = $addr; $sender_unq = unquote_rfc2821_local($addr);
$addr = $1 if $addr =~ /^<(.*)>\z/s;
- $sender = unquote_rfc2821_local($addr);
- my($requoted) = qquote_rfc2821_local($sender);
- do_log(0, "WARN: address modified (sender): <%s> -> %s",
- $addr, $requoted) if $requoted ne "<$addr>";
- if ($sender ne '' && defined $policy_bank{'MYUSERS'}
- && lookup(0,$sender,@{ca('local_domains_maps')})) {
+ my($requoted) = qquote_rfc2821_local($sender_unq);
+ do_log(0, "WARN: address modified (sender): %s -> %s",
+ $sender_quo, $requoted) if $requoted ne $sender_quo;
+ if ($sender_unq ne '' && defined $policy_bank{'MYUSERS'}
+ && lookup(0,$sender_unq,@{ca('local_domains_maps')})) {
Amavis::load_policy_bank('MYUSERS'); $policy_changed = 1;
}
- debug_oneshot(lookup(0,$sender,@{ca('debug_sender_maps')}) ? 1 : 0,
- $self->{proto} . "< $cmd");
+ debug_oneshot(
+ lookup(0,$sender_unq, @{ca('debug_sender_maps')}) ? 1 : 0,
+ $self->{proto} . "< $cmd");
# $submitter = $addr if !defined($submitter); # rfc2554: MAY
$submitter = '<>' if !defined($msginfo->auth_user);
$msginfo->auth_submitter($submitter);
@@ -12303,13 +13775,13 @@ sub process_smtp_request($$$$) {
$msginfo->dsn_ret($dsn_ret) if defined $dsn_ret;
$msginfo->dsn_envid($dsn_envid) if defined $dsn_envid;
}
- $msg = "250 2.1.0 Sender $addr OK";
+ $msg = "250 2.1.0 Sender $sender_quo OK";
};
$self->smtp_resp(0,$msg, !$msg_nopenalize && $msg=~/^5/ ? 1 : 0, $cmd);
last;
};
/^RCPT\z/ && do {
- if (!defined($sender)) {
+ if (!defined($sender_unq)) {
$self->smtp_resp(0,"503 5.5.1 Need MAIL command before RCPT",1,$cmd);
@recips = (); $got_rcpt = 0;
last;
@@ -12347,18 +13819,17 @@ sub process_smtp_request($$$$) {
}
last if defined $msg;
}
- $addr = $1 if $addr =~ /^<(.*)>\z/s;
my($addr_unq) = unquote_rfc2821_local($addr);
my($requoted) = qquote_rfc2821_local($addr_unq);
- if ($requoted ne "<$addr>") { # check for valid canonical quoting
- do_log(0, "WARN: address modified (recip): <%s> -> %s",
+ if ($requoted ne $addr) { # check for valid canonical quoting
+ do_log(0, "WARN: address modified (recip): %s -> %s",
$addr, $requoted);
# rfc3461: If no ORCPT parameter was present in the RCPT command when
# the message was received, an ORCPT parameter MAY be added to the
# RCPT command when the message is relayed. If an ORCPT parameter is
# added by the relaying MTA, it MUST contain the recipient address
#from the RCPT command used when the message was received by that MTA
- $orcpt = 'rfc822;'.xtext_encode($addr) if !defined($orcpt);
+ $orcpt = orcpt_encode($addr) if !defined($orcpt);
}
my($recip_size_limit); my($mslm) = ca('message_size_limit_maps');
$recip_size_limit = lookup(0,$addr_unq, @$mslm) if @$mslm;
@@ -12386,7 +13857,7 @@ sub process_smtp_request($$$$) {
$recip_size_limit && $mail_size > $recip_size_limit &&
$final_oversized_destiny == D_REJECT) {
$msg = "552 5.3.4 Declared message size ($mail_size B) ".
- "exceeds recipient's size limit <$addr>";
+ "exceeds size limit for recipient $addr";
$msg_nopenalize = 1;
do_log(0, "%s REJECT 'RCPT TO': %s", $self->{proto},$msg);
}
@@ -12396,6 +13867,7 @@ sub process_smtp_request($$$$) {
if (!defined($msg)) {
my($recip_obj) = Amavis::In::Message::PerRecip->new;
$recip_obj->recip_addr($addr_unq);
+ $recip_obj->recip_addr_smtp($addr);
$recip_obj->recip_destiny(D_PASS); # default is Pass
$recip_obj->dsn_notify($notify) if defined $notify;
$recip_obj->dsn_orcpt($orcpt) if defined $orcpt;
@@ -12406,19 +13878,24 @@ sub process_smtp_request($$$$) {
last;
};
/^DATA\z/ && !@recips && do {
- if (!defined($sender)) {
+ if (!defined($sender_unq)) {
$self->smtp_resp(1,"503 5.5.1 Need MAIL command before DATA",1,$cmd);
} elsif (!$got_rcpt) {
$self->smtp_resp(1,"503 5.5.1 Need RCPT command before DATA",1,$cmd);
} elsif ($lmtp) { # rfc2033 requires 503 code!
$self->smtp_resp(1,"503 5.1.1 Error (DATA): no valid recipients",
- 0,$cmd);
+ 0,$cmd); #flush!
} else {
$self->smtp_resp(1,"554 5.1.1 Error (DATA): no valid recipients",
- 0,$cmd);
+ 0,$cmd); #flush!
}
last;
};
+# /^DATA\z/ && uc($msginfo->body_type) eq "BINARYMIME" && do { # rfc3030
+# $self->smtp_resp(1,"503 5.5.1 DATA is incompatible with BINARYMIME",
+# 0,$cmd); #flush!
+# last;
+# };
/^DATA\z/ && do {
# set timer to the initial value, MTA timer starts here
if ($message_size_limit) { # enforce system-wide size limit
@@ -12430,12 +13907,14 @@ sub process_smtp_request($$$$) {
my($within_data_transfer,$complete);
my($size) = 0; my($over_size) = 0;
eval {
- $msginfo->sender($sender); $msginfo->per_recip_data(\@recips);
- ll(1) && do_log(1, "%s:%s:%s %s: <%s> -> %s%s Received: %s",
+ $msginfo->sender($sender_unq); $msginfo->sender_smtp($sender_quo);
+ $msginfo->per_recip_data(\@recips);
+ ll(1) && do_log(1, "%s:%s:%s %s: %s -> %s%s Received: %s",
$conn->smtp_proto,
$conn->socket_ip eq $inet_socket_bind?'':'['.$conn->socket_ip.']',
$conn->socket_port, $self->{tempdir}->path,
- $sender, join(',',qquote_rfc2821_local(@{$msginfo->recips})),
+ $sender_quo,
+ join(',', map { $_->recip_addr_smtp } @{$msginfo->per_recip_data}),
join('',
!defined $msginfo->msg_size ? () : ' SIZE='.$msginfo->msg_size,
!defined $msginfo->body_type ? () : ' BODY='.$msginfo->body_type,
@@ -12447,9 +13926,11 @@ sub process_smtp_request($$$$) {
' ENVID='.xtext_decode($msginfo->dsn_envid),
),
received_line($conn,$msginfo,undef,0) );
- $self->smtp_resp(1,"354 End data with <CR><LF>.<CR><LF>");
+ $self->smtp_resp(1,"354 End data with <CR><LF>.<CR><LF>"); #flush!
$within_data_transfer = 1;
section_time('SMTP pre-DATA-flush') if $self->{pipelining};
+ $Amavis::snmp_db->register_proc( # data transfering state
+ 2,0,'d',am_id()) if defined $Amavis::snmp_db;
$self->{tempdir}->empty(0);
switch_to_client_time('receiving data');
my($fh) = $self->{tempdir}->fh;
@@ -12506,10 +13987,13 @@ sub process_smtp_request($$$$) {
# this may have the effect of calling stdio's clearerr(3).
$fh->seek(0,1) or die "Can't seek on file: $!";
section_time('SMTP DATA');
- }; # end eval
+ 1;
+ } or do { # end eval
+ $@ = "errno=$!" if $@ eq '';
+ };
if ($@ ne '' || !$complete || # err or connection broken
($over_size && $final_oversized_destiny == D_REJECT)) {
- chomp($@);
+ chomp $@;
# on error, either send: '421 Shutting down',
# or: '451 Aborted, error in processing' and NOT shut down!
if ($over_size && $@ eq '' && !$within_data_transfer) {
@@ -12527,7 +14011,7 @@ sub process_smtp_request($$$$) {
$aborting .= ', ' if $aborting ne '' && $@ ne '';
$aborting .= $@;
$aborting .= " during waiting for input from client"
- if $@ eq 'timed out' && waiting_for_client();
+ if $@ =~ /^timed out\b/ && waiting_for_client();
$aborting = '???' if $aborting eq '';
do_log($@ ne '' ? -1 : 3,
"%s ABORTING: %s", $self->{proto},$aborting);
@@ -12538,7 +14022,7 @@ sub process_smtp_request($$$$) {
# mechanism and can not accept responsibility for delivery.
#
# check contents before responding
- # check_mail() expects open file on $self->{tempdir}->fh,
+ # check_mail() expects an open file handle in $msginfo->mail_text,
# need not be rewound
$msginfo->mail_tempdir($self->{tempdir}->path);
$msginfo->mail_text_fn($self->{tempdir}->path . '/email.txt');
@@ -12552,21 +14036,29 @@ sub process_smtp_request($$$$) {
do_log(4,"Actual message size %d B, declared %d B",
$size,$declared_size);
}
- # silly Perl, length(tainted) gives tainted result
$msginfo->msg_size(untaint($size)); # store actual mail size
+
+ # some fatal errors are not catchable by eval (like exceeding virtual
+ # memory), but may still allow processing to continue in a DESTROY or
+ # END method; turn on trouble flag here to allow DESTROY to deal with
+ # such a case correctly, then clear the flag after content checking
+ # if everything turned out well
+ $self->{tempdir}->preserve(1);
my($smtp_resp, $exit_code, $preserve_evidence) =
&$check_mail($conn,$msginfo,$lmtp); # do all the contents checking
+ $self->{tempdir}->preserve(0) if !$preserve_evidence; # clear if ok
prolong_timer('check done');
- if ($preserve_evidence) { $self->{tempdir}->preserve(1) }
+
if ($smtp_resp !~ /^4/ &&
grep { !$_->recip_done } @{$msginfo->per_recip_data}) {
if ($msginfo->delivery_method eq '') {
do_log(1,"NOT ALL RECIPIENTS DONE, FORWARD_METHOD IS EMPTY!");
} else {
- die "TROUBLE: (MISCONFIG) not all recipients done, " .
+ die "TROUBLE: (MISCONFIG?) not all recipients done, " .
"forward_method is: " . $msginfo->delivery_method;
}
}
+ section_time('SMTP pre-response');
if (!$lmtp) { # smtp
do_log(4, 'sending SMTP response: "%s"', $smtp_resp);
$self->smtp_resp(0, $smtp_resp);
@@ -12574,7 +14066,7 @@ sub process_smtp_request($$$$) {
my($bounced) = $msginfo->dsn_sent; # 1=bounced, 2=suppressed
for my $r (@{$msginfo->per_recip_data}) {
my($resp) = $r->recip_smtp_response;
- my($recip_quoted) = qquote_rfc2821_local($r->recip_addr);
+ my($recip_quoted) = $r->recip_addr_smtp;
if ($resp=~/^2/) {
# success, no need to change status
} elsif ($bounced == 1) { # genuine bounce
@@ -12590,14 +14082,16 @@ sub process_smtp_request($$$$) {
$resp = sprintf("250 2.5.0 Ok %s, DSN suppressed_2 (%s)",
$recip_quoted, $resp);
}
- do_log(4, 'sending LMTP response for %s: "%s"',
- $recip_quoted, $resp);
+ do_log(4, 'LMTP response for %s: "%s"', $recip_quoted, $resp);
$self->smtp_resp(0, $resp);
}
}
+ $self->smtp_resp_flush; # optional, but nice to report timing right
+ section_time('SMTP response');
}; # end all OK
$self->{tempdir}->clean;
- $sender = undef; @recips = (); $got_rcpt = 0; # implicit RSET
+ # implicit RSET
+ undef $sender_unq; undef $sender_quo; @recips = (); $got_rcpt = 0;
$max_recip_size_limit = undef; $msginfo = undef; # forget previous
if ($policy_changed)
{ %current_policy_bank = %baseline_policy_bank; $policy_changed = 0 }
@@ -12607,7 +14101,7 @@ sub process_smtp_request($$$$) {
Amavis::Timing::init(); snmp_counters_init();
last;
}; # DATA
- # catchall (EXPN, TURN, unknown):
+ # catchall (EXPN, TURN, unknown): #flush!
$self->smtp_resp(1,"502 5.5.1 Error: command $_ not implemented",1,$cmd);
# $self->smtp_resp(1,"500 5.5.2 Error: command $_ not recognized", 1,$cmd);
}; # end of 'switch' block
@@ -12616,7 +14110,7 @@ sub process_smtp_request($$$$) {
}
# rfc2920 requires a flush whenever the local TCP input buffer is
# emptied. Since we can't check it (unless we use sysread & select),
- # we should do a flush here to be in compliance.
+ # we should do a flush here to be in compliance. To be improved some day.
$self->smtp_resp_flush;
$0 = sprintf("amavisd (ch%d-%s-idle)",
$Amavis::child_invocation_count, am_id());
@@ -12632,8 +14126,12 @@ sub process_smtp_request($$$$) {
alarm(0); do_log(4,"SMTP session over, timer stopped");
Amavis::Timing::go_busy(7);
# flush just in case, session might have been disconnected
- eval { $self->smtp_resp_flush };
- do_log(1, "flush failed: %s", $@) if $@ ne '';
+ eval {
+ $self->smtp_resp_flush; 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ do_log(1, "flush failed: %s", $eval_stat);
+ };
my($msg) =
defined $aborting && !$eof ? "ABORTING the session: $aborting" :
defined $aborting ? $aborting :
@@ -12691,6 +14189,297 @@ 1;
__DATA__
#
+package Amavis::Out::SMTP::Protocol;
+use strict;
+use re 'taint';
+no warnings 'uninitialized';
+use warnings FATAL => 'utf8';
+
+BEGIN {
+ use Exporter ();
+ use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
+ $VERSION = '2.091';
+ @ISA = qw(Exporter);
+}
+
+use Errno qw(EAGAIN EINTR ECONNRESET);
+use IO::Socket;
+use IO::Socket::UNIX;
+use IO::Socket::INET;
+#use IO::Socket::INET6;
+
+BEGIN {
+ import Amavis::Conf qw(:platform);
+ import Amavis::Util qw(ll do_log min max);
+ use vars qw($have_inet6);
+ $have_inet6 = eval { require IO::Socket::INET6 };
+}
+
+sub init {
+ my($self)=shift;
+ delete $self->{domain}; delete $self->{supports};
+ $self->{pipelining} = 0;
+}
+
+sub new {
+ my($class,$peeraddress,%arg) = @_;
+ my($peerport) = $arg{Port};
+ my($localaddr,$localport) = ($arg{LocalAddr},$arg{LocalPort});
+ my($self) = bless {}, $class;
+ $self->init; $self->timeout($arg{Timeout}); $self->{last_is_nl} = 0;
+ my($socketname) = $peeraddress;
+ my($is_inet) = $socketname=~m{^/} ? 0 : 1; # simpleminded: unix vs. inet
+ my($is_inet4) = $is_inet && $socketname=~/^\d+.\d+.\d+.\d+\z/ ? 1 : 0;
+ my($is_inet6) = $is_inet && $socketname=~/:/ ? 1 : 0;
+ my($sock);
+ if ($is_inet4 || $is_inet && !$have_inet6) { # inet socket (IPv4)
+ do_log(3,"smtp creating socket by IO::Socket::INET: %s", $socketname);
+ $sock = IO::Socket::INET->new(
+ Proto => 'tcp', Blocking => 0,
+ PeerAddr => $peeraddress, PeerPort => $peerport,
+ LocalAddr => $localaddr, LocalPort => $localport,
+ ) or die "Can't connect to INET4 socket $socketname: $!\n";
+ } elsif ($is_inet6 || $is_inet) { # inet6 socket (IPv6) or unknown IP
+ do_log(3,"smtp creating socket by IO::Socket::INET6: %s", $socketname);
+ $sock = IO::Socket::INET6->new(
+ Proto => 'tcp', Blocking => 0,
+ PeerAddr => $peeraddress, PeerPort => $peerport,
+ LocalAddr => $localaddr, LocalPort => $localport,
+ ) or die "Can't connect to INET6 socket $socketname: $!\n";
+ } else { # unix socket
+ do_log(3,"smtp creating socket by IO::Socket::UNIX: %s", $socketname);
+ $sock = IO::Socket::UNIX->new(Type => SOCK_STREAM)
+ or die "Can't create UNIX socket: $!\n";
+ $sock->connect( pack_sockaddr_un($socketname) )
+ or die "Can't connect to UNIX socket $socketname: $!\n";
+ }
+ $self->{inp} = ''; $self->{out} = ''; $self->{inpeof} = 0;
+ $self->{socket} = $sock;
+ $self;
+}
+
+sub close {
+ my($self) = @_;
+ my($sock) = $self->{socket};
+ if (defined($sock) && defined(fileno($sock)))
+ { $sock->close or die "Error closing socket: $!" }
+ 1;
+}
+
+sub DESTROY { my($self) = @_; eval { $self->close } }
+
+sub ehlo_response_parse {
+ my($self,$smtp_resp) = @_;
+ delete $self->{domain}; delete $self->{supports};
+ my(@ehlo_lines) = split(/\n/,$smtp_resp,-1);
+ my($bad); my($first) = 1; local($1,$2);
+ for my $el (@ehlo_lines) {
+ if ($first) {
+ if ($el =~ /^(\d{3})(?:[ \t]+(.*))?\z/) { $self->{domain} = $2 }
+ elsif (!defined($bad)) { $bad = $el }
+ $first = 0;
+ } elsif ($el =~ /^([A-Z0-9][A-Z0-9-]*)(?:[ =](.*))?\z/i) {
+ $self->{supports}{uc($1)} = defined($2) ? $2 : '';
+ } elsif (!defined($bad)) { $bad = $el }
+ }
+ $self->{pipelining} = defined $self->{supports}{'PIPELINING'} ? 1 : 0;
+ do_log(-1,"Bad EHLO kw %s ignored in %s",$bad,$smtp_resp) if defined $bad;
+ 1;
+}
+
+sub rw_loop {
+ my($self,$needline,$flushoutput) = @_;
+#
+# rfc2920: Client SMTP implementations MAY elect to operate in a nonblocking
+# fashion, processing server responses immediately upon receipt, even if
+# there is still data pending transmission from the client's previous TCP
+# send operation. If nonblocking operation is not supported, however, client
+# SMTP implementations MUST also check the TCP window size and make sure that
+# each group of commands fits entirely within the window. The window size
+# is usually, but not always, 4K octets. Failure to perform this check can
+# lead to deadlock conditions.
+#
+# We choose to operate in a nonblocking mode, but responses are only
+# read and stored for later, and not immediately processed as they come.
+# This may require some sanity limiting against rogue servers, but should
+# do for our typical deployment setup.
+#
+ my($sock) = $self->{socket};
+ my($fd_sock) = fileno($sock);
+ my($timeout) = $self->timeout;
+ my($idle_cnt) = 0; my($failed_write_attempts) = 0;
+ for (;;) {
+ $idle_cnt++;
+ my($rout,$wout,$eout); my($rin,$win,$ein); $rin=$win=$ein='';
+ my($want_to_write) = $self->{out} ne '' && ($flushoutput || $needline);
+ ll(5) && do_log(5,"rw_loop: needline=%d, flush=%s, wr=%d, timeout=%s",
+ $needline, $flushoutput, $want_to_write, $timeout);
+ if (!defined($fd_sock)) {
+ do_log(3,"rw_loop read: got closed socket");
+ $self->{inpeof} = 1; last;
+ }
+ vec($rin,$fd_sock,1) = 1;
+ vec($win,$fd_sock,1) = $want_to_write ? 1 : 0;
+ $ein = $rin | $win;
+ my($nfound,$timeleft) =
+ select($rout=$rin, $wout=$win, $eout=$ein, $timeout);
+ $nfound >= 0 or die "Select failed: $!";
+ if (vec($rout,$fd_sock,1)) {
+ do_log(5,"rw_loop: receiving");
+ my($inbuf) = ''; $! = 0;
+ # my($paddr) = recv($sock,$inbuf,32768,0);
+ my($nread) = sysread($sock,$inbuf,32768);
+ if (!defined($nread)) {
+ do_log(-1,"rw_loop read failed: %s", $!);
+ $!==0 || $!==EAGAIN || $!==EINTR
+ or die "Error receiving from socket: $!";
+ } elsif ($nread < 1) { # sysread returns 0 at eof
+ $self->{inpeof} = 1; do_log(3,"rw_loop read: got eof");
+ } else {
+ $self->{inpeof} = 0;
+ ll(5) && do_log(5,"rw_loop read %d chars< %s", length($inbuf),$inbuf);
+ $self->{inp} .= $inbuf; $idle_cnt = 0;
+ length $self->{inp} < 500000
+ or die "rw_loop: Aborting on a runaway server";
+ }
+ }
+ if (vec($wout,$fd_sock,1)) {
+ do_log(5,"rw_loop: sending");
+ # my($nwrite) = send($sock, $self->{out}, 0);
+ my($nwrite) = syswrite($sock, $self->{out});
+ defined($nwrite) or die "Can't write to socket: $!";
+ my($out_l) = length($self->{out});
+ my($ll) = $nwrite != $out_l ? 3 : 5;
+ if (ll($ll)) {
+ my($msg) = $nwrite==$out_l ? sprintf("%d", $nwrite)
+ : sprintf("%d (of %d)", $nwrite,$out_l);
+ my($nlog) = min(200,$nwrite);
+ do_log($ll,"rw_loop send %s> %s%s",
+ $msg, substr($self->{out},0,$nlog), $nlog<$nwrite?' [...]':'');
+ };
+ $idle_cnt = 0;
+ if ($nwrite <= 0) { $failed_write_attempts++ }
+ else { substr($self->{out},0,$nwrite) = '' }
+ }
+ last if (!$needline || index($self->{inp},"\015\012") >= 0) &&
+ (!$flushoutput || $self->{out} eq '');
+ last if $self->{inpeof};
+ last if $idle_cnt > 0; # probably exceeded timeout
+ $failed_write_attempts < 100 or die "rw_loop: Aborting stalled sending";
+ }
+}
+
+sub timeout
+ { my($self)=shift; !@_ ? $self->{timeout} : ($self->{timeout}=shift) }
+
+sub eof
+ { my($self) = @_; $self->{inpeof} && $self->{inp} eq '' ? 1 : 0 }
+
+sub domain
+ { my($self) = @_; $self->{domain} }
+
+sub supports
+ { my($self,$keyword) = @_; $self->{supports}{uc($keyword)} }
+
+sub command {
+ my($self,$command, at args) = @_;
+ my($line) = $command =~ /:\z/ ? $command.join(' ', at args)
+ : join(' ',$command, at args);
+ do_log(3,"smtp cmd> %s", $line);
+ $self->{out} .= $line . "\015\012"; $self->{last_is_nl} = 1;
+ # rfc2920: comands that can all appear anywhere in a pipelined command group
+ # RSET, MAIL FROM, SEND FROM, SOML FROM, SAML FROM, RCPT TO, (data)
+ $self->flush if !$self->{pipelining} || length($self->{out}) > 40000 ||
+ $command !~ /^(?:RSET|MAIL|SEND|SOML|SAML|RCPT)\b/is;
+ 1;
+}
+
+*print = \&datasend; # alias name for datasend
+sub datasend {
+ my($self) = shift; my($buff) = join('', at _);
+ do_log(-1,"WARN: Unicode string passed to datasend")
+ if $unicode_aware && Encode::is_utf8($buff);
+# do_log(5,"smtp datasend %d bytes>", length($buff));
+ substr($buff,0,0) = '.' if substr($buff,0,1) eq '.' && $self->{last_is_nl};
+ $buff =~ s{\n\.}{\n..}gs; # dot stuffing
+ $self->{last_is_nl} = substr($buff,-1,1) eq "\n";
+ $buff =~ s{\n}{\015\012}gs;
+ $self->{out} .= $buff;
+ $self->flush if length($self->{out}) > 40000;
+ 1;
+}
+
+sub flush
+ { my($self) = @_; $self->rw_loop(0,1) if $self->{out} ne ''; 1 }
+
+sub dataend {
+ my($self) = @_;
+ $self->{out} .= $self->{last_is_nl} ? ".\015\012" : "\015\012.\015\012";
+ $self->flush if length($self->{out}) > 40000;
+ 1;
+}
+
+# get one full text line, or last partial line, or undef on eof/error/timeout
+sub get_response_line {
+ my($self) = @_;
+ my($ind); my($attempts) = 0;
+ for (;;) {
+ if (($ind=index($self->{inp},"\015\012")) >= 0) {
+ return substr($self->{inp},0,$ind+2,'');
+ } elsif ($self->{inpeof} && $self->{inp} eq '') {
+ return undef; # undef on end-of-file
+ } elsif ($self->{inpeof}) { # return partial last line
+ my($str) = $self->{inp}; $self->{inp} = ''; return $str;
+ } elsif ($attempts > 0) {
+ return undef; # timeout or error
+ }
+ # try reading some more input, one attempt only
+ $self->rw_loop(1,0); $attempts++;
+ }
+}
+
+sub smtp_response {
+ my($self) = @_;
+ my($resp) = ''; my($line,$code); my($first) = 1;
+ for (;;) {
+ $line = $self->get_response_line;
+ last if !defined($line); # eof, error, timeout
+ my($line_complete) = $line =~ s/\015\012\z//s;
+ $line .= ' INCOMPLETE' if !$line_complete;
+ my($more); local($1,$2);
+ $line =~ s/^(\d{3})(-| |\z)//s;
+ if ($first) { $code = $1; $first = 0 } else { $resp .= "\n" }
+ $resp .= $line; $more = $2 eq '-';
+ last if !$more || !$line_complete;
+ }
+ !defined($line) && !defined($code) ? undef : $code.' '.$resp;
+}
+
+sub helo { my($self) = shift; $self->init; $self->command("HELO", at _) }
+sub ehlo { my($self) = shift; $self->init; $self->command("EHLO", at _) }
+sub lhlo { my($self) = shift; $self->init; $self->command("LHLO", at _) }
+sub noop { my($self) = shift; $self->command("NOOP", at _) }
+sub rset { my($self) = shift; $self->command("RSET", at _) }
+sub auth { my($self) = shift; $self->command("AUTH", at _) }
+sub data { my($self) = shift; $self->command("DATA", at _) }
+sub quit { my($self) = shift; $self->command("QUIT", at _) }
+
+sub mail {
+ my($self,$reverse_path,%params) = @_;
+ my(@mail_parameters) =
+ map { my($v)=$params{$_}; defined($v) ? "$_=$v" : "$_" } (keys %params);
+ $self->command("MAIL FROM:", $reverse_path, @mail_parameters);
+}
+
+sub recipient {
+ my($self,$forward_path,%params) = @_;
+ my(@rcpt_parameters) =
+ map { my($v)=$params{$_}; defined($v) ? "$_=$v" : "$_" } (keys %params);
+ $self->command("RCPT TO:", $forward_path, @rcpt_parameters);
+}
+
+1;
+
package Amavis::Out::SMTP;
use strict;
use re 'taint';
@@ -12700,57 +14489,21 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT = qw(&mail_via_smtp);
}
use IO::Wrap;
-use Net::Cmd 2.24;
-use Net::SMTP 2.24;
# use Authen::SASL;
BEGIN {
import Amavis::Conf qw(:platform c cr ca);
import Amavis::Util qw(untaint min max ll do_log debug_oneshot snmp_count
- xtext_encode xtext_decode am_id prolong_timer);
+ xtext_encode xtext_decode prolong_timer);
import Amavis::Timing qw(section_time);
import Amavis::rfc2821_2822_Tools;
import Amavis::Out::EditHeader;
}
-
-#sub Net::Cmd::debug_print {
-# my($cmd,$out,$text) = @_;
-# do_log(0, "*** %s", $cmd->debug_text($out,$text)) if $out;
-#}
-
-# simple OO wrapper around Net::SMTP::datasend to provide a method 'print'
-# and to buffer data, avoiding a bottleneck in Net::Cmd::datasend
-#
-sub new_smtp_data {
- my($class, $handle) = @_;
- bless { handle => $handle, buff => '' }, $class;
-}
-
-sub close { my($self) = shift; $self->flush }
-
-sub print {
- my($self) = shift; $self->{buff} .= join('', at _);
- $self->flush if length($self->{buff}) >= 16384;
- 1;
-}
-
-sub flush {
- my($self) = shift;
- if ($self->{buff} ne '') {
- do_log(-1,"WARN: Unicode string passed to Net::Cmd::datasend (flush)")
- if $unicode_aware && Encode::is_utf8($self->{buff});
- $self->{handle}->datasend($self->{buff})
- or die "datasend timed out while sending buffered data\n";
- $self->{buff} = '';
- }
- 1;
-}
-
# Send mail using SMTP - do multiple transactions if necessary
# (e.g. due to '452 Too many recipients')
@@ -12780,6 +14533,26 @@ sub mail_via_smtp(@) {
1;
}
+# Insert a fabricated enhanced status code if missing in a response to RCPT TO
+#
+sub enhance_rcpt_smtp_response($$$) {
+ my($smtp_resp,$am_id,$mta_id) = @_;
+ local($1,$2,$3);
+ if ($smtp_resp =~ /^ (\d{3}) [ \t]+ ([245] \. \d{1,3} \. \d{1,3})?
+ \s* (.*) \z/xs) {
+ my($resp_code, $resp_enhcode, $resp_msg) = ($1, $2, $3);
+ if ($resp_enhcode eq '' && $resp_code =~ /^([245])/) {
+ my($c1) = $1;
+ $resp_enhcode = $resp_code eq '452' ? "$c1.5.3" : "$c1.1.0";
+ }
+ $smtp_resp = sprintf("%s %s %s, id=%s, from MTA(%s): %s",
+ $resp_code, $resp_enhcode,
+ ($resp_code=~/^2/ ? 'Ok' : 'Failed'),
+ $am_id, $mta_id, $smtp_resp);
+ }
+ $smtp_resp;
+}
+
# Send mail using SMTP - single transaction
# (e.g. forwarding original mail or sending notification)
# May throw exception (die) if temporary failure (4xx) or other problem
@@ -12788,34 +14561,57 @@ sub mail_via_smtp_single(@) {
my($via,$msginfo,$initial_submission,$dsn_per_recip_capable,$filter) = @_;
my($which_section) = 'fwd_init';
snmp_count('OutMsgs');
- local($1,$2,$3); # avoid Perl taint bug, still in 5.8.3
- $via =~ /^smtp: (?: \[ ([^\]]*) \] | ([^:]*) ) : ([^:]*) /six
- or die "Bad fwd method syntax: $via";
- my($relayhost, $relayhost_port) = ($1.$2, $3);
- my($mta_id) = sprintf("[%s]:%s", $relayhost, $relayhost_port);
- my($btype) = $msginfo->body_type;
- if (!defined $btype || uc($btype) eq '7BIT') { $btype = '' }
+ my($am_id) = $msginfo->log_id;
+ my($protocol, $relayhost, $relayhost_port); local($1,$2,$3,$4);
+ if ($via =~ m{^(smtp|lmtp): (?: \[ ([^\]]*) \] | ([^:]*) ) : ([^:]*) }six) {
+ ($protocol, $relayhost, $relayhost_port) = ($1, $2.$3, $4);
+ undef $relayhost_port if $relayhost_port eq '';
+ } elsif ($via =~ m{^(smtp|lmtp): (/[^ ].*) \z}six) { # looks like unix sockt
+ ($protocol, $relayhost) = ($1, $2);
+ } else { die "Bad fwd method syntax: $via" }
+ my($lmtp) = lc($protocol) eq 'lmtp' ? 1 : 0; # rfc2033
+ if ($lmtp && $relayhost_port == 25)
+ { die "rfc2033: LMTP protocol MUST NOT be used on the TCP port 25: $via" }
+ my($mta_id) = !defined $relayhost_port ? sprintf("[%s]", $relayhost)
+ : sprintf("[%s]:%s", $relayhost, $relayhost_port);
my($dsn_envid) = $msginfo->dsn_envid; my($dsn_ret) = $msginfo->dsn_ret;
- my($logmsg) = sprintf("%s via SMTP: %s", ($initial_submission?'SEND':'FWD'),
+ my($logmsg) = sprintf("%s via %s: %s",
+ $initial_submission?'SEND':'FWD', $lmtp?'LMTP':'SMTP',
qquote_rfc2821_local($msginfo->sender) );
my(@per_recip_data) = grep { !$_->recip_done && (!$filter || &$filter($_)) }
@{$msginfo->per_recip_data};
- if (!@per_recip_data) { do_log(5, "%s, nothing to do", $logmsg); return 1 }
+ if (!@per_recip_data) { do_log(5,"%s, nothing to do", $logmsg); return 1 }
ll(4) && do_log(4, "(about to connect to %s) %s -> %s", $mta_id, $logmsg,
join(',', qquote_rfc2821_local(
map {$_->recip_final_addr} @per_recip_data) ));
my($msg) = $msginfo->mail_text; # a file handle or a MIME::Entity object
- my($smtp_handle, $smtp_response); my($smtp_code, $smtp_msg, $received_cnt);
- my($any_valid_recips) = 0; my($any_tempfail_recips) = 0;
- my($dsn_capable) = 0; my($eightbitmime_capable) = 0;
+ my($smtp_handle, $smtp_resp); my($received_cnt);
+ my($any_valid_recips) = 0; my($any_tempfail_recips) = 0; my($pipelining) = 0;
+ my($mimetransport8bit_capable) = 0; # rfc1652
+ my($dsn_capable) = 0; my($auth_capable) = 0;
my($any_valid_recips_and_data_sent) = 0; my($in_datasend_mode) = 0;
+ my(%from_options);
+ my($smtp_connect_timeout) = 30;
+ my($smtp_helo_timeout) = 300;
+ my($smtp_xforward_timeout) = 300;
+ my($smtp_mail_timeout) = 300;
+ my($smtp_rcpt_timeout) = 300;
+ my($smtp_data_init_timeout) = 120;
+ my($smtp_data_xfer_timeout) = 180;
+ my($smtp_data_done_timeout) = 600;
+ my($smtp_quit_timeout) = 10; # 300
+ my($smtp_rset_timeout) = 20;
if (defined($msg) && !$msg->isa('MIME::Entity')) {
$msg = IO::Wrap::wraphandle($msg); # ensure we have an IO::Handle-like obj
- $msg->seek(0,0) or die "Can't rewind mail file: $!";
- }
- # NOTE: Net::SMTP uses alarm to do its own timing.
- # We need to restart our timer when Net::SMTP is done using it !!!
- my($remaining_time) = alarm(0); # check how much time is left, stop timer
+ $msg->seek(0,0) or die "mail_via_smtp_single: Can't rewind mail file: $!";
+ }
+ # can appear anywhere in a pipelined command group:
+ # RSET, MAIL FROM, SEND FROM, SOML FROM, SAML FROM, RCPT TO, data
+ # can only appear as the last command in a pipelined group: --> flush
+ # EHLO, DATA, VRFY, EXPN, TURN, QUIT, NOOP, (and all unknown)
+ my($remaining_time) = alarm(0); # check time left, stop the timer
+ my($deadline) = time + $remaining_time;
+ my($err);
eval {
$which_section = 'fwd-connect';
# Timeout should be more than MTA normally takes to check DNS and RBL,
@@ -12824,35 +14620,52 @@ sub mail_via_smtp_single(@) {
# for SMTP status line prematurely, resulting in status code 000.
# rfc2821 (section 4.5.3.2) requires timeout to be at least 5 minutes
my($localaddr) = c('local_client_bind_address'); # IP assigned to socket
- my($heloname) = c('localhost_name'); # host name used in HELO/EHLO
- $! = 0; $@ = undef; # seems like Net::SMTP puts its error status in $@
- $smtp_handle = Net::SMTP->new($relayhost, Port => $relayhost_port,
- ($localaddr eq '' ? () : (LocalAddr => $localaddr)),
- ($heloname eq '' ? () : (Hello => $heloname)),
- ExactAddresses => 1,
- Timeout => max(60, min(5*60, $remaining_time)), # for each operation
-# Timeout => 0, # no timeouts, disable nonblocking mode on socket
- # Debug => debug_oneshot(),
- );
+ my($heloname) = c('localhost_name'); # host name used in EHLO/HELO/LHLO
+ $heloname = 'localhost' if $heloname eq '';
+ $smtp_handle = Amavis::Out::SMTP::Protocol->new(
+ $relayhost, Port => $relayhost_port, LocalAddr => $localaddr);
defined($smtp_handle) # don't change die text, it is referred to later
or die "Can't connect to $relayhost port $relayhost_port, $@ ($!)";
+ $smtp_handle->timeout($smtp_connect_timeout);
+ $smtp_resp = $smtp_handle->smtp_response; # fetch greeting
+ do_log(3,"smtp resp to greeting: %s", $smtp_resp);
+ $smtp_resp=~/^2/ or die "Negative greeting: $smtp_resp";
+ # send EHLO/LHLO/HELO
+ $smtp_handle->timeout(max(60,min($smtp_helo_timeout,$deadline-time)));
+ if ($lmtp) { $smtp_handle->lhlo($heloname) } #flush!
+ else { $smtp_handle->ehlo($heloname) } #flush!
+ $smtp_resp = $smtp_handle->smtp_response; # fetch response to EHLO/LHLO
+ do_log(3,"smtp resp to %s: %s", $lmtp?'LHLO':'EHLO', $smtp_resp);
+ if ($smtp_resp =~ /^2/) { # good
+ } elsif ($lmtp) { # no fallback possible
+ $smtp_resp=~/^2/ or die "Negative SMTP resp. to LHLO: $smtp_resp";
+ } else { # fallback to HELO
+ $smtp_handle->helo($heloname); #flush!
+ $smtp_resp = $smtp_handle->smtp_response; # fetch response to HELO
+ do_log(3,"smtp resp to HELO: %s", $smtp_resp);
+ $smtp_resp=~/^2/ or die "Negative SMTP resp. to HELO: $smtp_resp";
+ }
+ $smtp_handle->ehlo_response_parse($smtp_resp);
$dsn_capable = c('propagate_dsn_if_possible') &&
- defined($smtp_handle->supports('DSN')); # undocumented method
- my($net_smtp_supports_orcpt) = Net::SMTP->VERSION > 2.29;
- my($net_smtp_supports_envid) = $net_smtp_supports_orcpt;
- my($net_smtp_supports_auth) = $net_smtp_supports_orcpt;
- $eightbitmime_capable =
- defined($smtp_handle->supports('8BITMIME')); # rfc1652
- ll(5) && do_log(5,"Remote host presents itself as: %s%s, %sORCPT",
- $smtp_handle->domain, $dsn_capable ? ', handles DSN' : '',
- $net_smtp_supports_orcpt ? '' : 'no ');
+ defined($smtp_handle->supports('DSN')); # rfc3461
+ $mimetransport8bit_capable =
+ defined($smtp_handle->supports('8BITMIME')); # rfc1652
+ $pipelining = defined($smtp_handle->supports('PIPELINING')); # rfc2920
+ do_log(3,"No announced PIPELINING support by MTA?") if !$pipelining;
+ ll(5) && do_log(5,"Remote host presents itself as: %s%s%s",
+ $smtp_handle->domain,
+ $dsn_capable ? ', handles DSN' : '',
+ $pipelining ? ', handles PIPELINING' : '');
+ if ($lmtp && !$pipelining) # rfc2033 requirements
+ { die "A LMTP server implementation MUST implement PIPELINING" }
+ if ($lmtp && !defined($smtp_handle->supports('ENHANCEDSTATUSCODES')))
+ { die "A LMTP server implementation MUST implement ENHANCEDSTATUSCODES" }
section_time($which_section);
- prolong_timer($which_section, $remaining_time); # restart timer
- $remaining_time = undef;
$which_section = 'fwd-xforward';
- if ($msginfo->client_addr ne '' && $smtp_handle->supports('XFORWARD')) {
- my($cmd) = join(' ', 'XFORWARD', map
+ if ($msginfo->client_addr ne '' &&
+ defined($smtp_handle->supports('XFORWARD'))) {
+ my(@params) = map
{ my($n,$v) = @$_;
# Postfix since version 20060610 uses xtext-encoded (rfc3461)
# strings in XCLIENT and XFORWARD attribute values, previous
@@ -12865,11 +14678,15 @@ sub mail_via_smtp_single(@) {
$v = substr($v,0,255) if length($v) > 255; # chop xtext, not nice
$v eq '' ? () : ("$n=$v") }
( ['ADDR', $msginfo->client_addr], ['NAME',$msginfo->client_name],
- ['PROTO',$msginfo->client_proto],['HELO',$msginfo->client_helo] ));
- do_log(5, "sending %s", $cmd);
- $smtp_handle->command($cmd);
- $smtp_handle->response()==2 or die "sending $cmd\n";
- section_time($which_section); prolong_timer($which_section);
+ ['PROTO',$msginfo->client_proto],['HELO',$msginfo->client_helo] );
+ $smtp_handle->timeout(
+ max(60,min($smtp_xforward_timeout,$deadline-time)));
+ $smtp_handle->command('XFORWARD', at params); #flush!
+ $smtp_resp = $smtp_handle->smtp_response; # fetch response to XFORWARD
+ do_log(3,"smtp resp to XFORWARD: %s", $smtp_resp);
+ $smtp_resp=~/^2/
+ or do_log(-1,"Negative SMTP resp. to XFORWARD: %s", $smtp_resp);
+ section_time($which_section);
}
$which_section = 'fwd-auth';
@@ -12886,43 +14703,63 @@ sub mail_via_smtp_single(@) {
} else {
do_log(3,"INFO: authenticating %s, server supports AUTH %s",
$auth_user,$mechanisms);
- my($sasl) = Authen::SASL->new(
- 'callback' => { 'user' => $auth_user, 'authname' => $auth_user,
- 'pass' => $msginfo->auth_pass });
- $smtp_handle->auth($sasl) or die "sending AUTH, user=$auth_user\n";
- section_time($which_section); prolong_timer($which_section);
+ $auth_capable = 1;
+# my($sasl) = Authen::SASL->new(
+# 'callback' => { 'user' => $auth_user, 'authname' => $auth_user,
+# 'pass' => $msginfo->auth_pass });
+# $smtp_handle->auth($sasl) or die "sending AUTH, user=$auth_user\n";
+ do_log(0,"Sorry, AUTH not supported in this version or amavisd!");
+ section_time($which_section);
}
$which_section = 'fwd-mail-from';
+ $smtp_handle->timeout(max(60,min($smtp_mail_timeout,$deadline-time)));
+ my($fetched_mail_resp) = 0; my($fetched_rcpt_resp) = 0;
+ my($data_command_accepted) = 0;
if ($initial_submission && $dsn_capable && !defined($dsn_envid)) {
# ENVID identifies transaction, not a message
$dsn_envid = xtext_encode(sprintf("AM.%s.%s@%s",
$msginfo->mail_id, iso8601_utc_timestamp(time), c('myhostname')));
}
my($submitter) = $msginfo->auth_submitter;
- $smtp_handle->mail(qquote_rfc2821_local($msginfo->sender),
- $eightbitmime_capable && uc($btype) eq '8BITMIME' ? (Bits => '8') : (),
- $dsn_capable && defined $dsn_ret ? (Return => $dsn_ret) : (),
- $dsn_capable && defined $dsn_envid ? # Net::SMTP expects non-encoded
- ( $net_smtp_supports_envid ? (ENVID => $dsn_envid)
- : (Envelope => xtext_decode($dsn_envid)) ) : (),
- $net_smtp_supports_auth && $mechanisms ne '' &&
- defined($submitter) && $submitter ne '' && $submitter ne '<>' ?
- (AUTH => xtext_encode($submitter)) : (),
- ) or die "sending MAIL FROM\n";
- section_time($which_section); prolong_timer($which_section);
+ my($btype) = $msginfo->body_type;
+ $from_options{'BODY'} = uc($btype) if $mimetransport8bit_capable
+ && defined($btype) && $btype ne '';
+ $from_options{'RET'} = $dsn_ret if $dsn_capable && defined $dsn_ret;
+ $from_options{'ENVID'} = $dsn_envid if $dsn_capable && defined $dsn_envid;
+ $from_options{'AUTH'} = xtext_encode($submitter) # rfc2554
+ if $auth_capable &&
+ defined($submitter) && $submitter ne '' && $submitter ne '<>';
+ my($faddr) = qquote_rfc2821_local($msginfo->sender);
+ $smtp_handle->mail($faddr, %from_options); # MAIL FROM
+ if (!$pipelining) {
+ $smtp_resp = $smtp_handle->smtp_response; $fetched_mail_resp = 1;
+ my($ok) = $smtp_resp =~ /^2/;
+ do_log($ok?3:1, "smtp resp to MAIL: %s", $smtp_resp);
+ if (!$ok) {
+ for my $r (@per_recip_data) {
+ next if $r->recip_done;
+ $r->recip_remote_mta($relayhost);
+ $r->recip_remote_mta_smtp_response($smtp_resp);
+ $r->recip_smtp_response($smtp_resp); $r->recip_done(2);
+ }
+ }
+ }
+ section_time($which_section) if !$pipelining; # otherwise it just shows 0
$which_section = 'fwd-rcpt-to';
+ $smtp_handle->timeout(max(60,min($smtp_rcpt_timeout,$deadline-time)));
my($skipping_resp);
for my $r (@per_recip_data) { # send recipient addresses
+ next if $r->recip_done;
if (defined $skipping_resp) {
$r->recip_smtp_response($skipping_resp); $r->recip_done(2);
next;
}
- # send a RCPT TO command and get the response
+ # prepare to send a RCPT TO command
my($raddr) = qquote_rfc2821_local($r->recip_final_addr);
if (!$dsn_capable) {
- $smtp_handle->recipient($raddr);
+ $smtp_handle->recipient($raddr); # RCPT TO
} else {
my(@dsn_notify); # implies a default when the list is empty
my($dn) = $r->dsn_notify;
@@ -12936,238 +14773,273 @@ sub mail_via_smtp_single(@) {
join(',', at dsn_notify));
}
}
- ll(5) && do_log(5, "sending RCPT TO:%s %s", $raddr,
- join(' ', (@dsn_notify ? 'NOTIFY='.join(',', at dsn_notify) : ()),
- $net_smtp_supports_orcpt && defined $r->dsn_orcpt
- ? 'ORCPT='.$r->dsn_orcpt : ''));
- $smtp_handle->recipient($raddr, {
- @dsn_notify ? (Notify => \@dsn_notify) : (),
- $net_smtp_supports_orcpt && defined $r->dsn_orcpt
- ? (ORCPT => $r->dsn_orcpt) : (),
- });
- }
- $smtp_code = $smtp_handle->code;
- $smtp_msg = $smtp_handle->message;
- chomp($smtp_msg);
- my($rcpt_smtp_resp) = "$smtp_code $smtp_msg";
- if ($smtp_code =~ /^2/) {
- $any_valid_recips++;
- do_log(3, 'response to RCPT TO for %s: "%s"', $raddr,$rcpt_smtp_resp);
- } else { # not ok
- do_log(1, 'response to RCPT TO for %s: "%s"', $raddr,$rcpt_smtp_resp);
- if ($rcpt_smtp_resp =~ /^0/) {
- # timeout, what to do, could cause duplicates
- do_log(-1, "response to RCPT TO not yet available");
- $rcpt_smtp_resp = "450 4.4.2 ($rcpt_smtp_resp - probably timed out)";
+ my(%rcpt_options);
+ $rcpt_options{'NOTIFY'} =
+ join(",", map { uc($_) } @dsn_notify) if @dsn_notify;
+ $rcpt_options{'ORCPT'} = $r->dsn_orcpt if defined $r->dsn_orcpt;
+ $smtp_handle->recipient($raddr, %rcpt_options); # RCPT TO
+ }
+ if (!$pipelining) { # must fetch responses to RCPT TO right away
+ $smtp_resp = $smtp_handle->smtp_response; $fetched_rcpt_resp = 1;
+ $r->recip_remote_mta($relayhost);
+ $r->recip_remote_mta_smtp_response($smtp_resp);
+ $smtp_resp = enhance_rcpt_smtp_response($smtp_resp,$am_id,$mta_id);
+ $r->recip_smtp_response($smtp_resp); # preliminary response
+ my($ok) = $smtp_resp =~ /^2/;
+ do_log($ok?3:1, "smtp resp to RCPT (%s): %s", $raddr,$smtp_resp);
+ if ($ok) { $any_valid_recips++ }
+ else {
+ if ($smtp_resp =~ /^452/) { # too many recipients - see rfc2821
+ do_log(-1, 'Only %d recips sent in one go: "%s"',
+ $any_valid_recips, $smtp_resp)
+ if !defined($skipping_resp);
+ $skipping_resp = $smtp_resp;
+ } elsif ($smtp_resp =~ /^4/) { $any_tempfail_recips++ }
+ $r->recip_done(2); # got a negative response to RCPT TO
}
- $r->recip_remote_mta($relayhost);
- $r->recip_remote_mta_smtp_response($rcpt_smtp_resp);
- if ($rcpt_smtp_resp =~ /^ (\d{3}) [ \t]+ ([245] \. \d{1,3} \. \d{1,3})?
- \s* (.*) \z/xs)
- {
- my($resp_code, $resp_enhcode, $resp_msg) = ($1, $2, $3);
- if ($resp_enhcode eq '' && $resp_code =~ /^([245])/) {
- my($c1) = $1;
- $resp_enhcode = $resp_code eq '452' ? "$c1.5.3" : "$c1.1.0";
- }
- $rcpt_smtp_resp = sprintf("%s %s %s, id=%s, from MTA(%s): %s",
- $resp_code, $resp_enhcode,
- ($resp_code=~/^2/ ? 'Ok' : 'Failed'),
- am_id(), $mta_id, $rcpt_smtp_resp);
- }
- if ($rcpt_smtp_resp =~ /^452/) { # too many recipients - see rfc2821
- do_log(-1, 'Only %d recips sent in one go: "%s"',
- $any_valid_recips, $rcpt_smtp_resp);
- $skipping_resp = $rcpt_smtp_resp;
- } elsif ($rcpt_smtp_resp =~ /^4/) {
- $any_tempfail_recips++;
- $smtp_response = $rcpt_smtp_resp if !defined($smtp_response);
- }
- $r->recip_smtp_response($rcpt_smtp_resp); $r->recip_done(2);
- $smtp_response = $rcpt_smtp_resp
- if $rcpt_smtp_resp =~ /^5/ && $smtp_response !~ /^5/; # keep first 5x
- }
- }
- section_time($which_section); prolong_timer($which_section);
- $smtp_code = $smtp_msg = undef;
-
- if (!$any_valid_recips) {
- do_log(-1,"mail_via_smtp: DATA skipped, no valid recips, %s",
- $any_tempfail_recips);
- } elsif ($any_tempfail_recips && !$dsn_per_recip_capable) {
- # we must not proceede if mail did not came in as LMTP,
+ }
+ }
+ section_time($which_section) if !$pipelining; # otherwise it just shows 0
+
+ my($what_cmd);
+ if ($fetched_rcpt_resp && !$any_valid_recips) { # no pipelining
+ # it is known there are no valid recipients, don't go into DATA section
+ do_log(0,"no valid recipients, skip data transfer");
+ # send RSET, just to be nice and explicit
+ $smtp_handle->timeout($smtp_rset_timeout);
+ $what_cmd = 'RSET'; $smtp_handle->rset; # RSET
+ } elsif ($fetched_rcpt_resp && $any_tempfail_recips &&
+ !$dsn_per_recip_capable) {
+ # we must not proceed if mail did not came in as LMTP,
# or we would generate mail duplicates on each delivery attempt
do_log(-1,"mail_via_smtp: DATA skipped, tempfailed recips: %s",
$any_tempfail_recips);
- } else { # send the message contents (enter DATA phase)
+ # send RSET, just to be nice and explicit
+ $smtp_handle->timeout($smtp_rset_timeout);
+ $what_cmd = 'RSET'; $smtp_handle->rset; # RSET
+ } else {
$which_section = 'fwd-data-cmd';
- $smtp_handle->data or die "sending DATA command\n";
+ # pipelining in effect, or we have at least one valid recipient, go DATA
+ $smtp_handle->timeout(
+ max(60,min($smtp_data_init_timeout,$deadline-time)));
+ $smtp_handle->data; #flush! DATA
$in_datasend_mode = 1;
-
- my($smtp_resp) = $smtp_handle->code . " " . $smtp_handle->message;
- section_time($which_section); prolong_timer($which_section);
- $which_section = 'fwd-data-contents';
- chomp($smtp_resp);
- do_log(4, 'response to DATA: "%s"', $smtp_resp);
-
- # provide OO wrapper and buffering around Net::Cmd::datasend
- my($smtp_data_fh) = Amavis::Out::SMTP->new_smtp_data($smtp_handle);
-
- my($hdr_edits) = $msginfo->header_edits;
- $hdr_edits = Amavis::Out::EditHeader->new if !$hdr_edits;
- $received_cnt =
- $hdr_edits->write_header($msg,$smtp_data_fh,!$initial_submission);
- if ($received_cnt > 100) {
- # loop detection required by rfc2821 6.2
- # Do not modify the signal text, it gets matched elsewhere!
- die "Too many hops: $received_cnt 'Received:' header lines\n";
- }
- if (!defined($msg)) {
- # empty mail
- } elsif ($msg->isa('MIME::Entity')) {
- $msg->print_body($smtp_data_fh);
+ if (!$fetched_mail_resp) { # pipelining in effect, late response to MAIL
+ $which_section = 'fwd-mail-pip';
+ $smtp_handle->timeout(max(60,min($smtp_mail_timeout,$deadline-time)));
+ $smtp_resp = $smtp_handle->smtp_response; $fetched_mail_resp = 1;
+ my($ok) = $smtp_resp =~ /^2/;
+ do_log($ok?3:1, "smtp resp to MAIL (pip): %s", $smtp_resp);
+ if (!$ok) {
+ for my $r (@per_recip_data) {
+ next if $r->recip_done;
+ $r->recip_remote_mta($relayhost);
+ $r->recip_remote_mta_smtp_response($smtp_resp);
+ $r->recip_smtp_response($smtp_resp); $r->recip_done(2);
+ }
+ }
+ section_time($which_section);
+ }
+ if (!$fetched_rcpt_resp) { # pipelining in effect, late response to RCPT
+ $which_section = 'fwd-rcpt-pip';
+ $smtp_handle->timeout(max(60,min($smtp_rcpt_timeout,$deadline-time)));
+ for my $r (@per_recip_data) {
+ next if $r->recip_done;
+ $smtp_resp = $smtp_handle->smtp_response; $fetched_rcpt_resp = 1;
+ $r->recip_remote_mta($relayhost);
+ $r->recip_remote_mta_smtp_response($smtp_resp);
+ $smtp_resp = enhance_rcpt_smtp_response($smtp_resp,$am_id,$mta_id);
+ $r->recip_smtp_response($smtp_resp); # preliminary response
+ my($ok) = $smtp_resp =~ /^2/;
+ my($raddr) = qquote_rfc2821_local($r->recip_final_addr);
+ do_log($ok?3:1, "smtp resp to RCPT (pip) (%s): %s",
+ $raddr,$smtp_resp);
+ if ($ok) { $any_valid_recips++ }
+ else {
+ if ($smtp_resp =~ /^452/) { # too many recipients - see rfc2821
+ do_log(-1, 'Only %d recips sent in one go: "%s"',
+ $any_valid_recips, $smtp_resp);
+ $skipping_resp = $smtp_resp;
+ } elsif ($smtp_resp =~ /^4/) { $any_tempfail_recips++ }
+ $r->recip_done(2); # got a negative response to RCPT TO
+ }
+ }
+ section_time($which_section);
+ }
+ $which_section = 'fwd-data-chkpnt' if $pipelining;
+ $smtp_handle->timeout(
+ max(60,min($smtp_data_init_timeout,$deadline-time)));
+ $smtp_resp = $smtp_handle->smtp_response; # fetch response to DATA
+ do_log(3,"smtp resp to DATA: %s", $smtp_resp);
+ section_time($which_section);
+ $data_command_accepted = $smtp_resp=~/^3/ ? 1 : 0;
+ if (!$data_command_accepted) {
+ do_log(0,"Negative SMTP resp to DATA: %s", $smtp_resp);
+ } elsif (!$any_valid_recips) { # pipelining
+ do_log(2,"Too late, DATA accepted but no valid recips, send dummy");
+ $which_section = 'fwd-data-contents';
+ $smtp_handle->timeout(
+ max(60,min($smtp_data_xfer_timeout,$deadline-time)));
+ $smtp_handle->dataend; # as required by rfc2920: if the DATA command
+ # was accepted the SMTP client should send a single dot
+ } elsif ($any_tempfail_recips && !$dsn_per_recip_capable) { # pipelining
+ # we must not proceed if mail did not came in as LMTP,
+ # or we would generate mail duplicates on each delivery attempt
+ do_log(2,"Too late, DATA accepted but tempfailed recips, bail out");
+ die "Bail out, DATA accepted but tempfailed recips, not a LMTP input";
} else {
- my($nbytes,$buff);
- # Using fixed-size reads instead of line-by-line approach
- # makes feeding mail back to MTA (e.g. Postfix) more than
- # twice as fast for larger mail.
-
-### # to reduce likelyhood of a qmail bare-LF bug (bare LF reported when
-### # CR and LF are separated by a TCP packet boundary) one may use this
-### # 'while' loop, reading line by line, instead of the normal one below
-### for ($! = 0; defined($buff=$msg->getline); $! = 0) {
-### $smtp_handle->datasend($buff)
-### or die "datasend timed out while sending body";
-### }
-### defined $buff || $!==0 or die "Error reading: $!";
-
- # must flush buffering through $smtp_data_fh, as from now on
- # we'll be calling Net::Cmd::datasend directly for speed
- $smtp_data_fh->flush or die "Error flushing smtp_data_fh: $!";
-
- while (($nbytes=$msg->read($buff,16384)) > 0) {
- do_log(-1,"WARN: Unicode string passed to Net::Cmd::datasend")
- if $unicode_aware && Encode::is_utf8($buff);
- $smtp_handle->datasend($buff)
- or die "datasend timed out while sending body";
+ $which_section = 'fwd-data-contents';
+ $smtp_handle->timeout(
+ max(60,min($smtp_data_xfer_timeout,$deadline-time)));
+ my($hdr_edits) = $msginfo->header_edits;
+ $hdr_edits = Amavis::Out::EditHeader->new if !$hdr_edits;
+ $received_cnt =
+ $hdr_edits->write_header($msg,$smtp_handle,!$initial_submission);
+ if ($received_cnt > 100) {
+ # loop detection required by rfc2821 6.2
+ # Do not modify the signal text, it gets matched elsewhere!
+ die "Too many hops: $received_cnt 'Received:' header lines\n";
}
- defined $nbytes or die "Error reading: $!";
- }
- $smtp_data_fh->close or die "Error closing smtp_data_fh: $!";
- $smtp_data_fh = undef;
- section_time($which_section); prolong_timer($which_section);
-
- $which_section = 'fwd-data-end';
- # don't check status of dataend here, it may not yet be available
- $smtp_handle->dataend;
- $in_datasend_mode = 0; $any_valid_recips_and_data_sent = 1;
- section_time($which_section); prolong_timer($which_section);
-
- $which_section = 'fwd-rundown-1';
- # figure out the final SMTP response
- $smtp_code = $smtp_handle->code;
- my(@msgs) = $smtp_handle->message;
- # only the 'command()' resets messages list, so now we have both:
- # 'End data with <CR><LF>.<CR><LF>' and 'Ok: queued as...' in @msgs
- # and only the last SMTP response code in $smtp_handle->code
- my($smtp_msg) = $msgs[$#msgs]; chomp($smtp_msg); # take the last one
- $smtp_response = "$smtp_code $smtp_msg";
- do_log(4, 'response to data end: "%s"', $smtp_response);
+ if (!defined($msg)) {
+ # empty mail
+ } elsif ($msg->isa('MIME::Entity')) {
+ $msg->print_body($smtp_handle);
+ } else {
+ my($nbytes,$buff);
+ while (($nbytes=$msg->read($buff,65536)) > 0)
+ { $smtp_handle->datasend($buff) }
+ defined $nbytes or die "Error reading: $!";
+ }
+ section_time($which_section);
+
+ $which_section = 'fwd-data-end';
+ $smtp_handle->timeout(
+ max(60,min($smtp_data_done_timeout,$deadline-time)));
+ $what_cmd = 'data-dot';
+ $smtp_handle->dataend; # .<CR><LF>
+ $in_datasend_mode = 0; $any_valid_recips_and_data_sent = 1;
+ section_time($which_section) if !$pipelining; # otherwise it shows 0
+ }
+ }
+ if ($pipelining) { $smtp_handle->quit } #flush! QUIT
+ $which_section = 'fwd-rundown-1';
+ if (!defined $what_cmd) { # not expecting a response?
+ } elsif ($what_cmd ne 'data-dot') { # must be a response to a rset
+ $smtp_resp = $smtp_handle->smtp_response; # resp to RSET
+ do_log(3,"smtp resp to %s: %s", $what_cmd,$smtp_resp);
+ $smtp_resp=~/^2/ or die "Negative SMTP resp. to $what_cmd: $smtp_resp";
+ } else { # get response(s) to data-dot
# replace success responses to RCPT TO commands with a final response
+ my($first) = 1;
for my $r (@per_recip_data) {
- next if $r->recip_done; # skip those that failed at RCPT TO
+ next if $r->recip_done; # skip those that failed at earlier stages
+ if ($lmtp || $first) {
+ $first = 0; my($raddr) = qquote_rfc2821_local($r->recip_final_addr);
+ $raddr .= ', etc.' if !$lmtp && @per_recip_data > 1;
+ $smtp_resp = $smtp_handle->smtp_response; # resp to data-dot
+ do_log(3,"smtp resp to %s (%s): %s", $what_cmd,$raddr,$smtp_resp);
+ $smtp_resp=~/^2/ or do_log(0,"Negative SMTP resp. to %s (%s): %s",
+ $what_cmd,$raddr,$smtp_resp);
+ }
$r->recip_remote_mta($relayhost);
- $r->recip_remote_mta_smtp_response($smtp_response);
- }
- }
- };
- my($err) = $@;
+ $r->recip_remote_mta_smtp_response($smtp_resp);
+ $r->recip_smtp_response($smtp_resp); $r->recip_done(2);
+ }
+ if ($first) { # fetch an uncollected response
+ # fetch unprocessed response if all recipients were rejected
+ # but we nevertheless somehow entered a data transfer mode
+ # (i.e. if a SMTP server failed to reject a DATA command).
+ # rfc2033: when there have been no successful RCPT commands in the
+ # mail transaction, the DATA command MUST fail with a 503 reply code
+ $smtp_resp = $smtp_handle->smtp_response; # resp to data-dot
+ do_log(3,"smtp resp to _dummy_ data %s: %s", $what_cmd,$smtp_resp);
+ }
+ }
+ $smtp_handle->timeout($smtp_quit_timeout);
+ if ($pipelining) {} # QUIT was already sent
+ else { $smtp_handle->quit } #flush! # QUIT
+ $smtp_resp = $smtp_handle->smtp_response;
+ do_log(3,"smtp resp to QUIT: %s", $smtp_resp);
+ $smtp_resp=~/^2/ or do_log(0,"Negative SMTP resp. to QUIT: %s",$smtp_resp);
+ $smtp_handle->close or die "Error closing Amavis::Out::SMTP::Protocol obj";
+ undef $smtp_handle;
+ 1;
+ # some unusual error conditions _are_ captured by eval, but fail to set $@
+ } or do { $err = $@ ne '' ? $@ : "errno=$!" };
my($saved_section_name) = $which_section;
- if ($err ne '') { chomp($err); $err = ' ' if $err eq '' } # careful chomp
- prolong_timer($which_section, $remaining_time); # restart the timer
- $which_section = 'fwd-rundown';
- if ($err ne '') { # fetch info about failure
- do_log(3, "mail_via_smtp: session failed: %s", $err);
- if (!defined($smtp_handle)) { $smtp_code = ''; $smtp_msg = '' }
- else {
- $smtp_code = $smtp_handle->code; $smtp_msg = $smtp_handle->message;
- chomp($smtp_msg);
- }
- }
+ $which_section = 'fwd-end-chkpnt';
+ if ($err ne '') { chomp $err; $err = ' ' if $err eq '' } # careful chomp
+ do_log(2,"mail_via_smtp: session failed: %s", $err) if $err ne '';
+ prolong_timer($which_section, $deadline - time); # restart timer
# terminate the SMTP session if still alive
- if (!defined $smtp_handle) {
+ if (!defined($smtp_handle)) {
# nothing
} elsif ($in_datasend_mode) {
- # We are aborting SMTP session. DATA send mode must NOT be normally
- # terminated with a dataend (dot), otherwise recipient will receive
- # a chopped-off mail (and possibly be receiving it over and over again
- # during each MTA retry.
+ # We are aborting SMTP session. Data transfer mode must NOT be terminated
+ # with a dataend (dot), otherwise recipient will receive a chopped-off mail
+ # (and possibly be receiving it over and over again during each MTA retry.
do_log(-1, "mail_via_smtp: NOTICE: aborting SMTP session, %s", $err);
$smtp_handle->close; # abruptly terminate the SMTP session, ignoring status
+ undef $smtp_handle;
} else {
- $smtp_handle->timeout(15); # don't wait too long for response to a QUIT
- $smtp_handle->quit; # send a QUIT regardless of success so far
- if ($err eq '' && $smtp_handle->status != CMD_OK) {
- do_log(-1,"WARN: sending SMTP QUIT command failed: %s %s",
- $smtp_handle->code, $smtp_handle->message);
- }
+ $smtp_handle->timeout(1); # don't wait for too long
+ $smtp_handle->quit; #flush! # send a QUIT regardless of success so far
+ for (my($cnt)=0; ; $cnt++) { # curious if there are any pending responses
+ my($smtp_resp) = $smtp_handle->smtp_response;
+ last if !defined($smtp_resp);
+ do_log(0,"discarding unprocessed reply: %s", $smtp_resp);
+ if ($cnt > 20) { do_log(-1,"aborting, discarding many replies"); last }
+ }
+ $smtp_handle->close; # terminate the SMTP session, ignoring status
+ undef $smtp_handle;
}
# prepare final smtp response and log abnormal events
- if ($err eq '') { # no errors
+ my($smtp_response);
+ for my $r (@per_recip_data) {
+ my($resp) = $r->recip_smtp_response;
+ $smtp_response = $resp if !defined($smtp_response) ||
+ $resp =~ /^4/ && $smtp_response !~ /^4/ ||
+ $resp !~ /^2/ && $smtp_response !~ /^[45]/;
+ }
+ if ($err eq '') { # no errors
local($1,$2);
if ($any_valid_recips_and_data_sent && $smtp_response !~ /^[245]/) {
$smtp_response =
sprintf("451 4.6.0 Bad SMTP code, id=%s, from MTA(%s): %s",
- am_id(), $mta_id, $smtp_response);
+ $am_id, $mta_id, $smtp_response);
} elsif ($smtp_response =~ /^((\d)\d{2})/) {
my($smtp_code,$smtp_status) = ($1,$2);
$smtp_response = sprintf("%s %d.6.0 %s, id=%s, from MTA(%s): %s",
$smtp_code, $smtp_status, ($smtp_status == 2 ? 'Ok' : 'Failed'),
- am_id(), $mta_id, $smtp_response);
- }
- } elsif ($err eq "timed out" || $err =~ /: Timeout\z/) {
- my($msg) = ($in_datasend_mode && $smtp_code =~ /^354/) ?
- '' : ", $smtp_code $smtp_msg";
- $smtp_response = sprintf("450 4.4.2 Timed out during %s%s, MTA(%s), id=%s",
- $saved_section_name, $msg, $mta_id, am_id());
+ $am_id, $mta_id, $smtp_response);
+ }
+ } elsif ($err =~ /^timed out\b/ || $err =~ /: Timeout\z/) {
+ $smtp_response = sprintf("450 4.4.2 Timed out during %s, MTA(%s), id=%s",
+ $saved_section_name, $mta_id, $am_id);
} elsif ($err =~ /^Can't connect/) {
$smtp_response = sprintf("450 4.4.1 %s, MTA(%s), id=%s",
- $err, $mta_id, am_id());
+ $err, $mta_id, $am_id);
} elsif ($err =~ /^Too many hops/) {
- $smtp_response = sprintf("554 5.4.6 Reject: %s, id=%s", $err, am_id());
- } elsif ($smtp_code =~ /^5/) { # 5xx
- $smtp_response = sprintf("%s 5.5.0 Rejected by MTA(%s): %s %s, id=%s",
- ($smtp_code !~ /^5\d\d\z/ ? "554" : $smtp_code),
- $mta_id, $smtp_code, $smtp_msg, am_id());
- } elsif ($smtp_code =~ /^0/) { # 000
- $smtp_response = sprintf("450 4.4.2 No response from MTA(%s) ".
- "during %s (%s), id=%s",
- $mta_id, $saved_section_name, $err, am_id());
+ $smtp_response = sprintf("554 5.4.6 Reject: %s, id=%s", $err, $am_id);
} else {
- $smtp_response = sprintf("%s 4.5.0 From MTA(%s) ".
- "during %s (%s): %s %s, id=%s",
- ($smtp_code !~ /^4\d\d\z/ ? "451" : $smtp_code),
- $mta_id, $saved_section_name, $err,
- $smtp_code, $smtp_msg, am_id());
+ $smtp_response = sprintf("451 4.5.0 From MTA(%s) during %s (%s): id=%s",
+ $mta_id, $saved_section_name, $err, $am_id);
}
my($ll) = $smtp_response =~ /^2/ ? 1 : -1;
ll($ll) && do_log($ll, "%s -> %s,%s %s", $logmsg,
join(',', qquote_rfc2821_local(
map {$_->recip_final_addr} @per_recip_data)),
- join('', $eightbitmime_capable && uc($btype) eq '8BITMIME' ?
- " BODY=$btype" :"",
- $dsn_capable && defined $dsn_ret ?" RET=$dsn_ret" :"",
- $dsn_capable && defined $dsn_envid ?" ENVID=$dsn_envid":""),
+ join(' ', map { my($v)=$from_options{$_}; defined($v)?"$_=$v":"$_" }
+ (keys %from_options)),
$smtp_response);
if (defined $smtp_response) {
$msginfo->dsn_passed_on($dsn_capable && $smtp_response=~/^2/ &&
!c('terminate_dsn_on_notify_success') ? 1 : 0);
for my $r (@per_recip_data) {
- # attach a SMTP response at data end for each recipient
+ # attach a SMTP response at data-end for each recipient
# that was not already rejected during RCPT TO command
if (!$r->recip_done) { # mark it as done
$r->recip_smtp_response($smtp_response); $r->recip_done(2);
- $r->recip_mbxname($r->recip_final_addr) if $smtp_response =~ /^2/;
} elsif ($any_valid_recips_and_data_sent
&& $r->recip_smtp_response =~ /^452/) {
# 'undo' the RCPT TO '452 Too many recipients' situation,
@@ -13175,11 +15047,12 @@ sub mail_via_smtp_single(@) {
$r->recip_smtp_response(undef); $r->recip_done(undef);
}
}
- if ( $smtp_response =~ /^2/) { snmp_count('OutMsgsDelivers') }
+ if ($smtp_response =~ /^2/) { snmp_count('OutMsgsDelivers') }
elsif ($smtp_response =~ /^4/) { snmp_count('OutAttemptFails') }
elsif ($smtp_response =~ /^5/) { snmp_count('OutMsgsRejects') }
}
section_time($which_section);
+ die $err if $err =~ /^timed out\b/; # resignal timeout
1;
}
@@ -13196,18 +15069,19 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT = qw(&mail_via_pipe);
}
use IO::Wrap;
+use Errno qw(ENOENT EACCES EAGAIN ESRCH);
use POSIX qw(WIFEXITED WIFSIGNALED WIFSTOPPED
WEXITSTATUS WTERMSIG WSTOPSIG);
BEGIN {
import Amavis::Conf qw(:platform c cr ca);
import Amavis::Util qw(untaint min max ll do_log
- am_id snmp_count run_command_consumer
+ snmp_count run_command_consumer
exit_status_str proc_status_ok);
import Amavis::Timing qw(section_time);
import Amavis::rfc2821_2822_Tools;
@@ -13296,7 +15170,7 @@ sub mail_via_pipe(@) {
do_log(5, "mail_via_pipe running command: %s", join(' ', @command));
local $SIG{CHLD} = 'DEFAULT';
local $SIG{PIPE} = 'IGNORE'; # write to broken pipe would throw a signal
- my($proc_fh,$pid) = run_command_consumer(undef,undef, at command);
+ my($proc_fh,$pid) = run_command_consumer(undef,'', at command);
# binmode on pipes and sockets is a default since Perl 5.8.1
binmode($proc_fh) or die "Can't set pipe to binmode: $!";
my($hdr_edits) = $msginfo->header_edits;
@@ -13311,14 +15185,16 @@ sub mail_via_pipe(@) {
$msg->print_body($proc_fh);
} else {
my($nbytes,$buff);
- while (($nbytes=$msg->read($buff,16384)) > 0)
+ while (($nbytes=$msg->read($buff,65536)) > 0)
{ $proc_fh->print($buff) or die "Submitting mail text failed: $!" }
defined $nbytes or die "Error reading: $!";
}
my($smtp_response);
if ($received_cnt > 100) { # loop detection required by rfc2821 6.2
do_log(-2, "Too many hops: %d 'Received:' header lines", $received_cnt);
- kill('TERM',$pid); # kill the process running mail submission program
+ kill('TERM',$pid) or $! == ESRCH # kill the mail submission process
+ or die sprintf("Can't kill process [%s] running %s: %s",
+ $pid,$command[0],$!);
$proc_fh->close; undef $proc_fh; undef $pid; # and ignore status
$smtp_response = "554 5.4.6 Reject: " .
"Too many hops: $received_cnt 'Received:' header lines";
@@ -13347,7 +15223,7 @@ sub mail_via_pipe(@) {
ll(3) && do_log(3,"mail_via_pipe %s, %s, %s", $command[0],
exit_status_str($child_stat,$err), $smtp_response);
}
- $smtp_response .= ", id=" . am_id();
+ $smtp_response .= ", id=" . $msginfo->log_id;
for my $r (@per_recip_data) {
next if $r->recip_done;
$r->recip_smtp_response($smtp_response); $r->recip_done(2);
@@ -13372,7 +15248,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT = qw(&mail_via_bsmtp);
}
@@ -13382,7 +15258,7 @@ use IO::Wrap;
use IO::Wrap;
BEGIN {
import Amavis::Conf qw(:platform $QUARANTINEDIR c cr ca);
- import Amavis::Util qw(untaint min max ll do_log am_id snmp_count);
+ import Amavis::Util qw(untaint min max ll do_log snmp_count);
import Amavis::Timing qw(section_time);
import Amavis::rfc2821_2822_Tools;
import Amavis::Out::EditHeader;
@@ -13407,9 +15283,9 @@ sub mail_via_bsmtp(@) {
$bsmtp_file_final =~ s{%(.)}
{ $1 eq 'b' ? $msginfo->body_digest
: $1 eq 'm' ? $msginfo->mail_id
+ : $1 eq 'n' ? $msginfo->log_id
: $1 eq 's' ? untaint($s)
: $1 eq 'i' ? iso8601_timestamp($msginfo->rx_time,1) #,'-')
- : $1 eq 'n' ? am_id()
: $1 eq '%' ? '%' : '%'.$1 }egs;
# prepend directory if not specified
my($bsmtp_file_final_to_show) = $bsmtp_file_final;
@@ -13429,7 +15305,7 @@ sub mail_via_bsmtp(@) {
$msg = IO::Wrap::wraphandle($msg); # ensure we have an IO::Handle-like obj
$msg->seek(0,0) or die "Can't rewind mail file: $!";
}
- my($mp);
+ my($mp); my($err);
eval {
my($errn) = lstat($bsmtp_file_tmp) ? 0 : 0+$!;
if ($errn == ENOENT) {} # good, no file, as expected
@@ -13453,7 +15329,7 @@ sub mail_via_bsmtp(@) {
$mp->printf("EHLO %s\n", c('localhost_name'))
or die "print failed (EHLO): $!";
my($btype) = $msginfo->body_type; # rfc1652: need "8bit Data"? (rfc2045)
- if (!defined $btype || uc($btype) eq '7BIT') { $btype = '' }
+ $btype = '' if !defined $btype;
my($dsn_envid) = $msginfo->dsn_envid; my($dsn_ret) = $msginfo->dsn_ret;
$mp->printf("MAIL FROM:%s\n", join(' ',
qquote_rfc2821_local($msginfo->sender),
@@ -13503,13 +15379,14 @@ sub mail_via_bsmtp(@) {
rename($bsmtp_file_tmp, $bsmtp_file_final)
or die "Can't rename BSMTP file to $bsmtp_file_final: $!";
$mbxname = $bsmtp_file_final;
- };
- my($err) = $@; my($smtp_response);
+ 1;
+ } or do { $err = $@ ne '' ? $@ : "errno=$!" };
+ my($smtp_response);
if ($err eq '') {
$smtp_response = "250 2.6.0 Ok, queued as BSMTP $bsmtp_file_final_to_show";
snmp_count('OutMsgsDelivers');
} else {
- chomp($err);
+ chomp $err;
unlink($bsmtp_file_tmp)
or do_log(-2,"Can't delete half-finished BSMTP file %s: %s",
$bsmtp_file_tmp, $!);
@@ -13521,8 +15398,9 @@ sub mail_via_bsmtp(@) {
$smtp_response = "451 4.5.0 Writing $bsmtp_file_tmp failed: $err";
snmp_count('OutAttemptFails');
}
- }
- $smtp_response .= ", id=" . am_id();
+ die $err if $err =~ /^timed out\b/; # resignal timeout
+ }
+ $smtp_response .= ", id=" . $msginfo->log_id;
$msginfo->dsn_passed_on($smtp_response=~/^2/ &&
!c('terminate_dsn_on_notify_success') ? 1 : 0);
for my $r (@per_recip_data) {
@@ -13547,7 +15425,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&mail_to_local_mailbox);
}
@@ -13560,7 +15438,7 @@ BEGIN {
BEGIN {
import Amavis::Conf qw(:platform $quarantine_subdir_levels c cr ca);
import Amavis::Lock;
- import Amavis::Util qw(ll do_log am_id run_command_consumer);
+ import Amavis::Util qw(ll do_log run_command_consumer);
import Amavis::Timing qw(section_time);
import Amavis::rfc2821_2822_Tools;
import Amavis::Out::EditHeader;
@@ -13638,8 +15516,8 @@ sub mail_to_local_mailbox(@) {
$suggested_filename =~ s{%(.)}
{ $1 eq 'b' ? $msginfo->body_digest
: $1 eq 'm' ? $msginfo->mail_id
+ : $1 eq 'n' ? $msginfo->log_id
: $1 eq 'i' ? iso8601_timestamp($msginfo->rx_time,1) #,'-')
- : $1 eq 'n' ? am_id()
: $1 eq '%' ? '%' : '%'.$1 }egs;
# $mbxname = File::Spec->catfile($mbxname, $suggested_filename);
$mbxname = "$mbxname/$suggested_filename";
@@ -13664,7 +15542,7 @@ sub mail_to_local_mailbox(@) {
}
}
do_log(1,"local delivery: <%s> -> <%s>, mbx=%s",$sender,$recip,$mbxname);
- my($mp,$pos,$pid);
+ my($eval_stat); my($mp,$pos,$pid);
my($errn) = stat($mbxname) ? 0 : 0+$!;
local $SIG{CHLD} = 'DEFAULT';
local $SIG{PIPE} = 'IGNORE'; # write to broken pipe would throw a signal
@@ -13676,7 +15554,7 @@ sub mail_to_local_mailbox(@) {
else
{ die "File $mbxname exists??? Refuse to overwrite it, $!" }
if ($mbxname =~ /\.gz\z/) {
- $mp = Amavis::IO::Zlib->new;
+ $mp = Amavis::IO::Zlib->new; # ?how to request an exclusive access?
$mp->open($mbxname,'wb')
or die "Can't create gzip file $mbxname: $!";
} else {
@@ -13707,15 +15585,19 @@ sub mail_to_local_mailbox(@) {
$mp->seek(0,2) or die "Can't position mailbox file to its tail: $!";
$pos = $mp->tell;
}
+ section_time('open-mbx');
if (defined($msg) && !$msg->isa('MIME::Entity'))
{ $msg->seek(0,0) or die "Can't rewind mail file: $!" }
+ 1;
+ } or do {
+ $eval_stat = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ $smtp_response =
+ $eval_stat =~ /^timed out\b/ ? "450 4.4.2" : "451 4.5.0";
+ $smtp_response .= " Local delivery(1) to $mbxname failed: $eval_stat";
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
};
- if ($@ ne '') {
- chomp($@);
- $smtp_response = $@ eq "timed out" ? "450 4.4.2" : "451 4.5.0";
- $smtp_response .= " Local delivery(1) to $mbxname failed: $@";
- last; # exit block, not the loop
- }
+ last if $eval_stat ne ''; # exit block, not the loop
+ my($failed) = 0; undef $eval_stat;
eval { # if things fail from here on, try to restore mailbox state
if ($ux) {
# a null return path may not appear in the 'From ' delimiter line
@@ -13744,7 +15626,7 @@ sub mail_to_local_mailbox(@) {
}
if (!$ux) { # do it in blocks for speed if we can
my($nbytes,$buff);
- while (($nbytes=$msg->read($buff,16384)) > 0)
+ while (($nbytes=$msg->read($buff,65536)) > 0)
{ $mp->print($buff) or die "Can't write to $mbxname: $!" }
defined $nbytes or die "Error reading: $!";
} else { # for UNIX-style mailbox delivery: escape 'From '
@@ -13763,10 +15645,9 @@ sub mail_to_local_mailbox(@) {
}
# must append an empty line for a Unix mailbox format
$mp->print($eol) or die "Can't write to $mbxname: $!" if $ux;
- };
- my($failed) = 0;
- if ($@ ne '') { # trouble
- chomp($@);
+ 1;
+ } or do { # trouble
+ $eval_stat = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
if ($ux && defined($pos)) {
$can_truncate or
do_log(-1, "Truncating a mailbox file will most likely fail");
@@ -13775,12 +15656,13 @@ sub mail_to_local_mailbox(@) {
$mp->truncate($pos) or die "Can't truncate file $mbxname: $!";
}
$failed = 1;
- }
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ };
unlock($mp) if $ux;
$mp->close or die "Error closing $mbxname: $!";
if (!$failed) {
$smtp_response = "250 2.6.0 Ok, delivered to $mbxname";
- } elsif ($@ eq "timed out") {
+ } elsif ($@ =~ /^timed out\b/) {
$smtp_response = "450 4.4.2 Local delivery to $mbxname timed out";
} elsif ($@ =~ /too many hops/i) {
$smtp_response = "554 5.4.6 Rejected delivery to mailbox $mbxname: $@";
@@ -13790,7 +15672,7 @@ sub mail_to_local_mailbox(@) {
}
} # end of block, 'last' within block brings us here
do_log(-1, "%s", $smtp_response) if $smtp_response !~ /^2/;
- $smtp_response .= ", id=" . am_id();
+ $smtp_response .= ", id=" . $msginfo->log_id;
$r->recip_smtp_response($smtp_response); $r->recip_done(2);
$r->recip_mbxname($mbxname) if $mbxname ne '' && $smtp_response =~ /^2/;
}
@@ -13813,7 +15695,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.0721';
+ $VERSION = '2.0911';
@ISA = qw(Exporter);
}
use Time::HiRes ();
@@ -13871,14 +15753,14 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
-use DBI;
+use DBI qw(:sql_types);
BEGIN {
- import Amavis::Conf qw(c cr ca);
+ import Amavis::Conf qw(:platform c cr ca);
import Amavis::Util qw(ll do_log);
import Amavis::Timing qw(section_time);
}
@@ -13942,11 +15824,18 @@ sub begin_work {
# the AutoCommit mode, you should no longer use the database handle.
# In other words, you should disconnect and reconnect again
$self->dbh or $self->connect_to_sql;
- my($stat) = eval { $self->dbh->begin_work(@_) };
- if (!$stat || $@ ne '') {
- chomp($@); do_log(-1,"sql begin transaction failed, ".
- "probably disconnected by server, reconnecting (%s)", $@);
- $self->disconnect_from_sql; $self->connect_to_sql;
+ my($stat); my($eval_stat);
+ eval {
+ $stat = $self->dbh->begin_work(@_); 1;
+ } or do {
+ $eval_stat = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ };
+ if ($eval_stat ne '' || !$stat) {
+ do_log(-1,"sql begin transaction failed, ".
+ "probably disconnected by server, reconnecting (%s)", $eval_stat);
+ $self->disconnect_from_sql;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ $self->connect_to_sql;
$self->dbh->begin_work(@_);
}
$self->in_transaction(1);
@@ -13972,12 +15861,16 @@ sub rollback {
my($self)=shift; do_log(5,"sql rollback");
$self->in_transaction(0);
$self->dbh or die "rollback: dbh not available";
- eval { $self->dbh->rollback(@_) };
- if ($@ ne '') {
- chomp($@); do_log(-1,"sql rollback error, reconnecting (%s)", $@);
- $self->disconnect_from_sql; $self->connect_to_sql;
+ eval {
+ $self->dbh->rollback(@_); 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ do_log(-1,"sql rollback error, reconnecting (%s)", $eval_stat);
+ $self->disconnect_from_sql;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ $self->connect_to_sql;
# $self->dbh->rollback(@_); # too late now, hopefully implied in disconnect
- }
+ };
};
sub last_insert_id {
@@ -14014,41 +15907,56 @@ sub execute {
$sth or die "sql: prepare failed: ".$DBI::errstr;
}
my($rv_err,$rv_str);
- eval { $sth->execute(@args); $rv_err = $sth->err; $rv_str = $sth->errstr };
- if ($@ ne '') {
- my($err) = $@; chomp($err);
+ eval {
+ for my $j (0..$#args) { # arg can be a scalar or [val,type] or [val,\%attr]
+ $sth->bind_param($j+1, !ref($args[$j]) ? $args[$j] : @{$args[$j]});
+ }
+ $sth->execute; $rv_err = $sth->err; $rv_str = $sth->errstr; 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
# man DBI: ->err code is typically an integer but you should not assume so
- # $DBI::errstr is normally already contained in $err
- my($msg) = sprintf("err=%s, %s, %s", $DBI::err,$DBI::state,$err);
+ # $DBI::errstr is normally already contained in $eval_stat
+ my($sqlerr) = $sth ? $sth->err : $DBI::err;
+ my($sqlstate) = $sth ? $sth->state : $DBI::state;
+ my($msg) = sprintf("err=%s, %s, %s", $sqlerr, $sqlstate, $eval_stat);
if (!$sth) {
die "sql execute (no handle): ".$msg;
- } elsif (! ($sth->err eq '2006' || $sth->err eq '2013' || # MySQL
- ($sth->err == -1 && $sth->state eq 'S1000')) ) { # PostgreSQL
+ } elsif (! ($sqlerr eq '2006' || $sqlerr eq '2013' || # MySQL
+ ($sqlerr == -1 && $sqlstate eq 'S1000') || # PostgreSQL 7
+ ($sqlerr == 7 && $sqlstate eq 'S8006')) ) { # PostgreSQL 8
+ # libpq-fe.h: ExecStatusType PGRES_FATAL_ERROR=7
die "sql exec: $msg\n";
} else { # Server has gone away; Lost connection to...
# MySQL: 2006, 2013; PostgreSQL: 7
if ($self->in_transaction) {
$self->disconnect_from_sql;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
die "sql execute failed within transaction, $msg";
} else { # try one more time
do_log(0,"NOTICE: reconnecting in response to: %s", $msg);
$self->disconnect_from_sql;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
$self->connect_to_sql;
$self->dbh or die "sql execute: reconnect failed";
do_log(4,"sql: preparing and executing (again): %s", $clause);
$sth = $self->dbh->prepare($clause); $self->sth($clause,$sth);
$sth or die "sql: prepare (reconnected) failed: ".$DBI::errstr;
undef $rv_err; undef $rv_str;
- eval { $sth->execute(@args); $rv_err=$sth->err; $rv_str=$sth->errstr };
- if ($@ ne '') {
- $err = $@; chomp($err);
- $msg = sprintf("err=%s, %s, %s", $DBI::err,$DBI::state,$err);
+ eval {
+ for my $j (0..$#args) { # a scalar or [val,type] or [val,\%attr]
+ $sth->bind_param($j+1, !ref($args[$j]) ? $args[$j] : @{$args[$j]});
+ }
+ $sth->execute; $rv_err = $sth->err; $rv_str = $sth->errstr; 1;
+ } or do {
+ $eval_stat = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ $msg = sprintf("err=%s, %s, %s", $DBI::err,$DBI::state,$eval_stat);
$self->disconnect_from_sql;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
die "sql execute failed again, $msg";
- }
- }
- }
- }
+ };
+ }
+ }
+ };
# $rv_err: undef indicates success, "" indicates an 'information',
# "0" indicates a 'warning', true indicates an error
do_log(2,"sql execute status: err=%s, errstr=%s",
@@ -14107,19 +16015,19 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
-use DBI;
use Encode; # Perl 5.8 UTF-8 support
+use DBI qw(:sql_types);
BEGIN {
import Amavis::Conf qw(:platform c cr ca
$QUARANTINEDIR $timestamp_fmt_mysql);
- import Amavis::rfc2821_2822_Tools qw(split_address iso8601_utc_timestamp);
- import Amavis::Util qw(ll do_log min max am_id snmp_count add_entropy
- untaint safe_decode safe_encode);
+ import Amavis::rfc2821_2822_Tools;
+ import Amavis::Util qw(ll do_log min max snmp_count add_entropy
+ untaint safe_decode safe_encode ccat_split ccat_maj);
import Amavis::Lookup qw(lookup);
import Amavis::Lookup::IP qw(lookup_ip_acl);
import Amavis::Out::SQL::Connection ();
@@ -14134,133 +16042,208 @@ sub DESTROY {
eval { do_log(5,"Amavis::Out::SQL::Log DESTROY called") };
}
-# find an existing e-mail address record or insert one, returning its id
+# find an existing e-mail address record or insert one, returning its id;
+# may return undef if 'sel_adr' or 'ins_adr' SQL clauses are not defined;
sub find_or_save_addr {
my($self,$addr) = @_;
my($id); my($existed) = 0; my($localpart,$domain);
- my($conn_h) = $self->{conn_h};
- my($sql_cl_r) = cr('sql_clause'); my($sel_adr) = $sql_cl_r->{'sel_adr'};
- my($naddr) = untaint($addr); # normalized addr (lowercased, max 255 ch...)
- if ($naddr ne '') {
+ my($naddr) = untaint($addr);
+ if ($naddr ne '') { # normalize address (lowercase, 7-bit, max 255 ch...)
($localpart,$domain) = split_address($naddr); $domain = lc($domain);
$localpart = lc($localpart) if !c('localpart_is_case_sensitive');
local($1);
$domain = $1 if $domain=~/^\@?(.*?)\.*\z/s; # chop leading @ and tr. dot
$naddr = $localpart.'@'.$domain;
$naddr = substr($naddr,0,255) if length($naddr) > 255;
- }
- my($a_ref,$a2_ref);
- $conn_h->begin_work_nontransaction; #(re)connect if not connected, autocommit
- $conn_h->execute($sel_adr,$naddr);
- if (defined($a_ref=$conn_h->fetchrow_arrayref($sel_adr))) { # exists?
- $id = $a_ref->[0]; $conn_h->finish($sel_adr);
- $existed = 1;
- } else { # does not exist, attempt to insert a new e-mail address record
- my($invdomain); # domain with reversed fields, chopped to 255 characters
- $invdomain = join('.', reverse split(/\./,$domain,-1));
- $invdomain = substr($invdomain,0,255) if length($invdomain) > 255;
- my($ins_adr) = $sql_cl_r->{'ins_adr'};
- $conn_h->begin_work_nontransaction; #(re)connect if not connected
- eval { $conn_h->execute($ins_adr,$naddr,$invdomain) };
- my($eval_stat) = $@; chomp($eval_stat);
- # INSERT may have failed because of race condition with other processes;
- # try the SELECT again, it will most likely succeed this time;
- # SELECT after INSERT also avoids the need for a working last_insert_id()
- $conn_h->begin_work_nontransaction; #(re)connect if not connected
- $conn_h->execute($sel_adr,$naddr); # try select again
- if ( defined($a2_ref=$conn_h->fetchrow_arrayref($sel_adr)) ) {
- $id = $a2_ref->[0]; $conn_h->finish($sel_adr);
- add_entropy($id);
- if ($eval_stat eq '') {
- do_log(5,"find_or_save_addr: record inserted, id=%s, %s",
- $id,$naddr);
- } else {
- $existed = 1;
- do_log(5,"find_or_save_addr: found on the next attempt, ".
- "id=%s, %s, (first attempt: %s)", $id,$naddr,$eval_stat);
- }
- } else { # still does not exist
- undef $id; undef $existed;
- die "find_or_save_addr: failed to insert addr $naddr: $eval_stat";
+ $naddr =~ s/[^\000-\177]/?/g; # avoid UTF-8 SQL troubles, only need 7 bits
+ }
+ my($conn_h) = $self->{conn_h}; my($sql_cl_r) = cr('sql_clause');
+ my($sel_adr) = $sql_cl_r->{'sel_adr'};
+ my($ins_adr) = $sql_cl_r->{'ins_adr'};
+ if (!defined($sel_adr) || $sel_adr eq '') {
+ # no way to query a database, behave as if no record was found
+ do_log(5,"find_or_save_addr: sel_adr query disabled, %s", $naddr);
+ } else {
+ $conn_h->begin_work_nontransaction; #(re)connect if necessary, autocommit
+ $conn_h->execute($sel_adr,$naddr);
+ my($a_ref,$a2_ref);
+ if (defined($a_ref=$conn_h->fetchrow_arrayref($sel_adr))) { # exists?
+ $id = $a_ref->[0]; $conn_h->finish($sel_adr);
+ $existed = 1;
+ } elsif (!defined($ins_adr) || $ins_adr eq '') {
+ # record does not exist, insertion is not allowed
+ do_log(5,"find_or_save_addr: ins_adr insertion disabled, %s", $naddr);
+ } else { # does not exist, attempt to insert a new e-mail address record
+ my($invdomain); # domain with reversed fields, chopped to 255 characters
+ $invdomain = join('.', reverse split(/\./,$domain,-1));
+ $invdomain = substr($invdomain,0,255) if length($invdomain) > 255;
+ $invdomain =~ s/[^\000-\177]/?/g; # only need 7 bits
+ $conn_h->begin_work_nontransaction; # (re)connect if not connected
+ my($eval_stat);
+ eval { $conn_h->execute($ins_adr,$naddr,$invdomain); 1 }
+ or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ # INSERT may have failed because of race condition with other processes;
+ # try the SELECT again, it will most likely succeed this time;
+ # SELECT after INSERT also avoids the need for a working last_insert_id()
+ $conn_h->begin_work_nontransaction; # (re)connect if not connected
+ $conn_h->execute($sel_adr,$naddr); # try select again
+ if ( defined($a2_ref=$conn_h->fetchrow_arrayref($sel_adr)) ) {
+ $id = $a2_ref->[0]; $conn_h->finish($sel_adr);
+ add_entropy($id);
+ if ($eval_stat eq '') {
+ do_log(5,"find_or_save_addr: record inserted, id=%s, %s",
+ $id,$naddr);
+ } else {
+ $existed = 1; chomp $eval_stat;
+ do_log(5,"find_or_save_addr: found on the next attempt, ".
+ "id=%s, %s, (first attempt: %s)", $id,$naddr,$eval_stat);
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ }
+ } else { # still does not exist
+ undef $id; undef $existed;
+ if ($eval_stat ne '') {
+ chomp $eval_stat;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ };
+ die "find_or_save_addr: failed to insert addr $naddr: $eval_stat";
+ }
}
}
($id, $existed);
}
-# find a penpals record which certifies that a local user sid really sent a
-# mail to recipient rid some time ago. Returns an interval time in seconds
+# find a penpals record which proves that a local user sid really sent a
+# mail to a recipient rid some time ago. Returns an interval time in seconds
# since the last such mail was sent by our local user to a specified recipient
-# (or undef if information is not available)
+# (or undef if information is not available). If @$message_id_list is a
+# nonempty list of Message-IDs as found in References haeder field, the query
+# also includes previous outgoing messages with a matching Message-ID but
+# possibly to recipients different from what the mail was originally sent to.
#
sub penpals_find {
- my($self, $sid,$rid,$now) = @_;
- my($send_time,$age,$ref_mail_id,$ref_subj); my($a_ref);
+ my($self, $sid,$rid,$message_id_list, $now) = @_;
+ my($a_ref,$found,$age,$send_time,$ref_mail_id,$ref_subj,$ref_mid,$ref_rid);
my($conn_h) = $self->{conn_h}; my($sql_cl_r) = cr('sql_clause');
my($sel_penpals) = $sql_cl_r->{'sel_penpals'};
- if (defined($sid) && defined($rid) && defined($sel_penpals)) {
+ my($sel_penpals_msgid) = $sql_cl_r->{'sel_penpals_msgid'};
+ if (defined($sel_penpals_msgid) && @$message_id_list && defined($sid)) {
+ # list of refs to Message-ID is nonempty, try reference or recipient match
+ my($n) = sprintf("%d",scalar(@$message_id_list)); # number of keys
+ my(@args) = ($sid,$rid); my(@pos_args); local($1);
+ my($sel_taint) = substr($sel_penpals_msgid,0,0); # taintedness
+ $sel_penpals_msgid =~
+ s{ ( %m | \? ) } # substitute %m for keys and ? for each arg
+ { push(@pos_args, $1 eq '%m' ? @$message_id_list : shift @args),
+ $1 eq '%m' ? join(',', ('?') x $n) : '?' }gxe;
+ # keep original clause taintedness
+ $sel_penpals_msgid = untaint($sel_penpals_msgid) . $sel_taint;
+ $_ = untaint($_) for @pos_args; # untaint arguments
+ do_log(4, "penpals: query args: %s", join(', ', at pos_args));
+ do_log(4, "penpals: %s", $sel_penpals_msgid);
+ $conn_h->begin_work_nontransaction; # (re)connect if not connected
+ $conn_h->execute($sel_penpals_msgid, at pos_args);
+ if (defined($a_ref=$conn_h->fetchrow_arrayref($sel_penpals_msgid))) {
+ ($send_time, $ref_mail_id, $ref_subj, $ref_mid, $ref_rid) = @$a_ref;
+ $found = 1; $conn_h->finish($sel_penpals_msgid);
+ if (ll(4)) {
+ my($rid_match) = defined $ref_rid && defined $rid && $rid eq $ref_rid;
+ my($mid_match) = grep { $ref_mid eq $_ } @$message_id_list;
+ my($t) = $mid_match && $rid_match ? 'MidRid' :
+ $mid_match && !defined($rid) ? 'MidNullRPath' :
+ $mid_match ? 'Mid' : $rid_match ? 'Rid' : 'none';
+ snmp_count('PenPalsHits'.$t);
+ do_log(4, "penpals: MATCH ON %s: %s", $t, join(", ",@$a_ref));
+ }
+ }
+ }
+ if (!$found && defined($sel_penpals) && defined($rid) && defined($sid)) {
+ # list of Message-ID references not given, try matching on recipient only
$conn_h->begin_work_nontransaction; # (re)connect if not connected
$conn_h->execute($sel_penpals,untaint($sid),untaint($rid));
if (defined($a_ref=$conn_h->fetchrow_arrayref($sel_penpals))) { # exists?
($send_time, $ref_mail_id, $ref_subj) = @$a_ref;
- $conn_h->finish($sel_penpals);
- }
- }
- if (!defined($send_time)) {
- do_log(4, "penpals: (%s,%s) not found", $sid,$rid);
+ $found = 1; $conn_h->finish($sel_penpals);
+ snmp_count('PenPalsHitsRid');
+ do_log(4, "penpals: MATCH ON RID %s", join(", ",@$a_ref));
+ }
+ }
+ snmp_count('PenPalsAttempts');
+ snmp_count('PenPalsHits') if $found;
+ if (!$found) {
+ ll(4) && do_log(4, "penpals: (%s,%s) not found%s", $sid,$rid,
+ !@$message_id_list ? '' : ' refs: '.join(", ",@$message_id_list));
} else {
$age = max(0, $now - $send_time);
- do_log(3, "penpals: (%s,%s) age %.2f days", $sid,$rid, $age/(24*60*60));
+ do_log(3, "penpals: (%s,%s) %s age %.2f days",
+ $sid,$rid, $ref_mail_id, $age/(24*60*60));
}
($age, $ref_mail_id, $ref_subj);
}
sub save_info_preliminary {
my($self, $conn,$msginfo) = @_;
- my($addr) = $msginfo->sender; my($mail_id) = $msginfo->mail_id;
+ my($mail_id) = $msginfo->mail_id;
+ my($sid,$existed); my($addr) = $msginfo->sender;
# find an existing e-mail address record for sender, or insert a new one
- my($sid,$existed) = $self->find_or_save_addr($addr);
- # there is perhaps 30-50% chance the sender address is already in the db
- snmp_count('SqlAddrSender');
- snmp_count('SqlAddrSenderHits') if $existed;
- $msginfo->sender_maddr_id($sid);
- do_log(4,"save_info_preliminary: %s, %s, %s",
- $sid, $addr, $existed ? 'exists' : 'new' );
+ ($sid,$existed) = $self->find_or_save_addr($addr); ## if $addr ne '';
+ if (defined $sid) {
+ $msginfo->sender_maddr_id($sid);
+ # there is perhaps 30-50% chance the sender address is already in the db
+ snmp_count('SqlAddrSender');
+ snmp_count('SqlAddrSenderHits') if $existed;
+ do_log(4,"save_info_preliminary: %s, %s, %s",
+ $sid, $addr, $existed ? 'exists' : 'new' );
+ }
# find existing address records for recipients, or insert them
for my $r (@{$msginfo->per_recip_data}) {
- my($addr) = $r->recip_addr;
- my($rid,$existed) = $self->find_or_save_addr($addr);
- # there is perhaps 90-100% chance the recipient addr is already in the db
- snmp_count('SqlAddrRecip');
- snmp_count('SqlAddrRecipHits') if $existed;
- $r->recip_maddr_id($rid);
- do_log(4,"save_info_preliminary %s, recip id: %s, %s, %s",
- $mail_id, $rid, $addr, $existed ? 'exists' : 'new' );
+ my($rid,$existed); my($addr) = $r->recip_addr;
+ ($rid,$existed) = $self->find_or_save_addr($addr) if $addr ne '';
+ if (defined $rid) {
+ $r->recip_maddr_id($rid);
+ # there is perhaps 90-100% chance the recipient addr is already in the db
+ snmp_count('SqlAddrRecip');
+ snmp_count('SqlAddrRecipHits') if $existed;
+ do_log(4,"save_info_preliminary %s, recip id: %s, %s, %s",
+ $mail_id, $rid, $addr, $existed ? 'exists' : 'new' );
+ }
}
my($conn_h) = $self->{conn_h}; my($sql_cl_r) = cr('sql_clause');
- $conn_h->begin_work; # SQL transaction starts
- eval {
- # MySQL does not like a standard iso8601 delimiter 'T' or a timezone,
- # when data type of msgs.time_iso is TIMESTAMP (instead of a string)
- my($time_iso) = $timestamp_fmt_mysql && $conn_h->driver_name eq 'mysql'
- ? iso8601_utc_timestamp($msginfo->rx_time,1,'')
- : iso8601_utc_timestamp($msginfo->rx_time);
- # insert a placeholder message record with sender information
- $conn_h->execute($sql_cl_r->{'ins_msg'},
- $msginfo->mail_id, $msginfo->secret_id, am_id(),
- $msginfo->rx_time, $time_iso,
- untaint($sid), c('policy_bank_path'), untaint($msginfo->client_addr),
- untaint($msginfo->msg_size), untaint(substr(c('myhostname'),0,255)) );
- $conn_h->commit;
- };
- if ($@ ne '') {
- my($err) = $@; chomp($err);
- if ($conn_h->in_transaction) {
- eval { $conn_h->rollback };
- do_log(1,"save_info_preliminary: rollback%s",
- $@ eq '' ? " done" : ": $@");
- }
- do_log(-1, "WARN save_info_preliminary: %s", $err);
- return 0;
+ my($ins_msg) = $sql_cl_r->{'ins_msg'};
+ if (!defined($ins_msg) || $ins_msg eq '') {
+ do_log(4,"save_info_preliminary: ins_msg undef, not saving");
+ } elsif (!defined($sid)) {
+ do_log(4,"save_info_preliminary: sid undef, not saving");
+ } else {
+ $conn_h->begin_work; # SQL transaction starts
+ eval {
+ # MySQL does not like a standard iso8601 delimiter 'T' or a timezone,
+ # when data type of msgs.time_iso is TIMESTAMP (instead of a string)
+ my($time_iso) = $timestamp_fmt_mysql && $conn_h->driver_name eq 'mysql'
+ ? iso8601_utc_timestamp($msginfo->rx_time,1,'')
+ : iso8601_utc_timestamp($msginfo->rx_time);
+ # insert a placeholder message record with sender information
+ $conn_h->execute($ins_msg,
+ $msginfo->mail_id, $msginfo->secret_id, $msginfo->log_id,
+ $msginfo->rx_time, $time_iso,
+ untaint($sid), c('policy_bank_path'), untaint($msginfo->client_addr),
+ 0+untaint($msginfo->msg_size), untaint(substr(c('myhostname'),0,255)));
+ $conn_h->commit; 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ if ($conn_h->in_transaction) {
+ eval {
+ $conn_h->rollback;
+ do_log(1,"save_info_preliminary: rollback done"); 1;
+ } or do {
+ $@ = "errno=$!" if $@ eq ''; chomp $@;
+ do_log(1,"save_info_preliminary: rollback %s", $@);
+ die $@ if $@ =~ /^timed out\b/; # resignal timeout
+ };
+ }
+ do_log(-1, "WARN save_info_preliminary: %s", $eval_stat);
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ return 0;
+ };
}
1;
}
@@ -14268,77 +16251,108 @@ sub save_info_final {
sub save_info_final {
my($self, $conn,$msginfo,$dsn_sent) = @_;
my($mail_id) = $msginfo->mail_id; my($spam_level) = $msginfo->spam_level;
- my($sql_cl_r) = cr('sql_clause'); my($ins_rcp) = $sql_cl_r->{'ins_rcp'};
- my($conn_h) = $self->{conn_h};
- $conn_h->begin_work; # SQL transaction starts
- eval {
- for my $r (@{$msginfo->per_recip_data}) {
- my($addr) = $r->recip_addr;
- my($dest,$resp) = ($r->recip_destiny, $r->recip_smtp_response);
- my($d) = $resp=~/^4/ ? 'TEMPFAIL'
- : ($dest==D_BOUNCE && $resp=~/^5/) ? 'BOUNCE'
- : ($dest!=D_BOUNCE && $resp=~/^5/) ? 'REJECT'
- : ($dest==D_PASS && ($resp=~/^2/ || !$r->recip_done)) ? 'PASS'
- : ($dest==D_DISCARD) ? 'DISCARD' : '?';
- # insert mail recipient record
- $conn_h->execute($ins_rcp,
- $mail_id, untaint($r->recip_maddr_id),
- # $msginfo->rx_time,
- substr($d,0,1), ' ',
- $r->recip_blacklisted_sender ? 'Y' : 'N',
- $r->recip_whitelisted_sender ? 'Y' : 'N',
- untaint(0+$spam_level + $r->recip_score_boost),
- untaint($resp) );
- };
- my($m_id) = $msginfo->orig_header_fields->{'message-id'};
- my($from) = $msginfo->orig_header_fields->{'from'};
- my($subj) = $msginfo->orig_header_fields->{'subject'};
- for ($m_id,$from,$subj) {
- local($1); chomp;
- s/\n([ \t])/$1/sg; s/^[ \t]+//s; s/[ \t]+\z//s; # unfold, trim
- if ($unicode_aware) {
- my($octets); # string of bytes (not logical chars), UTF8 encoded
- eval { $octets = safe_encode('utf8',safe_decode('MIME-Header',$_))};
- if ($@ eq '') { $_ = $octets }
- else { do_log(1,"save_info_final INFO: header field ".
- "not decodable, keeping raw bytes: %s", $@) }
- }
- $_ = substr($_,0,255) if length($_) > 255;
- }
- my($q_to) = $msginfo->quarantined_to; # a ref to a list of quar. locations
- if (!defined($q_to)) {}
- elsif (!@$q_to) { undef $q_to }
- else {
- $q_to = $q_to->[0]; # keep only the first quarantine location
- $q_to =~ s{^\Q$QUARANTINEDIR\E/}{}; # strip directory
- $q_to = substr($q_to,0,255) if length($q_to) > 255;
- }
- $q_to = defined $q_to ? untaint($q_to) : '';
- my($content_type) = $msginfo->setting_by_contents_category({
- CC_VIRUS,'V', CC_BANNED,'B', CC_SPAM,'S', CC_SPAMMY,'s',
- CC_BADH.",2",'M', CC_BADH,'H', CC_OVERSIZED,'O',
- CC_CLEAN,'C', CC_CATCHALL,'?'});
- my($quar_type) = $msginfo->quar_type;
- for ($quar_type,$content_type) { $_ = ' ' if !defined || /^ *\z/ }
- do_log(4,"save_info_final %s, %s, %s, %s, %s, %s, ".
- "Message-ID: %s, From: '%s', Subject: '%s'",
- $mail_id, $content_type, $quar_type, $q_to, $dsn_sent,
- $spam_level, $m_id, $from, $subj);
- # update message record with additional information
- $conn_h->execute($sql_cl_r->{'upd_msg'},
- $content_type, $quar_type, $q_to, $dsn_sent,
- untaint($spam_level), untaint($m_id), untaint($from),
- untaint($subj), $mail_id);
- $conn_h->commit;
- };
- if ($@ ne '') {
- my($err) = $@; chomp($err);
- if ($conn_h->in_transaction) {
- eval { $conn_h->rollback };
- do_log(1, "save_info_final: rollback%s", $@ eq '' ? " done" : ": $@" );
- }
- do_log(-1, "WARN save_info_final: %s", $err);
- return 0;
+ my($conn_h) = $self->{conn_h}; my($sql_cl_r) = cr('sql_clause');
+ my($ins_msg) = $sql_cl_r->{'ins_msg'};
+ my($upd_msg) = $sql_cl_r->{'upd_msg'};
+ my($ins_rcp) = $sql_cl_r->{'ins_rcp'};
+ my($sid) = $msginfo->sender_maddr_id;
+ if ($ins_msg eq '' || $upd_msg eq '' || $ins_rcp eq '') {
+ # updates disabled
+ } elsif (!defined($sid)) {
+ # sender not in table maddr, msgs record was not inserted by preliminary
+ } else {
+ $conn_h->begin_work; # SQL transaction starts
+ eval {
+ # insert per-recipient records into table msgrcpt
+ for my $r (@{$msginfo->per_recip_data}) {
+ my($rid) = $r->recip_maddr_id;
+ next if !defined $rid; # e.g. always_bcc, or table 'maddr' is disabled
+ my($dest,$resp) = ($r->recip_destiny, $r->recip_smtp_response);
+ my($d) = $resp=~/^4/ ? 'TEMPFAIL'
+ : ($dest==D_BOUNCE && $resp=~/^5/) ? 'BOUNCE'
+ : ($dest!=D_BOUNCE && $resp=~/^5/) ? 'REJECT'
+ : ($dest==D_PASS && ($resp=~/^2/ || !$r->recip_done)) ? 'PASS'
+ : ($dest==D_DISCARD) ? 'DISCARD' : '?';
+ $resp = substr($resp,0,255) if length($resp) > 255;
+ $resp =~ s/[^\000-\177]/?/gs; # just in case, only need 7 bits
+ $conn_h->execute($ins_rcp,
+ $mail_id, untaint($rid),
+ # $msginfo->rx_time,
+ substr($d,0,1), ' ',
+ $r->recip_blacklisted_sender ? 'Y' : 'N',
+ $r->recip_whitelisted_sender ? 'Y' : 'N',
+ 0+untaint($spam_level+$r->recip_score_boost), untaint($resp) );
+ }
+ my($q_to) = $msginfo->quarantined_to; # ref to a list of quar. locations
+ if (!defined($q_to) || !@$q_to) { undef $q_to }
+ else {
+ $q_to = $q_to->[0]; # keep only the first quarantine location
+ $q_to =~ s{^\Q$QUARANTINEDIR\E/}{}; # strip directory
+ }
+ my($m_id) = $msginfo->orig_header_fields->{'message-id'};
+ my($m_id) = parse_message_id($m_id) if $m_id ne ''; # strip CFWS, take #1
+ my($subj) = $msginfo->orig_header_fields->{'subject'};
+ my($from) = $msginfo->orig_header_fields->{'from'}; # raw full field
+ my($rfc2822_from) = $msginfo->rfc2822_from; # undef, scalar or listref
+ my($rfc2822_sender) = $msginfo->rfc2822_sender; # undef or scalar
+ $rfc2822_from = join(', ',@$rfc2822_from) if ref $rfc2822_from;
+ my($os_fp) = $msginfo->client_os_fingerprint;
+ $_ = !defined($_) ? '' :untaint($_) for ($m_id,$subj,$from,$q_to,$os_fp);
+ for ($m_id,$subj,$from) { # convert to UTF-8 octets, truncate to 255 char
+ local($1); chomp;
+ s/\n([ \t])/$1/sg; s/^[ \t]+//s; s/[ \t]+\z//s; # unfold, trim
+ if ($unicode_aware) {
+ my($octets); # string of bytes (not logical chars), UTF8 encoded
+ eval { $octets = safe_encode('utf8',safe_decode('MIME-Header',$_))};
+ if ($@ eq '') { $_ = $octets }
+ else {
+ chomp $@;
+ do_log(1,"save_info_final INFO: header field ".
+ "not decodable, keeping raw bytes: %s", $@);
+ die $@ if $@ =~ /^timed out\b/; # resignal timeout
+ }
+ }
+ if (length($_) > 255) # cleanly chop a UTF-8 byte sequence, RFC 3629
+ { $_ = $1 if /^ (.{0,255}) (?= [\x00-\x7F\xC0-\xFF] | \z )/xs }
+ }
+ for ($q_to,$os_fp) { # truncate to 255 chars, ensure 7-bit characters
+ $_ = substr($_,0,255) if length($_) > 255;
+ $_ =~ s/[^\000-\177]/?/gs; # only use 7 bits, compatible with UTF8
+ }
+ my($content_type) = $msginfo->setting_by_contents_category({
+ CC_VIRUS,'V', CC_BANNED,'B', CC_SPAM,'S', CC_SPAMMY,'s',
+ CC_BADH.",2",'M', CC_BADH,'H', CC_OVERSIZED,'O',
+ CC_CLEAN,'C', CC_CATCHALL,'?'});
+ my($quar_type) = $msginfo->quar_type;
+ for ($quar_type,$content_type) { $_ = ' ' if !defined || /^ *\z/ }
+ do_log(4,"save_info_final %s, %s, %s, %s, %s, %s, ".
+ "Message-ID: %s, From: '%s', Subject: '%s'",
+ $mail_id, $content_type, $quar_type, $q_to, $dsn_sent,
+ $spam_level, $m_id, $from, $subj);
+ # update message record with additional information
+ $conn_h->execute($upd_msg,
+ $content_type, $quar_type, $q_to, $dsn_sent,
+ 0+untaint($spam_level), $m_id, $from, $subj, # $os_fp,
+ $mail_id); # $rfc2822_sender, $rfc2822_from,
+ # SQL_CHAR, SQL_VARCHAR, SQL_VARBINARY, SQL_BLOB, SQL_INTEGER, SQL_FLOAT,
+ # SQL_TIMESTAMP, SQL_TYPE_TIMESTAMP_WITH_TIMEZONE, ...
+ $conn_h->commit; 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ if ($conn_h->in_transaction) {
+ eval {
+ $conn_h->rollback;
+ do_log(1,"save_info_final: rollback done"); 1;
+ } or do {
+ $@ = "errno=$!" if $@ eq ''; chomp $@;
+ do_log(1,"save_info_final: rollback %s", $@);
+ die $@ if $@ =~ /^timed out\b/; # resignal timeout
+ };
+ }
+ do_log(-1, "WARN save_info_final: %s", $eval_stat);
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
+ return 0;
+ }
}
1;
}
@@ -14358,11 +16372,12 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
use Errno qw(ENOENT EACCES EIO);
-use DBI;
+use DBI qw(:sql_types);
+# use DBD::Pg;
BEGIN {
import Amavis::Util qw(ll do_log untaint min max);
@@ -14376,15 +16391,25 @@ sub new {
sub open {
my($self) = shift; @$self{qw(conn_h clause dbkey mode maxbuf rx_time)} = @_;
- $self->{buf} = '';
+ my($conn_h) = $self->{conn_h}; $self->{buf} = '';
$self->{chunk_ind} = $self->{pos} = $self->{bufpos} = $self->{eof} = 0;
- if ($self->{mode} ne 'w') {
- eval { $self->{conn_h}->execute($self->{clause}, $self->{dbkey}) };
+ my($driver);
+ eval { $driver = $conn_h->driver_name; 1 }
+ or do { $@ = "errno=$!" if $@ eq ''; chomp $@ };
+ die $@ if $@ =~ /^timed out\b/; # resignal timeout
+ if ($self->{mode} eq 'w') {
+ ll(4) && do_log(4,"Amavis::IO::SQL::open %s drv=%s (%s); key=%s: %s",
+ $self->{mode}, $driver, $self->{clause}, $self->{dbkey});
+ } else {
+ eval { $conn_h->execute($self->{clause}, $self->{dbkey}); 1 }
+ or do { $@ = "errno=$!" if $@ eq '' };
my($ll) = $@ ne '' ? -1 : 4;
- ll($ll) && do_log($ll,"Amavis::IO::SQL::open (%s); key=%s: %s",
- $self->{clause}, $self->{dbkey}, $@);
+ ll($ll) && do_log($ll,"Amavis::IO::SQL::open %s drv=%s (%s); key=%s: %s",
+ $self->{mode}, $driver, $self->{clause}, $self->{dbkey}, $@);
if ($@ ne '') {
- chomp($@); die "Amavis::IO::SQL::open error: $@";
+ chomp $@;
+ if ($@ =~ /^timed out\b/) { die $@ } # resignal timeout
+ else { die "Amavis::IO::SQL::open $driver error: $@" }
$! = EIO; return undef; # not reached
}
}
@@ -14394,26 +16419,36 @@ sub DESTROY {
sub DESTROY {
my($self) = shift;
if (ref $self && $self->{conn_h}) {
- eval { $self->close or die "Error closing: $!" };
- if ($@ ne '') { warn "Amavis::IO::SQL::close error: $@" }
+ eval {
+ $self->close or die "Error closing: $!"; 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ warn "[$$] Amavis::IO::SQL::close error: $eval_stat";
+ };
delete $self->{conn_h};
}
}
sub close {
- my($self) = shift; $@ = undef;
+ my($self) = shift;
+ my($eval_stat);
eval {
if ($self->{mode} eq 'w') {
$self->flush or die "Can't flush: $!";
} elsif ($self->{conn_h} && $self->{clause} && !$self->{eof}) {
# reading, closing before eof was reached
$self->{conn_h}->finish($self->{clause}) or die "Can't finish: $!";
- }
+ };
+ 1;
+ } or do {
+ $eval_stat = $@ ne '' ? $@ : "errno=$!";
};
delete @$self{
qw(conn_h clause dbkey mode maxbuf rx_time buf chunk_ind pos bufpos eof) };
- if ($@ ne '') {
- chomp($@); die "Error closing, $@";
+ if ($eval_stat ne '') {
+ chomp $eval_stat;
+ if ($eval_stat =~ /^timed out\b/) { die $eval_stat } # resignal timeout
+ else { die "Error closing, $eval_stat" }
$! = EIO; return undef; # not reached
}
1;
@@ -14450,11 +16485,13 @@ sub read { # SCALAR,LENGTH,OFFSET
if (!defined($a_ref)) { $self->{eof} = 1 }
else { $self->{buf} .= $a_ref->[0]; $self->{chunk_ind}++ }
}
- };
- if ($@ ne '') {
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
# we can't stash an arbitrary error message string into $!,
# which forces us to use 'die' to properly report an error
- chomp($@); die "read: sql select failed, $@";
+ if ($eval_stat =~ /^timed out\b/) { die $eval_stat } # resignal timeout
+ else { die "read: sql select failed, $eval_stat" }
$! = EIO; return undef; # not reached
};
$_[0] = substr($self->{buf}, $self->{bufpos}, $req_len);
@@ -14480,9 +16517,11 @@ sub getline {
if (!defined($a_ref)) { $self->{eof} = 1 }
else { $self->{buf} .= $a_ref->[0]; $self->{chunk_ind}++ }
}
- };
- if ($@ ne '') {
- chomp($@); die "getline: reading sql select results failed, $@";
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ if ($eval_stat =~ /^timed out\b/) { die $eval_stat } # resignal timeout
+ else { die "getline: reading sql select results failed, $eval_stat" }
$! = EIO; return undef; # not reached
};
if ($ind < 0 && $self->{eof}) # imply a NL before eof if missing
@@ -14493,7 +16532,7 @@ sub getline {
my($nbytes) = length($line);
$self->{bufpos} += $nbytes; $self->{pos} += $nbytes;
if ($self->{bufpos} > 0 && $self->{chunk_ind} > 1) {
- # discard used-up part of the buf unless at ch.1, which may still be useful
+ # discard used part of the buf unless at ch.1, which may still be useful
ll(5) && do_log(5,"getline: moving on by %d chars", $self->{bufpos});
$self->{buf} = substr($self->{buf},$self->{bufpos}); $self->{bufpos} = 0;
}
@@ -14512,17 +16551,28 @@ sub flush {
$self->{dbkey}, $ind, $self->{rx_time},
min(length($self->{buf}),$self->{maxbuf}));
eval {
+ my($driver) = $conn_h->driver_name;
$conn_h->execute($self->{clause}, $self->{dbkey}, $ind,
# $self->{rx_time},
- untaint(substr($self->{buf},0,$self->{maxbuf})));
+ [ untaint(substr($self->{buf},0,$self->{maxbuf})),
+ $driver eq 'Pg' ? { pg_type => DBD::Pg::PG_BYTEA() }
+ : $driver eq 'mysql' ? SQL_BLOB : SQL_BLOB ] );
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ $msg = $eval_stat;
};
- if ($@ ne '') { $msg = $@; last }
+ last if defined $msg;
substr($self->{buf},0,$self->{maxbuf}) = ''; $self->{chunk_ind} = $ind;
}
- if (defined($msg)) {
- chomp($msg); $msg = "flush: sql inserting text failed, $msg";
- die $msg; # we can't stash an arbitrary error message string into $!,
- # which forces us to use 'die' to properly report an error
+ if (defined $msg) {
+ chomp $msg;
+ if ($msg =~ /^timed out\b/) { die $msg } # resignal timeout
+ else {
+ $msg = "flush: sql inserting text failed, $msg";
+ die $msg; # we can't stash an arbitrary error message string into $!,
+ # which forces us to use 'die' to properly report an error
+ }
$! = EIO; return undef; # not reached
}
1;
@@ -14540,14 +16590,19 @@ sub print {
ll(4) && do_log(4, "sql print: key: (%s, %d), size=%d",
$self->{dbkey}, $ind, $self->{maxbuf});
eval {
+ my($driver) = $conn_h->driver_name;
$conn_h->execute($self->{clause}, $self->{dbkey}, $ind,
# $self->{rx_time},
- untaint(substr($self->{buf},0,$self->{maxbuf})));
- };
- if ($@ ne '') {
+ [ untaint(substr($self->{buf},0,$self->{maxbuf})),
+ $driver eq 'Pg' ? { pg_type => DBD::Pg::PG_BYTEA() }
+ : $driver eq 'mysql' ? SQL_BLOB : SQL_BLOB ] );
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
# we can't stash an arbitrary error message string into $!,
# which forces us to use 'die' to properly report an error
- chomp($@); die "print: sql inserting mail text failed, $@";
+ if ($eval_stat =~ /^timed out\b/) { die $eval_stat } # resignal timeout
+ else { die "print: sql inserting mail text failed, $eval_stat" }
$! = EIO; return undef; # not reached
};
substr($self->{buf},0,$self->{maxbuf}) = ''; $self->{chunk_ind} = $ind;
@@ -14570,19 +16625,19 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT = qw(&mail_via_sql);
}
use subs @EXPORT;
-use DBI;
+use DBI qw(:sql_types);
use IO::Wrap;
BEGIN {
import Amavis::Conf qw(:platform c cr ca $sql_quarantine_chunksize_max);
import Amavis::rfc2821_2822_Tools qw(qquote_rfc2821_local);
- import Amavis::Util qw(ll do_log am_id snmp_count);
+ import Amavis::Util qw(ll do_log snmp_count);
import Amavis::Timing qw(section_time);
import Amavis::Out::SQL::Connection ();
}
@@ -14606,6 +16661,7 @@ sub mail_via_sql {
$msg = IO::Wrap::wraphandle($msg); # ensure we have an IO::Handle-like obj
$msg->seek(0,0) or die "Can't rewind mail file: $!";
}
+ my($err) = $@; my($smtp_response);
eval {
my($sql_cl_r) = cr('sql_clause');
$conn_h->begin_work; # SQL transaction starts
@@ -14625,38 +16681,46 @@ sub mail_via_sql {
$msg->print_body($mp);
} else {
my($nbytes,$buff);
- while (($nbytes=$msg->read($buff,16384)) > 0)
+ while (($nbytes=$msg->read($buff,65536)) > 0)
{ $mp->print($buff) or die "Can't write to SQL storage: $!" }
defined $nbytes or die "Error reading: $!";
}
$mp->close or die "Error closing Amavis::IO::SQL object: $!";
- $conn_h->commit;
- };
- if ($@ ne '') {
- my($msg) = $@; chomp($msg);
+ $conn_h->commit; 1;
+ } or do {
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err; my($msg) = $err;
$msg = "writing mail text to SQL failed: $msg"; do_log(0,"%s",$msg);
if ($conn_h->in_transaction) {
- eval { $conn_h->rollback };
- do_log(1, "mail_via_sql: rollback%s", $@ eq '' ? " done" : ": $@" );
- }
+ eval {
+ $conn_h->rollback;
+ do_log(1,"mail_via_sql: rollback done"); 1;
+ } or do {
+ $@ = "errno=$!" if $@ eq ''; chomp $@;
+ do_log(1,"mail_via_sql: rollback %s", $@);
+ die $@ if $@ =~ /^timed out\b/; # resignal timeout
+ };
+ }
+ die $err if $err =~ /^timed out\b/; # resignal timeout
die $msg;
- }
- };
- my($err) = $@; my($smtp_response);
+ };
+ 1;
+ } or do { $err = $@ ne '' ? $@ : "errno=$!" };
if ($err eq '') {
$smtp_response = "250 2.6.0 Ok, Stored to sql db as mail_id $mail_id";
snmp_count('OutMsgsDelivers');
} else {
- chomp($err);
+ chomp $err;
if ($err =~ /too many hops/i) {
$smtp_response = "554 5.4.6 Reject: $err";
snmp_count('OutMsgsRejects');
} else {
- $smtp_response = "451 4.5.0 Storing to sql db as mail_id $mail_id failed: $err";
+ $smtp_response =
+ "451 4.5.0 Storing to sql db as mail_id $mail_id failed: $err";
snmp_count('OutAttemptFails');
}
- }
- $smtp_response .= ", id=" . am_id();
+ die $err if $err =~ /^timed out\b/; # resignal timeout
+ }
+ $smtp_response .= ", id=" . $msginfo->log_id;
for my $r (@per_recip_data) {
next if $r->recip_done;
$r->recip_smtp_response($smtp_response); $r->recip_done(2);
@@ -14679,7 +16743,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
@@ -14695,9 +16759,11 @@ use vars @EXPORT;
BEGIN {
import Amavis::Conf qw(:platform :confvars c cr ca);
- import Amavis::Util qw(ll untaint min max do_log am_id rmdir_recursively
- exit_status_str proc_status_ok kill_proc run_command
- prolong_timer);
+ import Amavis::Util qw(ll untaint min max do_log rmdir_recursively
+ prolong_timer exit_status_str proc_status_ok
+ run_command run_as_subprocess
+ collect_results collect_results_structured);
+ import Amavis::Lookup qw(lookup);
import Amavis::Timing qw(section_time);
}
@@ -14716,28 +16782,33 @@ sub clamav_module_init($) {
ref $clamav_obj
or die "$av_name: Can't load db from $dbdir: $Mail::ClamAV::Error";
$clamav_obj->buildtrie;
- $clamav_obj->maxreclevel($MAXLEVELS) if $MAXLEVELS;
- $clamav_obj->maxfiles($MAXFILES);
+ $clamav_obj->maxreclevel($MAXLEVELS) if $MAXLEVELS > 0;
+ $clamav_obj->maxfiles($MAXFILES) if $MAXFILES > 0;
$clamav_obj->maxfilesize($MAX_EXPANSION_QUOTA || 30*1024*1024);
if ($clamav_version >= 0.12) {
$clamav_obj->maxratio($MAX_EXPANSION_FACTOR);
# $clamav_obj->archivememlim(0); # limit memory usage for bzip2 (0/1)
}
- do_log(2,"%s init", $av_name);
+ do_log(3,"clamav_module_init: %s init", $av_name);
section_time('clamav_module_init');
($clamav_obj,$clamav_version);
}
-# to be called from sub ask_clamav
+# to be called from sub ask_clamav, should not run as a subprocess
use vars qw($clamav_obj $clamav_version);
-sub clamav_module_internal($@) {
- my($query, $bare_fnames,$names_to_parts,$tempdir, $av_name) = @_;
+sub clamav_module_internal_pre($) {
+ my($av_name) = @_;
if (!defined $clamav_obj) {
($clamav_obj,$clamav_version) = clamav_module_init($av_name); # first time
} elsif ($clamav_obj->statchkdir) { # db reload needed?
do_log(2, "%s: reloading virus database", $av_name);
($clamav_obj,$clamav_version) = clamav_module_init($av_name);
}
+}
+
+# to be called from sub ask_clamav, may be called directly or in a subprocess
+sub clamav_module_internal($@) {
+ my($query, $bare_fnames,$names_to_parts,$tempdir, $av_name) = @_;
my($fname) = "$tempdir/parts/$query"; # file to be checked
my($part) = $names_to_parts->{$query}; # get corresponding parts object
my($options) = 0; # bitfield of options to Mail::ClamAV::scan
@@ -14768,7 +16839,15 @@ sub clamav_module_internal($@) {
# subroutine available for calling from @av_scanners list entries;
# it has the same args and returns as run_av() below
-sub ask_clamav { ask_av(\&clamav_module_internal, @_) }
+sub ask_clamav {
+ my($bare_fnames,$names_to_parts,$tempdir, $av_name) = @_;
+ clamav_module_internal_pre($av_name); # must not run as a subprocess
+# my(@results) = ask_av(\&clamav_module_internal, @_); # invoke directly
+ my($proc_fh,$pid) = run_as_subprocess(\&ask_av, \&clamav_module_internal, at _);
+ my($results_ref,$child_stat) =
+ collect_results_structured($proc_fh,$pid,$av_name,200*1024);
+ !$results_ref ? () : @$results_ref;
+}
my($savi_obj);
sub sophos_savi_init {
@@ -14788,7 +16867,7 @@ sub sophos_savi_init {
$av_name, $version->string, $version->major, $version->minor,
$version->count);
my($error);
- if ($MAXLEVELS) {
+ if ($MAXLEVELS > 0) {
$error = $savi_obj->set('MaxRecursionDepth', $MAXLEVELS);
!defined $error
or die "$av_name: error setting MaxRecursionDepth: err=$error";
@@ -14811,14 +16890,24 @@ sub sophos_savi_stale {
sub sophos_savi_reload {
if (defined $savi_obj) {
- my($status) = $savi_obj->load_data();
- !defined($status) or die "Failed to load SAVI virus data " .
- $savi_obj->error_string($status) . " ($status)";
+ do_log(3,"sophos_savi_reload: about to reload SAVI data");
+ eval {
+ my($status) = $savi_obj->load_data;
+ do_log(-1,"sophos_savi_reload: failed to load SAVI virus data %s (%s)",
+ $savi_obj->error_string($status), $status) if defined $status;
+ 1;
+ } or do {
+ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ do_log(-1,"sophos_savi_reload failed: %s", $eval_stat);
+ };
my($version) = $savi_obj->version;
- ref $version or die "Can't get SAVI version, err=$version";
- do_log(2,"Updated SAVI data: Version %s (engine %d.%d) ".
- "recognizing %d viruses", $version->string,
- $version->major, $version->minor, $version->count);
+ if (!ref($version)) {
+ do_log(-1,"sophos_savi_reload: Can't get SAVI version: %s", $version);
+ } else {
+ do_log(2,"Updated SAVI data: Version %s (engine %d.%d) ".
+ "recognizing %d viruses", $version->string,
+ $version->major, $version->minor, $version->count);
+ }
}
}
@@ -14866,9 +16955,16 @@ sub ask_sophos_savi {
$args = ["*"]; $sts_clean = [0]; $sts_infected = [1];
$how_to_get_names = qr/^(.*) FOUND$/;
}
- ask_av(\&sophos_savi_internal,
- $bare_fnames,$names_to_parts,$tempdir, $av_name,$command,$args,
- $sts_clean, $sts_infected, $how_to_get_names);
+ my($scan_status,$output,$virusnames) = ask_av(\&sophos_savi_internal,
+ $bare_fnames,$names_to_parts,$tempdir, $av_name,$command,$args,
+ $sts_clean, $sts_infected, $how_to_get_names);
+# my($proc_fh,$pid) = run_as_subprocess(\&ask_av, \&sophos_savi_internal,
+# $bare_fnames,$names_to_parts,$tempdir, $av_name,$command,$args,
+# $sts_clean, $sts_infected, $how_to_get_names);
+# my($results_ref,$child_stat) =
+# collect_results_structured($proc_fh,$pid,$av_name,200*1024);
+# my($scan_status,$output,$virusnames) = !$results_ref ? () : @$results_ref;
+ ($scan_status,$output,$virusnames);
}
@@ -14887,14 +16983,19 @@ sub ask_daemon_internal {
my($output) = ''; my($socketname,$is_inet);
if (!ref($sockets)) { $sockets = [ $sockets ] }
my($max_retries) = 2 * @$sockets; my($retries) = 0;
- $SIG{PIPE} = 'IGNORE'; # 'send' to broken pipe would throw a signal
# Sophie and Trophie can accept multiple requests per session
# and return a single line response each time
my($multisession) = $av_name =~ /^(Sophie|Trophie)/i ? 1 : 0;
- for (;;) { # gracefully handle cases when av child times out or restarts
+ my($remaining_time) = alarm(0); # check time left, stop the timer
+ my($deadline) = time + $remaining_time;
+ local $SIG{PIPE} = 'IGNORE'; # 'send' to a broken pipe would throw a signal
+ for (;;) { # gracefully handle cases when av process times out or restarts
+ # short timeout for connect and sending a request
+ alarm(10); do_log(5,"timer set to %d s (was %d s)", 10,$remaining_time);
@$sockets >= 1 or die "no sockets specified!?"; # sanity
$socketname = $sockets->[0]; # try the first one in the current list
$is_inet = $socketname =~ m{^/} ? 0 : 1; # simpleminded: unix vs. inet sock
+ my($eval_stat);
eval {
if (!$st_socket_created{$socketname}) {
ll(3) && do_log(3, "%s: Connecting to socket %s %s%s",
@@ -14918,6 +17019,8 @@ sub ask_daemon_internal {
# status/errno directly from 'send', not from 'getpeername':
defined send($st_sock{$socketname}, $query, 0)
or die "Can't send to socket $socketname: $!\n";
+ # normal timeout for reading a response
+ prolong_timer('ask_daemon_internal', int(0.8*($deadline-time)));
my($rv); my($buff) = ''; $! = 0;
while (defined($rv = $st_sock{$socketname}->recv($buff,8192,0))) {
$output .= $buff;
@@ -14933,18 +17036,21 @@ sub ask_daemon_internal {
}
$! = 0;
$output ne '' or die "Empty result from $socketname\n";
- };
- last if $@ eq '';
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ last if $eval_stat eq ''; # mission accomplished
# error handling (most interesting error codes are EPIPE and ENOTCONN)
- chomp($@); my($err) = "$!"; my($errn) = 0+$!;
+ chomp $eval_stat; my($err) = "$!"; my($errn) = 0+$!;
+ die "Exceeded allowed time" if time >= $deadline;
++$retries <= $max_retries
- or die "Too many retries to talk to $socketname ($@)";
+ or die "Too many retries to talk to $socketname ($eval_stat)";
# is ECONNREFUSED for INET sockets common enough too?
if ($retries <= 1 && $errn == EPIPE) { # common, don't cause concern
do_log(2,"%s broken pipe (don't worry), retrying (%d)",
$av_name,$retries);
} else {
- do_log( ($retries>1?-1:1), "%s: %s, retrying (%d)",$av_name,$@,$retries);
+ do_log( ($retries>1?-1:1),
+ "%s: %s, retrying (%d)", $av_name,$eval_stat,$retries);
if ($retries % @$sockets == 0) { # every time the list is exhausted
my($dly) = min(20, 1 + 5 * ($retries/@$sockets - 1));
do_log(3,"%s: sleeping for %s s", $av_name,$dly);
@@ -15032,7 +17138,7 @@ sub ask_av {
# Call a virus scanner and parse its output.
# Returns a triplet (or die in case of failure).
-# The first element of the triplet is interpreted as follows:
+# The first element of the triplet has the following semantics:
# - true if virus found,
# - 0 if no viruses found,
# - undef if it did not complete its job;
@@ -15053,9 +17159,20 @@ sub run_av {
) = @_;
my($scan_status,$virusnames,$error_str); my($output) = '';
&$pre_code(@_) if defined $pre_code;
+ my($remaining_time) = alarm(0); # check time left, stop the timer
+ my($deadline) = time + $remaining_time;
+ my($dt) = max(10, int(2 * $remaining_time / 3));
if (ref($command) eq 'CODE') {
do_log(3,"Using %s: (built-in interface)", $av_name);
- ($scan_status,$output,$virusnames) = &$command(@_);
+ alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat);
+ eval { ($scan_status,$output,$virusnames) = &$command(@_); 1 }
+ or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ prolong_timer('run_av', $deadline - time); # restart timer
+ if ($eval_stat ne '') {
+ chomp $eval_stat; $error_str = $eval_stat;
+ do_log(-1, "run_av (%s, built-in i/f): %s", $av_name,$eval_stat);
+ }
} else {
local($1); my(@args) = split(' ',$args);
if (grep { m{^({}/)?\*\z} } @args) { # {}/* or *, list each file
@@ -15067,30 +17184,22 @@ sub run_av {
# NOTE: RAV does not like '</dev/null' in its command!
ll(3) && do_log(3, "Using (%s): %s", $av_name, join(' ',$command, at args));
- my($remaining_time) = alarm(0); # check time left, stop the timer
- my($dt) = max(10, int(2 * $remaining_time / 3));
+ my($proc_fh,$pid); my($results_ref,$child_stat);
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
- my($proc_fh,$pid); my($child_stat);
+ my($eval_stat);
eval {
($proc_fh,$pid) = run_command(undef, "&1", $command, @args);
- my($nbytes,$buff);
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { $output .= $buff }
- defined $nbytes or die "Error reading: $!";
- my($err) = 0; $proc_fh->close or $err=$!; undef $proc_fh; undef $pid;
- $child_stat = $?; $error_str = exit_status_str($?,$err);
- };
- my($eval_stat) = $@;
- prolong_timer('run_av', $remaining_time-($dt-alarm(0))); # restart timer
+ ($results_ref,$child_stat) =
+ collect_results($proc_fh,$pid, $av_name,200*1024);
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ undef $proc_fh; undef $pid; $error_str = exit_status_str($child_stat,0);
+ prolong_timer('run_av', $deadline - time); # restart timer
if ($eval_stat ne '') {
- chomp($eval_stat);
- if (defined $pid) {
- do_log(-1, "%s is taking longer than %d s and will be killed",
- $command, $dt) if $eval_stat eq "timed out";
- kill_proc($pid,$command,1,$proc_fh); undef $pid;
- }
- do_log(-1, "run_av: %s", $eval_stat);
- $error_str = $eval_stat;
- }
+ chomp $eval_stat; $error_str = $eval_stat;
+ do_log(-1, "run_av (%s): %s", $av_name,$eval_stat);
+ }
+ if (defined $results_ref) { $output = $$results_ref; undef $results_ref }
chomp($output); my($output_trimmed) = $output;
$output_trimmed =~ s/\r\n/\n/gs;
$output_trimmed =~ s/([ \t\n\r])[ \t\n\r]{4,}/$1.../gs;
@@ -15139,8 +17248,8 @@ sub run_av {
($scan_status, $output, $virusnames);
}
-sub virus_scan($$$) {
- my($tempdir,$firsttime,$parts_root) = @_;
+sub virus_scan($$$$$) {
+ my($conn,$msginfo,$tempdir,$firsttime,$parts_root) = @_;
my($scan_status,$output, at virusname, at detecting_scanners);
my($anyone_done) = 0; my($anyone_tried) = 0;
my($bare_fnames_ref,$names_to_parts);
@@ -15160,6 +17269,7 @@ sub virus_scan($$$) {
do_log(2, "Not calling virus scanners, no files to scan in %s/parts",
$tempdir) if !@$bare_fnames_ref;
}
+ my($scanner_name) = $av->[0];
$anyone_tried = 1; my($this_status,$this_output,$this_vn);
if (!@$bare_fnames_ref) { # no files to scan?
($this_status,$this_output,$this_vn) = (0, '', []); # declare clean
@@ -15167,19 +17277,55 @@ sub virus_scan($$$) {
eval {
($this_status,$this_output,$this_vn) =
run_av($bare_fnames_ref,$names_to_parts,$tempdir, @$av);
- };
- if ($@ ne '') {
- my($err) = $@; chomp($err);
- $err = "$av->[0] av-scanner FAILED: $err";
+ 1;
+ } or do {
+ my($err) = $@ ne '' ? $@ : "errno=$!"; chomp $err;
+ $err = sprintf("%s av-scanner FAILED: %s", $scanner_name, $err);
do_log(-2,"%s",$err); push(@errors,$err);
$this_status = undef;
+ die $err if $err =~ /^timed out\b/; # resignal timeout
};
}
$anyone_done = 1 if defined $this_status;
$j++; section_time("AV-scan-$j");
- if ($this_status) { # virus detected by this scanner
- push(@detecting_scanners, $av->[0]);
+ if ($this_status && @$this_vn) {
+ # virus is reported by this scanner; is it for real, or is it just fraud?
+ my(@spam_hits); my($vnts) = ca('virus_name_to_spam_score_maps');
+ @spam_hits = # map each reported virus name to spam score or to undef
+ map { scalar(lookup(0,$_,@$vnts)) } @$this_vn if ref $vnts;
+ if (@spam_hits && !grep {!defined($_)} @spam_hits) { # all defined
+ # AV scanner did trigger, but all provided names are actually spam!
+ my($spam_level) = max(@spam_hits);
+ my($spam_status) = join(", ", map {"AV:$_=$spam_level"} @$this_vn);
+ my($spam_report) = $spam_status;
+ my($spam_summary) =
+ sprintf("AV scanner %s reported spam (not infection):\n%s\n",
+ $scanner_name, join(",",@$this_vn));
+ do_log(2,"Turning AV infection into a spam report: score=%s, %s",
+ $spam_level, $spam_status);
+ if (defined($msginfo->spam_level) || defined($msginfo->spam_status) ||
+ defined($msginfo->spam_report) || defined($msginfo->spam_summary)){
+ do_log(3,"adding AV/spam score %s to existing %s from an earlier ".
+ "(cached) spam check", $spam_level, $msginfo->spam_level);
+ $spam_level += $msginfo->spam_level if defined $msginfo->spam_level;
+ $spam_status = $msginfo->spam_status . ', ' . $spam_status
+ if $msginfo->spam_status ne '';
+ $spam_report = $msginfo->spam_report . ', ' . $spam_report
+ if $msginfo->spam_report ne '';
+ $spam_summary = $msginfo->spam_summary . "\n\n" . $spam_summary
+ if $msginfo->spam_summary ne '';
+ }
+ $msginfo->spam_level($spam_level);
+ $msginfo->spam_status($spam_status);
+ $msginfo->spam_report($spam_report);
+ $msginfo->spam_summary($spam_summary);
+ $this_status = 0; @$this_vn = (); # TURN OFF ALERT for this AV scanner!
+ }
+ }
+ if ($this_status) { # virus detected by this scanner, really!
+ push(@detecting_scanners, $scanner_name);
if (!@virusname) { # store results of the first scanner detecting
+ # @virusname = map { sprintf("[%s] %s",$scanner_name,$_) } @$this_vn;
@virusname = @$this_vn;
$scan_status = $this_status; $output = $this_output;
}
@@ -15275,7 +17421,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
@@ -15484,6 +17630,98 @@ 1;
__DATA__
#
+package Amavis::SpamControl::SpamAssassin::FileHandle;
+
+# Provides a virtual file (a filehandle tie - a TIEHANDLE) representing
+# a view to a mail message (accessed on an open file handle) prefixed by
+# a couple of synthesized mail header fields supplied as an array of lines.
+
+use strict;
+use re 'taint';
+no warnings 'uninitialized';
+use warnings FATAL => 'utf8';
+
+ sub new { shift->TIEHANDLE(@_) }
+
+ sub TIEHANDLE {
+ my($class) = shift;
+ my($self) = bless { 'fileno' => undef }, $class;
+ if (@_) { $self->OPEN(@_) or return undef }
+ $self;
+ }
+
+ sub UNTIE {
+ my($self,$count) = @_;
+ $self->CLOSE if !$count && defined $self->FILENO;
+ 1;
+ }
+
+ sub DESTROY {
+ my($self) = @_;
+ $self->CLOSE if defined $self->FILENO;
+ 1;
+ }
+
+ sub BINMODE { 1 }
+ sub FILENO { my($self) = @_; $self->{'fileno'} }
+ sub CLOSE { my($self) = @_; undef $self->{'fileno'}; 1 }
+ sub EOF { my($self) = @_; defined $self->{'fileno'} ? $self->{'eof'} : 1 }
+
+ sub OPEN {
+ my($self, $filehandle,$prefix_lines_ref) = @_;
+ $self->CLOSE if defined $self->FILENO;
+ $self->{'fileno'} = 9999; $self->{'eof'} = 0; $self->{'rec_ind'} = 0;
+ $self->{'prefix'} = $prefix_lines_ref;
+ $self->{'prefix_l'} = !ref($prefix_lines_ref) ? 0 : @{$prefix_lines_ref};
+ $self->{'handle'} = $filehandle;
+ seek($filehandle, 0,0); # also provides a return value and errno
+ };
+
+ sub SEEK {
+ my($self,$offset,$whence) = @_;
+ $whence == 0 or die "Only absolute SEEK is supported on this file";
+ $offset == 0 or die "Only SEEK(0,0) is supported on this file";
+ $self->{'eof'} = 0; $self->{'rec_ind'} = 0;
+ seek($self->{'handle'}, 0,0); # also provides a return value and errno
+ }
+
+# sub TELL (not implemented)
+# Returns the current position in bytes for FILEHANDLE, or -1 on error.
+
+ sub READLINE {
+ my($self) = @_;
+ if ($self->{'eof'}) {
+ return undef;
+ } elsif (wantarray) {
+ my($rec_ind) = $self->{'rec_ind'}; $self->{'eof'} = 1;
+ $self->{'rec_ind'} = $self->{'prefix_l'};
+ if ($rec_ind >= $self->{'prefix_l'}) {
+ return readline($self->{'handle'});
+ } elsif ($rec_ind == 0) { # common case: get the whole thing
+ return ( @{$self->{'prefix'}}, readline($self->{'handle'}) );
+ } else {
+ return ( @{$self->{'prefix'}}[$rec_ind .. @{$self->{'prefix'}}-1 ],
+ readline($self->{'handle'}) );
+ }
+ } elsif ($self->{'rec_ind'} < $self->{'prefix_l'}) {
+ return $self->{'prefix'}->[$self->{'rec_ind'}++];
+ } else {
+ my $line = readline($self->{'handle'}); # must be a scalar context!
+ if (!defined($line)) { $self->{'eof'} = 1 }
+ else { $self->{'rec_ind'}++ }
+ return $line;
+ }
+ }
+
+# sub READ { (not implemented)
+# my($self,$len,$offset) = @_;
+# # add to $$bufref, set $len to number of characters read
+# 0;
+# }
+
+1;
+
+
package Amavis::SpamControl::SpamAssassin;
use strict;
use re 'taint';
@@ -15493,18 +17731,19 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '1.000';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
}
-use Errno qw(EAGAIN);
+use Errno qw(ENOENT EACCES EAGAIN);
use FileHandle;
use Mail::SpamAssassin;
BEGIN {
import Amavis::Conf qw(:platform :confvars :sa $daemon_user c cr ca);
- import Amavis::Util qw(ll do_log sanitize_str run_command
- prolong_timer min max add_entropy
- exit_status_str proc_status_ok kill_proc);
+ import Amavis::Util qw(ll do_log sanitize_str prolong_timer add_entropy
+ min max exit_status_str proc_status_ok kill_proc
+ run_command run_as_subprocess
+ collect_results collect_results_structured);
import Amavis::rfc2821_2822_Tools;
import Amavis::Timing qw(section_time);
import Amavis::Lookup qw(lookup);
@@ -15516,93 +17755,142 @@ sub getCommonSAModules {
sub getCommonSAModules {
my($self) = shift;
my(@modules) = qw(
+ Mail::SpamAssassin::Locker
Mail::SpamAssassin::Locker::Flock
Mail::SpamAssassin::Locker::UnixNFSSafe
Mail::SpamAssassin::DBBasedAddrList
+ Mail::SpamAssassin::PersistentAddrList
Mail::SpamAssassin::SQLBasedAddrList
- Mail::SpamAssassin::PersistentAddrList
+ Mail::SpamAssassin::BayesStore
+ Mail::SpamAssassin::BayesStore::DBM
+ Mail::SpamAssassin::BayesStore::SQL
Mail::SpamAssassin::PerMsgLearner
Mail::SpamAssassin::AutoWhitelist
- Mail::SpamAssassin::BayesStore::DBM
- Mail::SpamAssassin::BayesStore::SQL
- Mail::SpamAssassin::Plugin::Hashcash
- Mail::SpamAssassin::Plugin::RelayCountry
- Mail::SpamAssassin::Plugin::SPF
- Mail::SpamAssassin::Plugin::URIDNSBL
- DBD::mysql Sys::Hostname::Long
- Mail::SPF::Query Razor2::Client::Agent Net::CIDR::Lite
Net::DNS::RR::SOA Net::DNS::RR::NS Net::DNS::RR::MX
Net::DNS::RR::A Net::DNS::RR::AAAA Net::DNS::RR::PTR
- Net::DNS::RR::CNAME Net::DNS::RR::TXT Net::Ping
- );
- # ??? ArchiveIterator Reporter Data::Dumper Getopt::Long Sys::Syslog lib
- # Mail::SpamAssassin::BayesStore::SDBM
- @modules;
-}
-
-sub getSA2Modules {
- my($self) = shift;
- my(@modules) = qw(
- Mail::SpamAssassin::UnixLocker Mail::SpamAssassin::BayesStoreDBM
- Mail::SpamAssassin::SpamCopURI
+ Net::DNS::RR::CNAME Net::DNS::RR::TXT Net::DNS::RR::SPF
+ Net::CIDR::Lite
+ Sys::Hostname::Long DBD::mysql
URI URI::Escape URI::Heuristic URI::QueryParam URI::Split URI::URL
URI::WithBase URI::_foreign URI::_generic URI::_ldap URI::_login
URI::_query URI::_segment URI::_server URI::_userpass URI::data URI::ftp
URI::gopher URI::http URI::https URI::ldap URI::ldapi URI::ldaps
URI::mailto URI::mms URI::news URI::nntp URI::pop URI::rlogin URI::rsync
URI::rtsp URI::rtspu URI::sip URI::sips URI::snews URI::ssh URI::telnet
- URI::tn3270 URI::urn URI::urn::isbn URI::urn::oid
+ URI::tn3270 URI::urn URI::urn::oid
URI::file URI::file::Base URI::file::Unix URI::file::Win32
);
+ # ??? ArchiveIterator Reporter Data::Dumper Getopt::Long Sys::Syslog lib
+ # Net::Ping
@modules;
}
+sub getSA2Modules {
+ qw(Mail::SpamAssassin::UnixLocker Mail::SpamAssassin::BayesStoreDBM
+ Mail::SpamAssassin::SpamCopURI
+ );
+}
+
sub getSA31Modules {
- my($self) = shift;
- my(@modules) = qw(
- Mail::SpamAssassin::BayesStore::MySQL
- Mail::SpamAssassin::BayesStore::PgSQL
- Mail::SpamAssassin::Plugin::AutoLearnThreshold
- Mail::SpamAssassin::Plugin::ReplaceTags
- Mail::SpamAssassin::Plugin::MIMEHeader
- Mail::SpamAssassin::Plugin::AWL Mail::SpamAssassin::Plugin::DCC
- Mail::SpamAssassin::Plugin::Pyzor Mail::SpamAssassin::Plugin::Razor2
- Mail::SpamAssassin::Plugin::SpamCop Mail::SpamAssassin::Plugin::URIDetail
- Mail::SpamAssassin::Plugin::WhiteListSubject
- Mail::SpamAssassin::Plugin::DKIM
- Mail::SpamAssassin::Plugin::DomainKeys
- Mail::DKIM Mail::DKIM::Verifier
- Mail::DomainKeys Mail::DomainKeys::Message Mail::DomainKeys::Policy
- IP::Country::Fast
- Crypt::OpenSSL::RSA auto::Crypt::OpenSSL::RSA::new_public_key
- auto::Crypt::OpenSSL::RSA::new_public_key
- auto::Crypt::OpenSSL::RSA::new_key_from_parameters
- auto::Crypt::OpenSSL::RSA::get_key_parameters
- auto::Crypt::OpenSSL::RSA::import_random_seed
+ qw(Mail::SpamAssassin::BayesStore::MySQL
+ Mail::SpamAssassin::BayesStore::PgSQL);
+}
+
+sub getSA32Modules {
+ qw(Mail::SpamAssassin::BayesStore::MySQL
+ Mail::SpamAssassin::BayesStore::PgSQL
+ Mail::SpamAssassin::Bayes Mail::SpamAssassin::Bayes::CombineChi
+ Mail::SpamAssassin::Locales
+ Encode::Detect
);
- # BayesStore::SDBM Plugin::AntiVirus Plugin::TextCat
- # Mail::SpamAssassin::ImageInfo Mail::SpamAssassin::FuzzyOcr String::Approx
- # auto::Crypt::OpenSSL::RSA::_new auto::Crypt::OpenSSL::RSA::DESTROY
- # auto::Crypt::OpenSSL::RSA::load_public_key Digest::SHA Error
+# /var/db/spamassassin/compiled/.../Mail/SpamAssassin/CompiledRegexps/body_0.pm
+}
+
+sub getSAPlugins {
+ my($self, $sa_version_num) = @_;
+ my(@modules);
+ push(@modules, qw(Hashcash RelayCountry SPF URIDNSBL)) if $sa_version_num>=3;
+ push(@modules, qw(DKIM)) if $sa_version_num >= 3.001002;
+ if ($sa_version_num >= 3.001000) {
+ push(@modules, qw(
+ AWL AccessDB AntiVirus AutoLearnThreshold DCC MIMEHeader Pyzor Razor2
+ ReplaceTags SpamCop TextCat URIDetail WhiteListSubject));
+ # 'DomainKeys' plugin fell out of fashion with SA 3.2.0, don't load it
+ }
+ if ($sa_version_num >= 3.002000) {
+ push(@modules, qw(
+ BodyEval DNSEval HTMLEval HeaderEval MIMEEval RelayEval URIEval WLBLEval
+ ASN Bayes BodyRuleBaseExtractor Check HTTPSMismatch OneLineBodyRuleType
+ ImageInfo Rule2XSBody Shortcircuit VBounce));
+ }
+ $_ = 'Mail::SpamAssassin::Plugin::'.$_ for (@modules);
+ my(%mod_names) = map { ($_,1) } @modules;
+ # add supporting modules
+ push(@modules, qw(Razor2::Client::Agent))
+ if $mod_names{'Mail::SpamAssassin::Plugin::Razor2'};
+ push(@modules, qw(IP::Country::Fast))
+ if $mod_names{'Mail::SpamAssassin::Plugin::RelayCountry'};
+ push(@modules,
+ qw(Mail::DomainKeys Mail::DomainKeys::Message Mail::DomainKeys::Policy))
+ if $mod_names{'Mail::SpamAssassin::Plugin::DomainKeys'};
+ push(@modules, qw(Mail::DKIM Mail::DKIM::Verifier))
+ if $mod_names{'Mail::SpamAssassin::Plugin::DKIM'};
+ push(@modules, qw(Image::Info Image::Info::GIF Image::Info::JPEG
+ Image::Info::PNG Image::Info::TIFF))
+ if $mod_names{'Mail::SpamAssassin::Plugin::ImageInfo'};
+ if ($mod_names{'Mail::SpamAssassin::Plugin::SPF'}) {
+ if ($sa_version_num >= 3.002000) {
+ push(@modules, qw(
+ Mail::SPF Mail::SPF::Server Mail::SPF::Request
+ Mail::SPF::Mech Mail::SPF::Mech::A Mail::SPF::Mech::PTR
+ Mail::SPF::Mech::All Mail::SPF::Mech::Exists Mail::SPF::Mech::IP4
+ Mail::SPF::Mech::IP6 Mail::SPF::Mech::Include Mail::SPF::Mech::MX
+ Mail::SPF::Mod Mail::SPF::Mod::Exp Mail::SPF::Mod::Redirect
+ Mail::SPF::SenderIPAddrMech
+ Mail::SPF::v1::Record Mail::SPF::v2::Record
+ NetAddr::IP NetAddr::IP::Util
+ auto::NetAddr::IP::Util::inet_n2dx auto::NetAddr::IP::Util::ipv6_n2d));
+ }
+ # the 3.2.0 could support the old Mail::SPF::Query too
+ push(@modules, qw(Mail::SPF::Query)); # if $sa_version_num < 3.002000;
+ }
+ if ($mod_names{'Mail::SpamAssassin::Plugin::DomainKeys'} ||
+ $mod_names{'Mail::SpamAssassin::Plugin::DKIM'}) {
+ push(@modules, qw(
+ Crypt::OpenSSL::RSA
+ auto::Crypt::OpenSSL::RSA::new_public_key
+ auto::Crypt::OpenSSL::RSA::new_key_from_parameters
+ auto::Crypt::OpenSSL::RSA::get_key_parameters
+ auto::Crypt::OpenSSL::RSA::import_random_seed
+ Digest::SHA Error));
+ }
+# do_log(5, "getSAPlugins %s: %s", $sa_version_num, join(', ', at modules));
@modules;
}
sub loadSpamAssassinModules {
my($self) = shift;
- # must be loaded before chroot takes place
- my(@modules) = $self->getCommonSAModules;
- my($sa_version) = $self->sa_version; # could be 3.0.1, which is not numeric!
+ my(@modules); # modules to be loaded before chroot takes place
+ my($sa_version) = $self->sa_version; # turn '3.1.8-pre1' into 3.001008
+ my($sa_version_num); local($1,$2,$3);
+ $sa_version_num = sprintf("%d.%03d%03d", $1,$2,$3)
+ if $sa_version =~ /^(\d+)\.(\d+)(?:\.(\d+))/; # ignore trailing non-digits
+ push(@modules, $self->getCommonSAModules);
if (!defined($sa_version)) {
die "loadSpamAssassinModules: unknown version of Mail::SpamAssassin";
- } elsif ($sa_version=~/^(\d+(?:\.\d+)?)/ && $1 < 3) {
+ } elsif ($sa_version_num < 3) {
push(@modules, $self->getSA2Modules);
- } elsif ($sa_version=~/^(\d+(?:\.\d+)?)/ && $1 >= 3.1) {
+ } elsif ($sa_version_num >= 3.001 && $sa_version < 3.002) {
push(@modules, $self->getSA31Modules);
- }
+ } elsif ($sa_version_num >= 3.002) {
+ push(@modules, $self->getSA32Modules);
+ }
+ push(@modules, $self->getSAPlugins($sa_version_num));
my($missing) = Amavis::Boot::fetch_modules('PRE-COMPILE OPTIONAL MODULES', 0,
@modules) if @modules;
- do_log(2, 'INFO: no optional modules: '.join(' ',@$missing))
- if ref $missing && @$missing;
+ do_log(2, 'INFO: SA version: %s, %.6f, no optional modules: %s',
+ $sa_version, $sa_version_num, join(' ',@$missing))
+ if ref $missing && @$missing;
}
sub initializeSpamAssassin {
@@ -15611,6 +17899,9 @@ sub initializeSpamAssassin {
my($saved_umask) = umask;
local($1,$2,$3,$4,$5,$6); # avoid Perl bug, $1 gets tainted in compile_now
my($sa_version) = $self->sa_version;
+ my($sa_version_num); # turn '3.1.8-pre1' into 3.001008
+ $sa_version_num = sprintf("%d.%03d%03d", $1,$2,$3)
+ if $sa_version =~ /^(\d+)\.(\d+)(?:\.(\d+))/; # ignore trailing non-digits
my($sa_args) = {
debug => !$sa_debug ? undef : @sa_debug_fac ? \@sa_debug_fac : 'all',
save_pattern_hits => $sa_debug ne '' ? 1 : 0,
@@ -15624,19 +17915,24 @@ sub initializeSpamAssassin {
# LOCAL_RULES_DIR => '/usr/local/etc/mail/spamassassin',
#see man Mail::SpamAssassin for other options
};
- if (Mail::SpamAssassin->VERSION < 3.001005 &&
- !defined $sa_args->{LOCAL_STATE_DIR})
+ if ($sa_version_num < 3.001005 && !defined $sa_args->{LOCAL_STATE_DIR})
{ $sa_args->{LOCAL_STATE_DIR} = '/var/lib' } # don't ignore sa-update rules
my($spamassassin_obj) = Mail::SpamAssassin->new($sa_args);
# $Mail::SpamAssassin::DEBUG->{rbl}=-3;
# $Mail::SpamAssassin::DEBUG->{rulesrun}=4+64;
- if ($sa_auto_whitelist && $sa_version=~/^(\d+(?:\.\d+)?)/ && $1 < 3) {
+ if ($sa_auto_whitelist && $sa_version_num < 3) {
do_log(1, "SpamControl: turning on SA auto-whitelisting (AWL)");
# create a factory for the persistent address list
my($addrlstfactory) = Mail::SpamAssassin::DBBasedAddrList->new;
$spamassassin_obj->set_persistent_address_list_factory($addrlstfactory);
}
$spamassassin_obj->compile_now; # try to preloaded most modules
+ if ($spamassassin_obj->UNIVERSAL::can("get_loaded_plugins_list")) {
+ my(@plugins) = $spamassassin_obj->get_loaded_plugins_list;
+# printf STDERR ("%s\n", join(", ", at plugins));
+# not in use: AccessDB AntiVirus TextCat; ASN BodyRuleBaseExtractor
+# OneLineBodyRuleType Rule2XSBody Shortcircuit
+ }
alarm(0); # seems like SA forgets to clear alarm in some cases
umask($saved_umask); # restore our umask, SA clobbered it
$self->{'spamassassin_obj'} = $spamassassin_obj;
@@ -15671,27 +17967,125 @@ sub init_pre_fork {
$self->{'initialized_stage'} = 3;
}
+sub call_spamassassin($$$) {
+ my($self,$msginfo,$lines) = @_;
+ my($mail_obj,$per_msg_status);
+ my($which_section) = 'SA prepare';
+ my($data_representation) = 'GLOB'; # pass data as ARRAY or a GLOB to SA
+ my($saved_umask) = umask; my($saved_pid) = $$;
+ my($spamassassin_obj) = $self->{'spamassassin_obj'};
+ my($sa_version) = $self->sa_version;
+ my($sa_version_num); # turn '3.1.8-pre1' into 3.001008
+ $sa_version_num = sprintf("%d.%03d%03d", $1,$2,$3)
+ if $sa_version =~ /^(\d+)\.(\d+)(?:\.(\d+))/; # ignore trailing non-digits
+ my($spam_level,$sa_tests,$spam_report,$spam_summary,%supplementary_info);
+ my($fh) = $msginfo->mail_text;
+ $fh->seek(0,0) or die "Can't rewind mail file: $!";
+
+ if ($data_representation eq 'ARRAY') { # read mail into memory, horrors!
+ $which_section = 'SA msg read'; my($ln);
+ for ($! = 0; defined($ln=<$fh>); $! = 0) # header
+ { push(@$lines,$ln); last if $ln eq $eol }
+ defined $ln || $!==0 or die "Error reading mail header: $!";
+ for ($! = 0; defined($ln=<$fh>); $! = 0) # body
+ { push(@$lines,$ln) }
+ defined $ln || $!==0 or die "Error reading mail body: $!";
+ section_time($which_section);
+ }
+ my($eval_stat);
+ eval {
+ # my($recips) = $msginfo->recips;
+ # if (@$recips == 1) {
+ # do_log(3,"changing SA user to %s", $recips->[0]);
+ # # $spamassassin_obj->load_scoreonly_...
+ # $spamassassin_obj->signal_user_changed({username => $recips->[0]});
+ # }
+ local(*F);
+ if ($data_representation eq 'GLOB') { # pass mail as a GLOB to SpamAssassin
+ # present a virtual file to SA, an original mail file prefixed by @$lines
+ tie(*F,'Amavis::SpamControl::SpamAssassin::FileHandle');
+ open(F, $fh,$lines) or die "Can't open SA virtual file: $!";
+ binmode(F) or die "Can't set binmode on a SA virtual file: $!";
+ }
+ do_log(5,"calling SA parse, SA version %s, %.6f, data as %s",
+ $sa_version, $sa_version_num, $data_representation);
+ $which_section = 'SA parse';
+ my($data) = $data_representation eq 'ARRAY' ? $lines : \*F;
+ $mail_obj = $sa_version_num >= 3 ? $spamassassin_obj->parse($data)
+ : Mail::SpamAssassin::NoMailAudit->new(data=>$data, add_From_line=>0);
+ section_time($which_section);
+
+ $which_section = 'SA check';
+ do_log(4,"CALLING SA check");
+ { local($1,$2,$3,$4,$5,$6); # avoid Perl 5.8.x bug, $1 gets tainted
+ $per_msg_status = $spamassassin_obj->check($mail_obj);
+ }
+ section_time($which_section);
+
+ $which_section = 'SA collect';
+ { local($1,$2,$3,$4,$5,$6); # avoid Perl 5.8.0..5.8.3...? taint bug
+ if ($sa_version_num < 3) {
+ $spam_level = $per_msg_status->get_hits;
+ $sa_tests = $per_msg_status->get_names_of_tests_hit; # only names
+ } else {
+ $spam_level = $per_msg_status->get_score;
+ $sa_tests = $per_msg_status->get_tag('TESTSSCORES',',');
+ # $supplementary_info{'AUTOLEARN'}=$per_msg_status->get_autolearn_status;
+ for my $t (qw(AUTOLEARN AUTOLEARNSCORE LANGUAGES RELAYCOUNTRY
+ SC SCRULE SCTYPE)) {
+ $supplementary_info{$t} = $per_msg_status->get_tag($t);
+ }
+ }
+ $spam_summary = $per_msg_status->get_report; # taints $1 and $2 !
+ # $spam_summary = $per_msg_status->get_tag('SUMMARY');
+ $spam_report = $per_msg_status->get_tag('REPORT');
+ }
+ if ($data_representation eq 'GLOB') {
+ close(F) or die "Can't close SA virtual file: $!";
+ untie(*F);
+ }
+ # section_time($which_section); # don't bother reporting separately, short
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+
+ $which_section = 'SA finish';
+ if (defined $per_msg_status)
+ { $per_msg_status->finish; undef $per_msg_status }
+ if (defined $mail_obj && $sa_version_num >= 3)
+ { $mail_obj->finish; undef $mail_obj }
+ umask($saved_umask); # SA changes umask to 0077
+ if ($$ != $saved_pid) {
+ eval { do_log(-2,"PANIC, SA produced a clone process ".
+ "of [%s], TERMINATING CLONE [%s]", $saved_pid,$$) };
+ POSIX::_exit(8); # avoid END and destructor processing
+ }
+# section_time($which_section);
+ if ($eval_stat ne '') { chomp $eval_stat; die $eval_stat } # resignal
+ ($spam_level, $sa_tests, $spam_report, $spam_summary, \%supplementary_info);
+}
+
sub check {
my($self,$msginfo) = @_;
$self->{'initialized_stage'} == 3
or die "Wrong initialization sequence: " . $self->{'initialized_stage'};
- my($spamassassin_obj) = $self->{'spamassassin_obj'};
my($dspam_signature,$dspam_result,$dspam_fname);
my($which_section); my(@lines);
- my($spam_level,$sa_tests,$spam_report,$spam_summary,$autolearn_status);
+ my($spam_level,$sa_tests,$spam_report,$spam_summary,$supplementary_info_ref);
my($fh) = $msginfo->mail_text;
my($hdr_edits) = $msginfo->header_edits;
if (!$hdr_edits) {
$hdr_edits = Amavis::Out::EditHeader->new;
$msginfo->header_edits($hdr_edits);
}
- push(@lines, sprintf("Return-Path: %s\n", # fake a local delivery agent
- qquote_rfc2821_local($msginfo->sender)));
+ # fake a local delivery agent by inserting Return-Path
+ push(@lines, sprintf("Return-Path: %s\n", $msginfo->sender_smtp));
push(@lines, sprintf("X-Envelope-To: %s\n",
- join(",\n ",qquote_rfc2821_local(@{$msginfo->recips}))));
+ join(",\n ",qquote_rfc2821_local(@{$msginfo->recips}))));
my($os_fp) = $msginfo->client_os_fingerprint;
push(@lines, sprintf("X-Amavis-OS-Fingerprint: %s\n",
- sanitize_str($os_fp))) if $os_fp ne '';
+ sanitize_str($os_fp))) if $os_fp ne '';
+ push(@lines, sprintf("X-Amavis-AV-Status: %s\n",
+ sanitize_str($msginfo->spam_status))) if $msginfo->spam_status ne '';
my($pbpath) = c('policy_bank_path');
push(@lines, sprintf("X-Amavis-PolicyBank: %s\n",$pbpath)) if $pbpath ne '';
my($mbsl) = c('sa_mail_body_size_limit');
@@ -15702,7 +18096,7 @@ sub check {
do_log(1,"spam_scan: not wasting time on SA, ".
"message longer than %s bytes: %s+%s",
$mbsl, $msginfo->orig_header_size, $msginfo->orig_body_size);
- } else {
+ } else { # message not too large, do spam checking
if (!defined($dspam) || $dspam eq '') {
do_log(5,"spam_scan: DSPAM not available, skipping it");
} else {
@@ -15716,14 +18110,14 @@ sub check {
no warnings 'qw';
my($proc_fh,$pid) = run_command('&'.fileno($fh), "&1", $dspam,
qw(--stdout --deliver=spam,innocent
- --mode=tum --feature=chained,noise
+ --mode=tum --tokenizer=chained,noise
--enable-signature-headers
--user), $daemon_user,
); # --mode=teft
# qw(--stdout --deliver-spam) # dspam < 3.0
+ # use option --feature instead of --tokenizer with dspam < 3.8.0
# keep X-DSPAM-*, ignore other changes e.g. Content-Transfer-Encoding
- my($all_local) = !grep { !lookup(0,$_,@{ca('local_domains_maps')}) }
- @{$msginfo->recips};
+ my($all_local) = !grep {!$_->recip_is_local} @{$msginfo->per_recip_data};
my($first_line); my($ln);
my($allowed_hdrs) = cr('allowed_added_header_fields');
# scan mail header from DSPAM
@@ -15749,7 +18143,7 @@ sub check {
defined $ln || $!==0 || $!==EAGAIN
or die "Error reading from DSPAM process: $!";
my($nbytes,$buff);
- while (($nbytes=$proc_fh->read($buff,16384)) > 0) { #copy body from DSPAM
+ while (($nbytes=$proc_fh->read($buff,65536)) > 0) { #copy body from DSPAM
$dspam_fh->print($buff) or die "Can't write to $dspam_fname: $!";
}
defined $nbytes or die "Error reading: $!";
@@ -15761,107 +18155,60 @@ sub check {
do_log(4,"spam_scan: DSPAM gave: %s, %s",
$dspam_signature,$dspam_result);
section_time($which_section);
- }
-
- $which_section = 'SA msg read';
- # read mail into memory (horror!) in preparation for SpamAssasin
- $fh->seek(0,0) or die "Can't rewind mail file: $!";
- my($ln);
- for ($! = 0; defined($ln=<$fh>); $! = 0) # header
- { push(@lines,$ln); last if $ln eq $eol }
- defined $ln || $!==0 or die "Error reading mail header: $!";
- for ($! = 0; defined($ln=<$fh>); $! = 0) # body
- { push(@lines,$ln) }
- defined $ln || $!==0 or die "Error reading mail body: $!";
- section_time($which_section);
-
- my($per_msg_status);
- my($saved_umask) = umask; my($saved_pid) = $$;
+ } # end dspam section
+
my($start_time) = time; # SA may use timer for its own purposes, get time
my($remaining_time) = alarm(0); # check time left, stop the timer
+ $which_section = 'SA call';
+ my($proc_fh,$pid); my($eval_stat);
eval {
- $which_section = 'SA parse';
# NOTE ON TIMEOUTS: SpamAssassin may use timer for its own purpose,
# disabling it before returning. It seems it only uses timer when
# external tests are enabled.
local $SIG{ALRM} = sub {
my($s) = Carp::longmess("SA TIMED OUT, backtrace:");
# crop at some rather arbitrary limit
- if (length($s) > 900) { $s = substr($s,0,900-3) . "..." }
+ if (length($s) > 900) { $s = substr($s,0,900-3) . "[...]" }
do_log(-1,"%s",$s);
};
my($dt) = max(10, int(2 * $remaining_time / 3));
$dt = $sa_timeout if $sa_timeout > $dt;
alarm($dt);
do_log(5,"timer set to %d s for SA (was %d s)", $dt,$remaining_time);
- my($mail_obj); my($sa_version) = $self->sa_version;
- do_log(5,"calling SA parse, SA version %s", $sa_version);
- # *** note that $sa_version could be 3.0.1, which is not really numeric!
- if ($sa_version=~/^(\d+(?:\.\d+)?)/ && $1 >= 3) {
- $mail_obj = $spamassassin_obj->parse(\@lines);
- } else { # 2.64 or earlier
- $mail_obj = Mail::SpamAssassin::NoMailAudit->new(data => \@lines,
- add_From_line => 0);
- }
- section_time($which_section);
-
- $which_section = 'SA check';
- do_log(4,"CALLING SA check");
- { local($1,$2,$3,$4,$5,$6); # avoid Perl 5.8.0 bug, $1 gets tainted
- $per_msg_status = $spamassassin_obj->check($mail_obj);
- }
- section_time($which_section);
-
- $which_section = 'SA collect';
- { local($1,$2,$3,$4); # avoid Perl 5.8.0..5.8.3...? taint bug
- if ($sa_version=~/^(\d+(?:\.\d+)?)/ && $1 >= 3) {
- $spam_level = $per_msg_status->get_score;
- $sa_tests = $per_msg_status->get_tag('TESTSSCORES',',');
- $autolearn_status = $per_msg_status->get_autolearn_status;
- } else {
- $spam_level = $per_msg_status->get_hits;
- $sa_tests = $per_msg_status->get_names_of_tests_hit; # only names
- }
- $spam_summary = $per_msg_status->get_report; # taints $1 and $2 !
- # $spam_summary = $per_msg_status->get_tag('SUMMARY');
- $spam_report = $per_msg_status->get_tag('REPORT');
-
- #examples of obtaining aditional information from SA:
- # my($rly_trusted) = $per_msg_status->get_tag('RELAYSTRUSTED');
- # my($rly_untrusted) = $per_msg_status->get_tag('RELAYSUNTRUSTED');
- # my($rly_internal) = $per_msg_status->get_tag('RELAYSINTERNAL');
- # my($rly_external) = $per_msg_status->get_tag('RELAYSEXTERNAL');
- # my($rly_country) = $per_msg_status->get_tag('RELAYCOUNTRY');
- # $hdr_edits->add_header('X-TESTING',$rly_trusted);
- # $hdr_edits->add_header('X-Relay-Countries',$rly_country)
- # if $rly_country ne '' && $all_local &&
- # $allowed_hdrs && $allowed_hdrs->{lc('X-Relay-Countries')};
-
- #Experimental, never finished:
- # $per_msg_status->rewrite_mail;
- # my($entity) = nomailaudit_to_mime_entity($mail_obj);
- }
- };
- my($eval_stat) = $@;
- # section_time($which_section); # don't bother reporting separately, short
-
- $which_section = 'SA finish';
+ # note: array @lines at this point contains only prepended synthesized
+ # header fields, but will be extended in sub call_spamassassin() by
+ # reading-in the rest of the message; this may or may not happen in
+ # a separate process (called through run_as_subprocess or directly)
+ #
+ my(@results);
+ if (!$sa_spawned) {
+ @results = call_spamassassin($self,$msginfo,\@lines);
+ } else {
+ ($proc_fh,$pid) =
+ run_as_subprocess(\&call_spamassassin, $self,$msginfo,\@lines);
+ my($results_ref,$child_stat) =
+ collect_results_structured($proc_fh,$pid,'spawned SA',200*1024);
+ @results = @$results_ref if defined $results_ref;
+ }
+ ($spam_level,$sa_tests,$spam_report,$spam_summary,
+ $supplementary_info_ref) = @results;
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ section_time($which_section) if $sa_spawned;
+ $which_section = 'SA done';
prolong_timer('spam_scan_sa_finish',
max(20, $remaining_time - max(0,time-$start_time))); # restart the timer
- if (defined $per_msg_status)
- { $per_msg_status->finish; undef $per_msg_status }
- if ($$ != $saved_pid) {
- eval { do_log(-2,"PANIC, SA produced a clone process ".
- "of [%s], TERMINATING CLONE [%s]", $saved_pid,$$) };
- POSIX::_exit(1); # avoid END and destructor processing
- }
- umask($saved_umask); # SA changes umask to 0077
- section_time($which_section);
-
+ @lines = (); # release storage
if ($eval_stat ne '') { # SA timed out?
- chomp($eval_stat);
- die "$eval_stat\n" if $eval_stat ne "timed out";
- }
+ kill_proc($pid,'spawned SA',1,$proc_fh,$eval_stat) if defined $pid;
+ undef $proc_fh; undef $pid; chomp $eval_stat;
+ do_log(-1, "SA failed: %s", $eval_stat);
+ # die "$eval_stat\n" if $eval_stat !~ /timed out\b/;
+ }
+ # $hdr_edits->add_header('X-TESTING',$rly_trusted);
+ # $hdr_edits->add_header('X-Relay-Countries',$rly_country)
+ # if $rly_country ne '' && $all_local &&
+ # $allowed_hdrs && $allowed_hdrs->{lc('X-Relay-Countries')};
if (defined $dspam && $dspam ne '' && defined $spam_level) { # auto-learn
$which_section = 'DSPAM learn';
my($eat, at options);
@@ -15875,14 +18222,8 @@ sub check {
if (defined $eat && $dspam_signature ne '') {
do_log(2,"DSPAM learn %s (%s), %s", $eat,$spam_level,$dspam_signature);
my($proc_fh,$pid) = run_command($dspam_fname, "&1", $dspam, @options);
- # consume remaining output to avoid broken pipe
- my($nbytes,$buff);
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { }
- defined $nbytes or die "Error reading from DSPAM process: $!";
- my($err) = 0; $proc_fh->close or $err = $!; my($retval) = $?;
-# do_log(-1,"DSPAM learn %s response: %s",$eat,$output) if $output ne '';
- proc_status_ok($retval,$err)
- or die("DSPAM learn $eat FAILED: ".exit_status_str($retval,$err));
+ # consume subprocess output to avoid broken pipe
+ collect_results($proc_fh,$pid,"DSPAM learn $eat",16384,[0]);
}
section_time($which_section);
}
@@ -15896,30 +18237,26 @@ sub check {
add_entropy($spam_level,$sa_tests);
do_log(2,"OS_fingerprint: %s %s %s", $msginfo->client_addr,
defined $spam_level ? $spam_level : '-', $os_fp) if $os_fp ne '';
- do_log(3,"spam_scan: score=%s tests=[%s]", $spam_level,$sa_tests);
+ do_log(3,"spam_scan: score=%s autolearn=%s tests=[%s]",
+ $spam_level, $supplementary_info_ref->{'AUTOLEARN'}, $sa_tests);
+ if (defined($msginfo->spam_level) || defined($msginfo->spam_status) ||
+ defined($msginfo->spam_report) || defined($msginfo->spam_summary)) {
+ do_log(3,"adding SA score %s to existing %s from an earlier spam check",
+ $spam_level, $msginfo->spam_level);
+ $spam_level += $msginfo->spam_level if defined $msginfo->spam_level;
+ $sa_tests = $msginfo->spam_status . ', ' . $sa_tests
+ if $msginfo->spam_status ne '';
+ $spam_report = $msginfo->spam_report . ', ' . $spam_report
+ if $msginfo->spam_report ne '';
+ $spam_summary = $msginfo->spam_summary . "\n\n" . $spam_summary
+ if $msginfo->spam_summary ne '';
+ }
$msginfo->spam_level($spam_level); $msginfo->spam_status($sa_tests);
$msginfo->spam_report($spam_report); $msginfo->spam_summary($spam_summary);
- $msginfo->autolearn_status($autolearn_status);
+ for (keys %$supplementary_info_ref)
+ { $msginfo->supplementary_info($_, $supplementary_info_ref->{$_}) }
$spam_level;
}
-
-#sub nomailaudit_to_mime_entity($) {
-# my($mail_obj) = @_; # expect a Mail::SpamAssassin::MsgContainer object
-# my(@m_hdr) = $mail_obj->header; # in array context returns array of lines
-# my($m_body) = $mail_obj->body; # returns array ref
-# my($entity);
-# # make sure _our_ source line number is reported in case of failure
-# eval {$entity = MIME::Entity->build(
-# Type => 'text/plain', Encoding => '-SUGGEST',
-# Data => $m_body); 1} or do {chomp($@); die $@};
-# my($head) = $entity->head;
-# # insert header fields from template into MIME::Head entity
-# for my $hdr_line (@m_hdr) {
-# # make sure _our_ source line number is reported in case of failure
-# eval {$head->replace($fhead,$fbody); 1} or do {chomp($@); die $@};
-# }
-# $entity; # return the built MIME::Entity
-#}
1;
@@ -15934,7 +18271,7 @@ BEGIN {
BEGIN {
use Exporter ();
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);
- $VERSION = '2.072';
+ $VERSION = '2.091';
@ISA = qw(Exporter);
@EXPORT_OK = qw(&init &decompose_part &determine_file_types);
}
@@ -15943,13 +18280,15 @@ use IO::File qw(O_CREAT O_EXCL O_WRONLY)
use IO::File qw(O_CREAT O_EXCL O_WRONLY);
use File::Basename qw(basename);
use Convert::TNEF;
-use Convert::UUlib 1.05 qw(:constants); # avoid security bug in 1.04 and older
+ # avoid an exploitable security hole in Convert::UUlib 1.04 and older!
+use Convert::UUlib 1.05 qw(:constants); # 1.08 or newer is preferred!
use Compress::Zlib 1.35; # avoid security vulnerability in <= 1.34
-use Archive::Tar;
use Archive::Zip 1.14 qw(:CONSTANTS :ERROR_CODES);
BEGIN {
- import Amavis::Util qw(untaint min max ll do_log run_command
+ import Amavis::Util qw(untaint min max ll do_log
+ run_command run_command_consumer run_as_subprocess
+ collect_results collect_results_structured
exit_status_str proc_status_ok kill_proc snmp_count
prolong_timer rmdir_recursively add_entropy);
import Amavis::Conf qw(:platform :confvars $file c cr ca);
@@ -16049,8 +18388,8 @@ sub determine_file_types($$) {
chdir($tempdir) or die "Can't chdir to $tempdir: $!";
my($index)=0; my($ln);
for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) {
+ do_log(5, "result line from file(1): %s", $ln);
chomp($ln);
- do_log(5, "result line from file(1): %s", $ln);
if ($index > $#file_list) {
do_log(-1, "NOTICE: Skipping extra output from file(1): %s", $ln);
} else {
@@ -16078,13 +18417,15 @@ sub determine_file_types($$) {
}
defined $ln || $!==0 || $!==EAGAIN
or die "Error reading from file(1) utility: $!";
+ my($err) = 0; $proc_fh->close or $err = $!;
+ my(@errmsg);
+ proc_status_ok($?,$err)
+ or push(@errmsg, "failed, ".exit_status_str($?,$err));
if ($index < @part_list) {
- die sprintf("parsing file(1) results - missing last %d results",
- @part_list - $index);
- }
- my($err) = 0; $proc_fh->close or $err = $!;
- proc_status_ok($?,$err)
- or die "'file' utility ($file) failed, ".exit_status_str($?,$err);
+ push(@errmsg, sprintf("parsing failure - missing last %d results",
+ @part_list - $index));
+ }
+ !@errmsg or die "file(1) utility ($file) ".join(", ", at errmsg);
section_time(sprintf('get-file-type%d', scalar(@part_list)));
}
}
@@ -16097,7 +18438,7 @@ sub decompose_mail($$) {
# fetch all not-yet-visited part names, and start a new cycle
TIER:
while (@parts = @{$file_generator_object->parts_list}) {
- if ($MAXLEVELS && $depth > $MAXLEVELS) {
+ if ($MAXLEVELS > 0 && $depth > $MAXLEVELS) {
$hold = "Maximum decoding depth ($MAXLEVELS) exceeded";
last;
}
@@ -16158,40 +18499,39 @@ sub decompose_part($$) {
# 0 - truly atomic, or unknown or archiver failure; consider atomic
# 1 - some archive, successfully unpacked, result replaces original
# 2 - probably unpacked, but keep the original (eg self-extracting archive)
- my($hold,$none_called);
- my($sts) = eval {
+ my($hold); my($eval_stat); my($sts) = 0; my($any_called) = 0;
+ eval {
my($type_short) = $part->type_short;
my(@ts) = !defined $type_short ? ()
: !ref $type_short ? ($type_short) : @$type_short;
- return 0 if !@ts; # consider atomic if unknown (returns from eval)
- snmp_count("OpsDecType-".join('.', at ts));
- for my $dec_tuple (@{ca('decoders')}) { # first matching decoder wins
- next if !defined $dec_tuple;
- my($dec_ts,$code, at args) = @$dec_tuple;
- if ($code && grep {$_ eq $dec_ts} @ts)
- { return &$code($part,$tempdir, at args) } # returns from eval
- }
- # falling through (e.g. HTML) - no match, consider atomic
- $none_called = 1;
- return 0; # returns from eval
+ if (@ts) { # when one or more short types are known
+ snmp_count("OpsDecType-".join('.', at ts));
+ for my $dec_tuple (@{ca('decoders')}) { # first matching decoder wins
+ next if !defined $dec_tuple;
+ my($dec_ts,$code, at args) = @$dec_tuple;
+ if ($code && grep {$_ eq $dec_ts} @ts)
+ { $any_called = 1; $sts = &$code($part,$tempdir, at args); last }
+ }
+ }
+ 1;
+ } or do {
+ $eval_stat = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
+ if ($eval_stat =~ /^Exceeded storage quota/ ||
+ $eval_stat =~ /^Maximum number of files\b.*\bexceeded/) {
+ $hold = $eval_stat;
+ } else {
+ do_log(-1,"Decoding of %s (%s) failed, leaving it unpacked: %s",
+ $part->base_name, $part->type_long, $eval_stat);
+ }
+ $sts = 2; # keep the original, along with possible decoded files
+ chdir($tempdir) or die "Can't chdir to $tempdir: $!"; # just in case
};
- if ($@ ne '') {
- chomp($@);
- if ($@ =~ /^Exceeded storage quota/ ||
- $@ =~ /^Maximum number of files\b.*\bexceeded/) { $hold = $@ }
- else {
- do_log(-1,"Decoding of %s (%s) failed, leaving it unpacked: %s",
- $part->base_name, $part->type_long, $@);
- }
- $sts = 2;
- chdir($tempdir) or die "Can't chdir to $tempdir: $!"; # just in case
- }
if ($sts == 1 && lookup(0,$part->type_long, @keep_decoded_original_maps)) {
# don't trust this file type or unpacker,
# keep both the original and the unpacked file
ll(4) && do_log(4,"file type is %s, retain original %s",
$part->type_long, $part->base_name);
- $sts = 2;
+ $sts = 2; # keep the original, along with possible decoded files
}
if ($sts == 1) {
ll(5) && do_log(5,"decompose_part: deleting %s", $part->full_name);
@@ -16200,7 +18540,8 @@ sub decompose_part($$) {
}
ll(4) && do_log(4,"decompose_part: %s - %s", $part->base_name,
['atomic','archive, unpacked','source retained']->[$sts]);
- section_time('decompose_part') unless $none_called;
+ section_time('decompose_part') if $any_called;
+ die $eval_stat if $eval_stat =~ /^timed out\b/; # resignal timeout
$hold;
}
@@ -16234,7 +18575,11 @@ sub do_ascii($$) {
local($SIG{ALRM}); my($sigset,$action,$oldaction);
if ($] < 5.008) { # in old Perl signals could be delivered at any time
$SIG{ALRM} = sub { die "timed out\n" };
- } else { # Perl >= 5.8.0 has 'safe signals'
+ } elsif ($] < 5.008001) { # Perl 5.8.0
+ # 5.8.0 does not have POSIX::SigAction::safe but uses safe signals, which
+ # means a runaway uulib can't be aborted; tough luck, upgrade your Perl!
+ $SIG{ALRM} = sub { die "timed out\n" }; # old way, but won't abort
+ } else { # Perl >= 5.8.0 has 'safe signals', and SigAction::safe available
# POSIX::sigaction can bypass safe Perl signals on request;
# alternatively, use Perl module Sys::SigAction
$sigset = POSIX::SigSet->new(SIGALRM); $oldaction = POSIX::SigAction->new;
@@ -16245,6 +18590,7 @@ sub do_ascii($$) {
or die "Can't set ALRM handler: $!";
do_log(4,"do_ascii: Setting sigaction handler, was %d", $oldaction->safe);
}
+ my($eval_stat);
eval { # must not go away without calling Convert::UUlib::CleanUp !
my($sts,$count);
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
@@ -16332,9 +18678,9 @@ sub do_ascii($$) {
}
}
}
- };
- alarm(0); # cancel alarm
- my($eval_stat) = $@;
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ prolong_timer('do_ascii', $remaining_time-($dt-alarm(0))); # restart timer
if (defined $oldaction) {
POSIX::sigaction(SIGALRM,$oldaction)
or die "Can't restore ALRM handler: $!";
@@ -16343,8 +18689,8 @@ sub do_ascii($$) {
snmp_count('OpsDecByUUlib') if $any_decoded;
if (defined $old_env_tmpdir) { $ENV{TMPDIR} = $old_env_tmpdir }
else { delete $ENV{TMPDIR} }
- if ($eval_stat ne '') { chomp($eval_stat); die "do_ascii: $eval_stat\n" }
- ($any_decoded && !$any_errors) ? 1 : $any_errors ? 2 : 0;
+ if ($eval_stat ne '') { chomp $eval_stat; die "do_ascii: $eval_stat\n" }
+ $any_errors ? 2 : $any_decoded ? 1 : 0;
}
# use Archive-Zip
@@ -16374,7 +18720,7 @@ sub do_unzip($$;$$) {
$retval = 0;
} else {
my($item_num) = 0; my($parent_placement) = $part->mime_placement;
- for my $mem ($zip->members()) {
+ for my $mem ($zip->members) {
my($newpart_obj) = Amavis::Unpackers::Part->new("$tempdir/parts",$part);
$item_num++; $newpart_obj->mime_placement("$parent_placement/$item_num");
$newpart_obj->name_declared($mem->fileName);
@@ -16390,7 +18736,7 @@ sub do_unzip($$;$$) {
} else {
# want to read uncompressed - set to COMPRESSION_STORED
my($oldc) = $mem->desiredCompressionMethod(COMPRESSION_STORED);
- $sts = $mem->rewindData();
+ $sts = $mem->rewindData;
$sts == AZ_OK or die sprintf("%s: error rew. member data: %s (%s)",
$part->base_name, $err_nm[$sts], $sts);
my($newpart) = $newpart_obj->full_name;
@@ -16401,7 +18747,7 @@ sub do_unzip($$;$$) {
my($size) = 0;
while ($sts == AZ_OK) {
my($buf_ref);
- ($buf_ref, $sts) = $mem->readChunk();
+ ($buf_ref, $sts) = $mem->readChunk;
$sts == AZ_OK || $sts == AZ_STREAM_END
or die sprintf("%s: error reading member: %s (%s)",
$part->base_name, $err_nm[$sts], $sts);
@@ -16416,7 +18762,7 @@ sub do_unzip($$;$$) {
$newpart_obj->size($size);
$outpart->close or die "Error closing $newpart: $!";
$mem->desiredCompressionMethod($oldc);
- $mem->endRead();
+ $mem->endRead;
$extractedcount++;
}
}
@@ -16428,7 +18774,7 @@ sub do_unzip($$;$$) {
$part->base_name, $any_unsupp_compmeth);
} elsif ($any_zero_length) { # possible zip vulnerability exploit
$retval = 2;
- do_log(1, "do_unzip: %s, zero length members, archive retained",
+ do_log(1, "do_unzip: %s, members of zero length, archive retained",
$part->base_name);
} elsif ($encryptedcount) {
$retval = 2;
@@ -16451,7 +18797,7 @@ sub do_uncompress($$$) {
$newpart_obj->mime_placement($part->mime_placement."/1");
my($newpart) = $newpart_obj->full_name;
my($type_short, $name_declared) = ($part->type_short, $part->name_declared);
- my(@rn); # collect recommended file names
+ local($1); my(@rn); # collect recommended file names
push(@rn,$1)
if $part->type_long =~ /^\S+\s+compressed data, was "(.+)"(\z|, from\b)/;
for my $name_d (!ref $name_declared ? ($name_declared) : @$name_declared) {
@@ -16474,9 +18820,10 @@ sub do_uncompress($$$) {
my($remaining_time) = alarm(0); # check time left, stop the timer
my($dt) = max(10, int(2 * $remaining_time / 3));
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat);
eval {
($proc_fh,$pid) =
- run_command($part->full_name, undef, split(' ',$decompressor));
+ run_command($part->full_name, '', split(' ',$decompressor));
my($rv,$err) = run_command_copy($newpart,$proc_fh);
undef $proc_fh; undef $pid;
if (proc_status_ok($rv,$err)) {}
@@ -16487,16 +18834,13 @@ sub do_uncompress($$$) {
# bzip2 and gzip use status 2 as a warning about corrupted file
if (proc_status_ok($rv,$err, 2)) {do_log(0,"%s",$msg)} else {die $msg}
}
- };
- my($eval_stat) = $@;
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
prolong_timer('do_uncompress',$remaining_time-($dt-alarm(0))); #restart timer
if ($eval_stat ne '') {
- $retval = 0; chomp($eval_stat);
- if (defined $pid) {
- do_log(-1, "%s is taking longer than %d s and will be killed",
- $decompressor, $dt) if $eval_stat eq "timed out";
- kill_proc($pid,$decompressor,1,$proc_fh); undef $pid;
- }
+ $retval = 0; chomp $eval_stat;
+ kill_proc($pid,$decompressor,1,$proc_fh,$eval_stat) if defined $pid;
+ undef $proc_fh; undef $pid;
do_log(-1, "do_uncompress: %s", $eval_stat);
}
$retval;
@@ -16518,7 +18862,7 @@ sub do_gunzip($$) {
or die "Can't create file $newpart: $!";
binmode($outpart) or die "Can't set file $newpart to binmode: $!";
my($nbytes,$buff); my($size) = 0;
- while (($nbytes=$gz->read($buff,16384)) > 0) {
+ while (($nbytes=$gz->read($buff,256*1024)) > 0) {
$outpart->print($buff) or die "Can't write to $newpart: $!";
$size += $nbytes; consumed_bytes($nbytes, 'do_gunzip');
}
@@ -16544,38 +18888,122 @@ sub do_gunzip($$) {
$retval;
}
+# DROPED SUPPORT for Archive::Tar; main drawback of this module is: it either
+# loads an entire tar into memory (horrors!), or when using extract_archive()
+# it does not relativize absolute paths (which makes it possible to store
+# members in any directory writable by uid), and does not provide a way to
+# capture contents of members with the same name. Use pax program instead!
+#
# untar any tar archives with Archive-Tar, extract each file individually
-sub do_tar($$) {
- my($part, $tempdir) = @_;
- snmp_count('OpsDecByArTar');
- # Work around bug in Archive-Tar
- my $tar = eval { Archive::Tar->new($part->full_name) };
- if (!defined($tar)) {
- chomp($@);
- do_log(4, "Faulty archive %s: %s", $part->full_name, $@);
- return 0;
- }
- do_log(4,"Untarring %s", $part->base_name);
- my($item_num) = 0; my($parent_placement) = $part->mime_placement;
- my(@list) = $tar->list_files();
- for (@list) {
- next if /\/\z/; # ignore directories
- # this is bad (reads whole file into scalar)
- # need some error handling, too
- my $data = $tar->get_content($_);
- my($newpart_obj) = Amavis::Unpackers::Part->new("$tempdir/parts",$part);
- $item_num++; $newpart_obj->mime_placement("$parent_placement/$item_num");
- my($newpart) = $newpart_obj->full_name;
- my($outpart) = IO::File->new;
- $outpart->open($newpart, O_CREAT|O_EXCL|O_WRONLY, 0640)
- or die "Can't create file $newpart: $!";
- binmode($outpart) or die "Can't set file $newpart to binmode: $!";
- $outpart->print($data) or die "Can't write to $newpart: $!";
- $newpart_obj->size(length($data));
- consumed_bytes(length($data), 'do_tar');
- $outpart->close or die "Error closing $newpart: $!";
- }
- 1;
+#
+#use Archive::Tar;
+#sub do_tar($$) {
+# my($part, $tempdir) = @_;
+# snmp_count('OpsDecByArTar');
+# # Work around bug in Archive-Tar
+# my $tar = eval { Archive::Tar->new($part->full_name) };
+# if (!defined($tar)) {
+# chomp $@;
+# do_log(4, "Faulty archive %s: %s", $part->full_name, $@);
+# die $@ if $@ =~ /^timed out\b/; # resignal timeout
+# return 0;
+# }
+# do_log(4,"Untarring %s", $part->base_name);
+# my($item_num) = 0; my($parent_placement) = $part->mime_placement;
+# my(@list) = $tar->list_files;
+# for (@list) {
+# next if /\/\z/; # ignore directories
+# # this is bad (reads whole file into scalar)
+# # need some error handling, too
+# my $data = $tar->get_content($_);
+# my($newpart_obj) = Amavis::Unpackers::Part->new("$tempdir/parts",$part);
+# $item_num++; $newpart_obj->mime_placement("$parent_placement/$item_num");
+# my($newpart) = $newpart_obj->full_name;
+# my($outpart) = IO::File->new;
+# $outpart->open($newpart, O_CREAT|O_EXCL|O_WRONLY, 0640)
+# or die "Can't create file $newpart: $!";
+# binmode($outpart) or die "Can't set file $newpart to binmode: $!";
+# $outpart->print($data) or die "Can't write to $newpart: $!";
+# $newpart_obj->size(length($data));
+# consumed_bytes(length($data), 'do_tar');
+# $outpart->close or die "Error closing $newpart: $!";
+# }
+# 1;
+#}
+
+# use external program to expand 7-Zip archives
+sub do_7zip($$$;$) {
+ my($part, $tempdir, $archiver, $testing_for_sfx) = @_;
+ ll(4) && do_log(4, "Expanding 7-Zip archive %s", $part->base_name);
+ my($decompressor_name) = basename((split(' ',$archiver))[0]);
+ snmp_count("OpsDecBy\u${decompressor_name}Attempt");
+ my($last_line); my($bytes) = 0; my($mem_cnt) = 0;
+ my($retval) = 1; my($proc_fh,$pid); my($fn) = $part->full_name;
+ my($remaining_time) = alarm(0); # check time left, stop the timer
+ my($dt) = max(10, int(2 * $remaining_time / 3));
+ alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat);
+ eval {
+ ($proc_fh,$pid) = run_command(undef, "&1", $archiver,
+ 'l', '-slt', '-pfoobar', "-w$tempdir/parts", $fn);
+ my($ln); my($name,$size,$attr); my($entries_cnt) = 0;
+ for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) {
+ $last_line = $ln if $ln !~ /^\s*$/; # keep last nonempty line
+ chomp($ln); local($1);
+ if ($ln =~ /^\s*\z/) {
+ if (defined $name || defined $size) {
+ do_log(5,'do_7zip: member: %s "%s", %s bytes', $attr,$name,$size);
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
+ if (defined $size && $size > 0) { $bytes += $size; $mem_cnt++ }
+ }
+ undef $name; undef $size; undef $attr;
+ } elsif ($ln =~ /^Path = (.*)\z/s) { $name = $1 }
+ elsif ($ln =~ /^Size = ([0-9]+)\z/s) { $size = $1 }
+ elsif ($ln =~ /^Attributes = (.*)\z/s) { $attr = $1 }
+ }
+ defined $ln || $!==0 || $!==EAGAIN or die "Error reading: $!";
+ if (defined $name || defined $size) {
+ do_log(5,'do_7zip: member: %s "%s", %s bytes', $attr,$name,$size);
+ if (defined $size && $size > 0) { $bytes += $size; $mem_cnt++ }
+ }
+ # consume all remaining output to avoid broken pipe
+ for ($! = 0; defined($ln=$proc_fh->getline); $! = 0)
+ { $last_line = $ln if $ln !~ /^\s*$/ }
+ defined $ln || $!==0 || $!==EAGAIN or die "Error reading: $!";
+ my($err) = 0; $proc_fh->close or $err = $!; my($rv) = $?;
+ undef $proc_fh; undef $pid; local($1,$2);
+ if (proc_status_ok($rv,$err,1) && $mem_cnt > 0 && $bytes > 0) { # just warn
+ do_log(4,"do_7zip: warning, %s", exit_status_str($rv,$err));
+ } elsif (!proc_status_ok($rv,$err)) {
+ die("can't get a list of archive members: " .
+ exit_status_str($rv,$err) ."; ".$last_line);
+ }
+ if ($mem_cnt > 0 || $bytes > 0) {
+ consumed_bytes($bytes, 'do_7zip-pre', 1); # pre-check on estimated size
+ snmp_count("OpsDecBy\u${decompressor_name}");
+ ($proc_fh,$pid) = run_command(undef, "&1", $archiver, 'x', '-bd', '-y',
+ '-pfoobar', "-w$tempdir/parts", "-o$tempdir/parts/7zip", $fn);
+ collect_results($proc_fh,$pid,$archiver,16384,[0,1]);
+ undef $proc_fh; undef $pid;
+ my($errn) = lstat("$tempdir/parts/7zip") ? 0 : 0+$!;
+ if ($errn != ENOENT) {
+ my($b) = flatten_and_tidy_dir("$tempdir/parts/7zip",
+ "$tempdir/parts", $part);
+ consumed_bytes($b, 'do_7zip');
+ }
+ }
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
+ prolong_timer('do_7zip', $remaining_time-($dt-alarm(0))); # restart timer
+ if ($eval_stat ne '') {
+ $retval = 0; chomp $eval_stat;
+ kill_proc($pid,$archiver,1,$proc_fh,$eval_stat) if defined $pid;
+ undef $proc_fh; undef $pid;
+ if ($testing_for_sfx) { die "do_7zip: $eval_stat" }
+ else { do_log(-1, "do_7zip: %s", $eval_stat) };
+ }
+ $retval;
}
# use external program to expand RAR archives
@@ -16596,10 +19024,11 @@ sub do_unrar($$$;$) {
my($remaining_time) = alarm(0); # check time left, stop the timer
my($dt) = max(10, int(2 * $remaining_time / 3));
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat);
eval {
($proc_fh,$pid) =
run_command(undef, "&1", $archiver, 'v', at common_rar_switches,'--',$fn);
- local($_);
+ local($_); my($entries_cnt) = 0;
# jump hoops because there is no simple way to just list all the files
for ($! = 0; defined($_=$proc_fh->getline); $! = 0) {
$last_line = $_ if !/^\s*$/; # keep last nonempty line
@@ -16615,6 +19044,8 @@ sub do_unrar($$$;$) {
} elsif ($hypcount == 1) {
$lcnt++; local($1,$2,$3);
if ($lcnt % 2 == 0) { # information line (every other line)
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
if (!/^\s+(\d+)\s+(\d+)\s+(\d+%|-->|<--)/) {
do_log($testing_for_sfx ? 4 : -1,
"do_unrar: can't parse info line for \"%s\" %s",
@@ -16683,12 +19114,9 @@ sub do_unrar($$$;$) {
($proc_fh,$pid) =
run_command(undef, "&1", $archiver, qw(x -inul -ver -o- -kb),
@common_rar_switches, '--', $fn, "$tempdir/parts/rar/");
- my($nbytes,$buff); my($output) = '';
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { $output .= $buff }
- defined $nbytes or die "Error reading: $!";
- my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- proc_status_ok($?,$err, 0,1,3) # one of: SUCCESS, WARNING, CRC
- or do_log(-1, 'do_unrar %s', exit_status_str($?,$err));
+ collect_results($proc_fh,$pid,$archiver,16384,
+ [0,1,3] ); # one of: SUCCESS, WARNING, CRC
+ undef $proc_fh; undef $pid;
my($errn) = lstat("$tempdir/parts/rar") ? 0 : 0+$!;
if ($errn != ENOENT) {
my($b) = flatten_and_tidy_dir("$tempdir/parts/rar",
@@ -16696,8 +19124,8 @@ sub do_unrar($$$;$) {
consumed_bytes($b, 'do_unrar');
}
}
- };
- my($eval_stat) = $@;
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
prolong_timer('do_unrar', $remaining_time-($dt-alarm(0))); # restart timer
if ($encryptedcount) {
do_log(1,
@@ -16706,12 +19134,9 @@ sub do_unrar($$$;$) {
$retval = 2;
}
if ($eval_stat ne '') {
- $retval = 0; chomp($eval_stat);
- if (defined $pid) {
- do_log(-1, "%s is taking longer than %d s and will be killed",
- $archiver, $dt) if $eval_stat eq "timed out";
- kill_proc($pid,$archiver,1,$proc_fh); undef $pid;
- }
+ $retval = 0; chomp $eval_stat;
+ kill_proc($pid,$archiver,1,$proc_fh,$eval_stat) if defined $pid;
+ undef $proc_fh; undef $pid;
if ($testing_for_sfx) { die "do_unrar: $eval_stat" }
else { do_log(-1, "do_unrar: %s", $eval_stat) };
}
@@ -16725,6 +19150,7 @@ sub do_lha($$$;$) {
my($decompressor_name) = basename((split(' ',$archiver))[0]);
snmp_count("OpsDecBy\u${decompressor_name}Attempt");
# lha needs extension .exe to understand SFX!
+ # the downside is that in this case it only sees MS files in an archive
my($fn) = $part->full_name;
symlink($fn, $fn.".exe")
or die sprintf("Can't symlink %s %s.exe: %s", $fn, $fn, $!);
@@ -16733,16 +19159,23 @@ sub do_lha($$$;$) {
my($remaining_time) = alarm(0); # check time left, stop the timer
my($dt) = max(10, int(2 * $remaining_time / 3));
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat);
eval {
+ # ($proc_fh,$pid) = run_command(undef, "&1", $archiver, 'lq', $fn);
($proc_fh,$pid) = run_command(undef, "&1", $archiver, 'lq', $fn.".exe");
- my($ln);
+ my($ln); my($entries_cnt) = 0;
for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) {
chomp($ln); local($1);
- next if $ln =~ m{/\z}; # ignore directories
- if ($ln =~ /^LHa: (Warning|Fatal error): /)
- { push(@checkerr,$ln) if @checkerr < 3 }
- elsif ($ln =~ /^(?:\S+\s+){6}\S+\s*(\S.*?)\s*\z/s) { push(@list,$1) }
- else { do_log(5,"do_lha: skip: %s", $ln) }
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
+ if ($ln =~ m{/\z}) {
+ # ignore directories
+ } elsif ($ln =~ /^LHa: (Warning|Fatal error): /) {
+ push(@checkerr,$ln) if @checkerr < 3;
+ } elsif ($ln=~m{^(?:\S+\s+\d+/\d+|.{23})(?:\s+\S+){5}\s*(\S.*?)\s*\z}s) {
+ my($name) = $1; $name = $1 if $name =~ m{^(.*) -> (.*)\z}s; # symlink
+ push(@list, $name);
+ } else { do_log(5,"do_lha: skip: %s", $ln) }
}
defined $ln || $!==0 || $!==EAGAIN or die "Error reading: $!";
my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
@@ -16752,21 +19185,19 @@ sub do_lha($$$;$) {
$part->attributes_add('U') if !$testing_for_sfx;
die "no archive members, or not an archive at all";
}
- };
- my($eval_stat) = $@;
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
prolong_timer('do_lha', $remaining_time-($dt-alarm(0))); # restart timer
if ($eval_stat ne '') {
unlink($fn.".exe") or do_log(-1, "Can't unlink %s.exe: %s", $fn,$!);
- $retval = 0; chomp($eval_stat);
- if (defined $pid) {
- do_log(-1, "%s is taking longer than %d s and will be killed",
- $archiver, $dt) if $eval_stat eq "timed out";
- kill_proc($pid,$archiver,1,$proc_fh); undef $pid;
- }
+ $retval = 0; chomp $eval_stat;
+ kill_proc($pid,$archiver,1,$proc_fh,$eval_stat) if defined $pid;
+ undef $proc_fh; undef $pid;
if ($testing_for_sfx) { die "do_lha: $eval_stat" }
else { do_log(-1, "do_lha: %s", $eval_stat) };
} else { # preliminary archive traversal done, now extract files
snmp_count("OpsDecBy\u${decompressor_name}");
+ # my($rv) = store_mgr($tempdir, $part, \@list, $archiver, 'pq', $fn);
my($rv) = store_mgr($tempdir, $part, \@list, $archiver, 'pq', $fn.".exe");
$rv==0 or die exit_status_str($rv);
unlink($fn.".exe") or die "Can't unlink $fn.exe: $!";
@@ -16785,14 +19216,18 @@ sub do_arc($$$) {
ll(4) && do_log(4,"Unarcing %s, using %s",
$part->base_name, ($is_nomarch ? "nomarch" : "arc") );
my($cmdargs) = ($is_nomarch ? "-l -U" : "ln") . " " . $part->full_name;
- my($proc_fh,$pid) = run_command(undef,undef,$archiver, split(' ',$cmdargs));
- my(@list); my($ln);
- for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) { push(@list,$ln) }
+ my($proc_fh,$pid) = run_command(undef,'',$archiver, split(' ',$cmdargs));
+ my(@list); my($ln); my($entries_cnt) = 0;
+ for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) {
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
+ push(@list,$ln);
+ }
defined $ln || $!==0 || $!==EAGAIN or die "Error reading: $!";
my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
proc_status_ok($?,$err) or do_log(-1, 'do_arc: %s',exit_status_str($?,$err));
#*** no spaces in filenames allowed???
- map { s/^([^ \t\r\n]*).*\z/$1/s } @list; # keep only filenames
+ local($1); map { s/^([^ \t\r\n]*).*\z/$1/s } @list; # keep only filenames
if (@list) {
my($rv) = store_mgr($tempdir, $part, \@list, $archiver,
($is_nomarch ? ('-p', '-U') : 'p'), $part->full_name);
@@ -16818,6 +19253,7 @@ sub do_zoo($$$) {
my($remaining_time) = alarm(0); # check time left, stop the timer
my($dt) = max(10, int(2 * $remaining_time / 3));
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat); my($entries_cnt) = 0;
eval {
($proc_fh,$pid) = run_command(undef, "&1", $archiver,
$is_unzoo ? qw(-l) : qw(l), "$fn.zoo");
@@ -16826,6 +19262,8 @@ sub do_zoo($$$) {
if ($ln =~ /^------/) { $separ_count++ }
elsif ($separ_count == 1) {
local($1,$2);
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
if ($ln !~ /^\s*(\d+)(?:\s+\S+){6}\s+(?:[0-7]{3,})?\s*(.*)$/) {
do_log(3,"do_zoo: can't parse line %s", $ln);
} else {
@@ -16864,26 +19302,19 @@ sub do_zoo($$$) {
run_command(undef, "&1", $archiver,
$is_unzoo ? qw(-x -j X) : qw(x),
"$fn.zoo", $is_unzoo ? '*;*' : () );
- my($nbytes,$buff); my($output) = '';
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { $output .= $buff }
- defined $nbytes or die "Error reading: $!";
- my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- proc_status_ok($?,$err)
- or do_log(-1,'do_zoo %s', exit_status_str($?,$err));
+ collect_results($proc_fh,$pid,$archiver,16384,[0]);
+ undef $proc_fh; undef $pid;
my($b) = flatten_and_tidy_dir("$tempdir/parts/zoo",
"$tempdir/parts", $part);
consumed_bytes($b, 'do_zoo');
}
- };
- my($eval_stat) = $@;
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
prolong_timer('do_zoo', $remaining_time-($dt-alarm(0))); # restart timer
if ($eval_stat ne '') {
- $retval = 0; chomp($eval_stat);
- if (defined $pid) {
- do_log(-1,"%s is taking longer than %d s and will be killed",
- $archiver, $dt) if $eval_stat eq "timed out";
- kill_proc($pid,$archiver,1,$proc_fh); undef $pid;
- }
+ $retval = 0; chomp $eval_stat;
+ kill_proc($pid,$archiver,1,$proc_fh,$eval_stat) if defined $pid;
+ undef $proc_fh; undef $pid;
do_log(-1,"do_zoo: %s", $eval_stat);
}
chdir($tempdir) or die "Can't chdir to $tempdir: $!";
@@ -16909,6 +19340,7 @@ sub do_unarj($$$;$) {
my($remaining_time) = alarm(0); # check time left, stop the timer
my($dt) = max(10, int(2 * $remaining_time / 3));
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat);
eval {
# obtain total original size of archive members from the index/listing
($proc_fh,$pid) = run_command(undef, "&1", $archiver, 'l', $fn.".arj");
@@ -16937,8 +19369,10 @@ sub do_unarj($$$;$) {
or die "Can't chdir to $tempdir/parts/arj: $!";
snmp_count("OpsDecBy\u${decompressor_name}");
($proc_fh,$pid) = run_command(undef, "&1", $archiver, 'e', $fn.".arj");
- my($encryptedcount,$skippedcount) = (0,0);
+ my($encryptedcount,$skippedcount) = (0,0); my($entries_cnt) = 0;
for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) {
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
$encryptedcount++
if $ln =~ /^(Extracting.*\bBad file data or bad password|File is password encrypted, Skipped)\b/s;
$skippedcount++
@@ -16972,17 +19406,14 @@ sub do_unarj($$$;$) {
$part->base_name, $encryptedcount, $skippedcount);
$retval = 2;
}
- };
- my($eval_stat) = $@;
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
prolong_timer('do_unarj', $remaining_time-($dt-alarm(0))); # restart timer
unlink($fn.".arj") or die "Can't unlink $fn.arj: $!";
if ($eval_stat ne '') {
- $retval = 0; chomp($eval_stat);
- if (defined $pid) {
- do_log(-1, "%s is taking longer than %d s and will be killed",
- $archiver, $dt) if $eval_stat eq "timed out";
- kill_proc($pid,$archiver,1,$proc_fh); undef $pid;
- }
+ $retval = 0; chomp $eval_stat;
+ kill_proc($pid,$archiver,1,$proc_fh,$eval_stat) if defined $pid;
+ undef $proc_fh; undef $pid;
if ($testing_for_sfx) { die "do_unarj: $eval_stat" }
else { do_log(-1, "do_unarj: %s", $eval_stat) };
}
@@ -16999,29 +19430,21 @@ sub do_tnef_ext($$$) {
or die "Can't mkdir $tempdir/parts/tnef: $!";
my($retval) = 1; my($proc_fh,$pid);
- my($rem_quota) = max(10*1024, consumed_bytes(0,'do_tnef_ext'));
+ my($rem_quota) = max(10*1024, untaint(consumed_bytes(0,'do_tnef_ext')));
my($remaining_time) = alarm(0); # check time left, stop the timer
my($dt) = max(10, int(2 * $remaining_time / 3));
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat);
eval {
($proc_fh,$pid) = run_command(undef, "&1", $archiver,
'--number-backups', '-x', "$rem_quota",
'-C', "$tempdir/parts/tnef", '-f', $part->full_name);
- my($nbytes,$buff); my($output) = '';
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { $output .= $buff }
- defined $nbytes or die "Error reading: $!";
- my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- proc_status_ok($?,$err) or die(exit_status_str($?,$err).' '.$output);
- };
- my($eval_stat) = $@;
+ collect_results($proc_fh,$pid,$archiver,16384,[0]);
+ undef $proc_fh; undef $pid; 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
prolong_timer('do_tnef_ext', $remaining_time-($dt-alarm(0))); # restart timer
if ($eval_stat ne '') {
- $retval = 0; chomp($eval_stat);
- if (defined $pid) {
- do_log(-1, "%s is taking longer than %d s and will be killed",
- $archiver, $dt) if $eval_stat eq "timed out";
- kill_proc($pid,$archiver,1,$proc_fh); undef $pid;
- }
+ $retval = 0; chomp $eval_stat;
do_log(-1, "tnef_ext: %s", $eval_stat);
}
my($b) = flatten_and_tidy_dir("$tempdir/parts/tnef","$tempdir/parts",$part);
@@ -17038,7 +19461,7 @@ sub do_tnef($$) {
do_log(4, "Extracting from TNEF encapsulation (int) %s", $part->base_name);
snmp_count('OpsDecByTnef');
my($tnef) = Convert::TNEF->read_in($part->full_name,
- {output_dir=>"$tempdir/parts", buffer_size=>16384, ignore_checksum=>1});
+ {output_dir=>"$tempdir/parts", buffer_size=>65536, ignore_checksum=>1});
defined $tnef or die "Convert::TNEF failed: ".$Convert::TNEF::errstr;
my($item_num) = 0; my($parent_placement) = $part->mime_placement;
for my $a ($tnef->message, $tnef->attachments) {
@@ -17054,11 +19477,11 @@ sub do_tnef($$) {
$outpart->open($newpart, O_CREAT|O_EXCL|O_WRONLY, 0640)
or die "Can't create file $newpart: $!";
binmode($outpart) or die "Can't set file $newpart to binmode: $!";
- my($file) = $dh->path; my($size) = 0;
- if (defined $file) {
+ my($filepath) = $dh->path; my($size) = 0;
+ if (defined $filepath) {
my($io,$nbytes,$buff); $dh->binmode(1);
$io = $dh->open("r") or die "Can't open MIME::Body handle: $!";
- while (($nbytes=$io->read($buff,16384)) > 0) {
+ while (($nbytes=$io->read($buff,65536)) > 0) {
$outpart->print($buff) or die "Can't write to $newpart: $!";
$size += $nbytes; consumed_bytes($nbytes, 'do_tnef_1');
}
@@ -17094,12 +19517,14 @@ sub do_pax_cpio($$$) {
"the pax(1) utility is available on the system!",
$archiver_name) if !$is_pax;
my(@cmdargs) = $is_pax ? qw(-v) : qw(-i -t -v);
- my($proc_fh,$pid) = run_command($part->full_name, undef, $archiver, at cmdargs);
- my($bytes) = 0; local($1,$2); local($_);
+ my($proc_fh,$pid) = run_command($part->full_name, '', $archiver, at cmdargs);
+ my($bytes) = 0; local($1,$2); local($_); my($entries_cnt) = 0;
for ($! = 0; defined($_=$proc_fh->getline); $! = 0) {
chomp;
next if /^\d+ blocks\z/;
last if /^(cpio|pax): (.*bytes read|End of archive volume)/;
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
if (!/^ (?: \S+\s+ ){4} (\d+) \s+ (.+) \z/xs) {
do_log(-1,"do_pax_cpio: can't parse toc line: %s", $_);
} else {
@@ -17117,12 +19542,8 @@ sub do_pax_cpio($$$) {
}
defined $_ || $!==0 || $!==EAGAIN or die "Error reading: $!";
# consume remaining output to avoid broken pipe
- my($nbytes,$buff);
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { }
- defined $nbytes or die "Error reading: $!";
- my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- proc_status_ok($?,$err)
- or do_log(-1, 'do_pax_cpio/1: %s', exit_status_str($?,$err));
+ collect_results($proc_fh,$pid,'do_pax_cpio/1',16384,[0]);
+ undef $proc_fh; undef $pid;
consumed_bytes($bytes, 'do_pax_cpio/pre', 1); # pre-check on estimated size
mkdir("$tempdir/parts/arch", 0750)
or die "Can't mkdir $tempdir/parts/arch: $!";
@@ -17132,15 +19553,18 @@ sub do_pax_cpio($$$) {
my($remaining_time) = alarm(0); # check time left, stop the timer
my($dt) = max(10, int(2 * $remaining_time / 3));
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat);
eval {
chdir("$tempdir/parts/arch")
or die "Can't chdir to $tempdir/parts/arch: $!";
my(@cmdargs) = $is_pax ? qw(-r -k -p am -s /[^A-Za-z0-9_]/-/gp)
: qw(-i -d --no-absolute-filenames --no-preserve-owner);
($proc_fh,$pid) = run_command($part->full_name, "&1", $archiver, @cmdargs);
- my($output) = ''; my($ln);
+ my($output) = ''; my($ln); my($entries_cnt) = 0;
for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) {
chomp($ln);
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
if (!$is_pax || $ln !~ /^(.*) >> (\S*)\z/) { $output .= $ln."\n" }
else { # parse output from pax -s///p
my($member_name,$file_name) = ($1,$2);
@@ -17159,20 +19583,17 @@ sub do_pax_cpio($$$) {
my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
chomp($output);
proc_status_ok($?,$err) or die(exit_status_str($?,$err).' '.$output);
- };
- my($eval_stat) = $@;
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
prolong_timer('do_pax_cpio', $remaining_time-($dt-alarm(0))); # restart timer
chdir($tempdir) or die "Can't chdir to $tempdir: $!";
my($b) = flatten_and_tidy_dir("$tempdir/parts/arch", "$tempdir/parts",
$part, 0, \%orig_names);
consumed_bytes($b, 'do_pax_cpio');
if ($eval_stat ne '') {
- chomp($eval_stat);
- if (defined $pid) {
- do_log(-1, "%s is taking longer than %d s and will be killed",
- $archiver, $dt) if $eval_stat eq "timed out";
- kill_proc($pid,$archiver,1,$proc_fh); undef $pid;
- }
+ chomp $eval_stat;
+ kill_proc($pid,$archiver,1,$proc_fh,$eval_stat) if defined $pid;
+ undef $proc_fh; undef $pid;
die "do_pax_cpio: $eval_stat\n";
}
$name_clash ? 2 : 1;
@@ -17190,12 +19611,8 @@ sub do_unstuff($$$) {
or die "Can't mkdir $tempdir/parts/unstuff: $!";
my($proc_fh,$pid) = run_command(undef, "&1", $archiver, # '-q',
"-d=$tempdir/parts/unstuff", $part->full_name);
- my($nbytes,$buff); my($output) = '';
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { $output .= $buff }
- defined $nbytes or die "Error reading: $!";
- my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- do_log(proc_status_ok($?,$err) ? 5 : -1,
- 'do_unstuff %s %s', exit_status_str($?,$err), $output);
+ collect_results($proc_fh,$pid,$archiver,16384,[0]);
+ undef $proc_fh; undef $pid;
my($b) = flatten_and_tidy_dir("$tempdir/parts/unstuff",
"$tempdir/parts", $part);
consumed_bytes($b, 'do_unstuff');
@@ -17208,10 +19625,12 @@ sub do_ar($$$) {
ll(4) && do_log(4,"Expanding Unix ar archive %s", $part->full_name);
my($archiver_name) = basename((split(' ',$archiver))[0]);
snmp_count("OpsDecBy\u${archiver_name}");
- my($proc_fh,$pid) = run_command(undef,undef,$archiver,'tv',$part->full_name);
- my($ln); my($bytes) = 0; local($1,$2,$3);
+ my($proc_fh,$pid) = run_command(undef,'',$archiver,'tv',$part->full_name);
+ my($ln); my($bytes) = 0; local($1,$2,$3); my($entries_cnt) = 0;
for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) {
chomp($ln);
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
if ($ln !~ /^(?:\S+\s+){2}(\d+)\s+((?:\S+\s+){3}\S+)\s+(.*)\z/) {
do_log(-1,"do_ar: can't parse contents listing line: %s", $ln);
} else {
@@ -17221,22 +19640,15 @@ sub do_ar($$$) {
}
defined $ln || $!==0 || $!==EAGAIN or die "Error reading: $!";
# consume remaining output to avoid broken pipe
- my($nbytes,$buff);
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { }
- defined $nbytes or die "Error reading: $!";
- my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- proc_status_ok($?,$err) or do_log(-1, 'ar-1 %s', exit_status_str($?,$err));
+ collect_results($proc_fh,$pid,'ar-1',16384,[0]);
+ undef $proc_fh; undef $pid;
consumed_bytes($bytes, 'do_ar-pre', 1); # pre-check on estimated size
mkdir("$tempdir/parts/ar", 0750)
or die "Can't mkdir $tempdir/parts/ar: $!";
chdir("$tempdir/parts/ar") or die "Can't chdir to $tempdir/parts/ar: $!";
($proc_fh,$pid) = run_command(undef, "&1", $archiver, 'x', $part->full_name);
- my($output) = '';
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { $output .= $buff }
- defined $nbytes or die "Error reading: $!";
- $err = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- proc_status_ok($?,$err)
- or do_log(-1, 'ar-2 %s %s', exit_status_str($?,$err), $output);
+ collect_results($proc_fh,$pid,'ar-2',16384,[0]);
+ undef $proc_fh; undef $pid;
chdir($tempdir) or die "Can't chdir to $tempdir: $!";
my($b) = flatten_and_tidy_dir("$tempdir/parts/ar","$tempdir/parts",$part);
consumed_bytes($b, 'do_ar');
@@ -17248,12 +19660,14 @@ sub do_cabextract($$$) {
do_log(4, "Expanding cab archive %s", $part->base_name);
my($archiver_name) = basename((split(' ',$archiver))[0]);
snmp_count("OpsDecBy\u${archiver_name}");
- local($_); my($bytes) = 0; my($ln);
+ local($_,$1,$2); my($bytes) = 0; my($ln); my($entries_cnt) = 0;
my($proc_fh,$pid) =
- run_command(undef,undef,$archiver,'-l',$part->full_name);
+ run_command(undef,'',$archiver,'-l',$part->full_name);
for ($! = 0; defined($ln=$proc_fh->getline); $! = 0) {
chomp($ln);
next if $ln =~ /^(File size|----|Viewing cabinet:|\z)/;
+ if ($entries_cnt++, $MAXFILES && $entries_cnt > $MAXFILES)
+ { die "Maximum number of files ($MAXFILES) exceeded" }
if ($ln !~ /^\s* (\d+) \s* \| [^|]* \| \s (.*) \z/x) {
do_log(-1, "do_cabextract: can't parse toc line: %s", $ln);
} else {
@@ -17263,22 +19677,13 @@ sub do_cabextract($$$) {
}
defined $ln || $!==0 || $!==EAGAIN or die "Error reading: $!";
# consume remaining output to avoid broken pipe (just in case)
- my($nbytes,$buff);
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { }
- defined $nbytes or die "Error reading: $!";
- my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- proc_status_ok($?,$err)
- or do_log(-1, 'cabextract-1 %s', exit_status_str($?,$err));
- consumed_bytes($bytes, 'do_cabextract-pre', 1); # pre-check on estimated size
+ collect_results($proc_fh,$pid,'cabextract-1',16384,[0]);
+ undef $proc_fh; undef $pid;
mkdir("$tempdir/parts/cab",0750) or die "Can't mkdir $tempdir/parts/cab: $!";
- ($proc_fh,$pid) = run_command(undef, undef, $archiver, '-q', '-d',
+ ($proc_fh,$pid) = run_command(undef, '', $archiver, '-q', '-d',
"$tempdir/parts/cab", $part->full_name);
- my($output) = '';
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { $output .= $buff }
- defined $nbytes or die "Error reading: $!";
- $err = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- proc_status_ok($?,$err)
- or do_log(-1, 'cabextract-2 %s %s', exit_status_str($?,$err), $output);
+ collect_results($proc_fh,$pid,'cabextract-2',16384,[0]);
+ undef $proc_fh; undef $pid;
my($b) = flatten_and_tidy_dir("$tempdir/parts/cab", "$tempdir/parts", $part);
consumed_bytes($b, 'do_cabextract');
1;
@@ -17292,12 +19697,8 @@ sub do_ole($$$) {
mkdir("$tempdir/parts/ole",0750) or die "Can't mkdir $tempdir/parts/ole: $!";
my($proc_fh,$pid) = run_command(undef, "&1", $archiver, '-v',
'-i', $part->full_name, '-d',"$tempdir/parts/ole");
- my($nbytes,$buff); my($output) = '';
- while (($nbytes=$proc_fh->read($buff,4096)) > 0) { $output .= $buff }
- defined $nbytes or die "Error reading: $!";
- my($err) = 0; $proc_fh->close or $err = $!; undef $proc_fh; undef $pid;
- proc_status_ok($?,$err)
- or do_log(0, 'ripOLE %s %s', exit_status_str($?,$err), $output);
+ collect_results($proc_fh,$pid,$archiver,16384,[0]);
+ undef $proc_fh; undef $pid;
my($b) = flatten_and_tidy_dir("$tempdir/parts/ole", "$tempdir/parts", $part);
if ($b > 0) {
do_log(4, "ripOLE extracted %d bytes from an OLE document", $b);
@@ -17316,22 +19717,22 @@ sub do_executable($$@) {
$part->base_name);
# ZIP?
return 2 if eval { do_unzip($part,$tempdir,undef,1) };
- chomp($@);
+ chomp $@;
do_log(3, "do_executable: not a ZIP sfx, ignoring: %s", $@) if $@ ne '';
# RAR?
return 2 if defined $unrar && eval { do_unrar($part,$tempdir,$unrar,1) };
- chomp($@);
+ chomp $@;
do_log(3, "do_executable: not a RAR sfx, ignoring: %s", $@) if $@ ne '';
# LHA?
return 2 if defined $lha && eval { do_lha($part,$tempdir,$lha,1) };
- chomp($@);
+ chomp $@;
do_log(3, "do_executable: not a LHA sfx, ignoring: %s", $@) if $@ ne '';
# ARJ?
return 2 if defined $unarj && eval { do_unarj($part,$tempdir,$unarj,1) };
- chomp($@);
+ chomp $@;
do_log(3, "do_executable: not an ARJ sfx, ignoring: %s", $@) if $@ ne '';
return 0;
@@ -17344,7 +19745,7 @@ sub do_executable($$@) {
# }
# Given a file handle (typically opened pipe to a subprocess, as returned
-# from run_command), copy from it to a specified output file in binary mode.
+# by run_command), copy from it to a specified output file in binary mode.
sub run_command_copy($$) {
my($outfile, $ifh) = @_;
my($ofh) = IO::File->new;
@@ -17353,7 +19754,7 @@ sub run_command_copy($$) {
binmode($ofh) or die "Can't set file $outfile to binmode: $!";
binmode($ifh) or die "Can't set binmode on pipe: $!";
my($len, $buf, $offset, $written);
- for ($! = 0; ($len=$ifh->sysread($buf,16384)) > 0; $! = 0) {
+ for ($! = 0; ($len=$ifh->sysread($buf,256*1024)) > 0; $! = 0) {
$offset = 0;
while ($len > 0) { # handle partial writes
$written = syswrite($ofh, $buf, $len, $offset);
@@ -17379,6 +19780,7 @@ sub store_mgr($$$@) {
my($remaining_time) = alarm(0); # check time left, stop the timer
my($dt) = max(10, int(2 * $remaining_time / 3));
alarm($dt); do_log(5,"timer set to %d s (was %d s)", $dt,$remaining_time);
+ my($eval_stat);
eval {
for my $f (@$list) {
next if $f =~ m{/\z}; # ignore directories
@@ -17393,23 +19795,20 @@ sub store_mgr($$$@) {
} else { # this is not too bad, as run_command does not use shell
do_log(1, 'store_mgr: NOTICE: suspicious file name "%s"', $f);
}
- ($proc_fh,$pid) = run_command(undef,undef,$archiver, at args,untaint($f));
+ ($proc_fh,$pid) = run_command(undef,'',$archiver, at args,untaint($f));
my($rv,$err) = run_command_copy($newpart,$proc_fh);
my($ll) = proc_status_ok($rv,$err) ? 5 : 1;
ll($ll) && do_log($ll,"store_mgr: extracted by %s, %s",
$archiver, exit_status_str($rv,$err));
$retval = $rv if $retval == 0 && $rv != 0;
}
- };
- my($eval_stat) = $@;
+ 1;
+ } or do { $eval_stat = $@ ne '' ? $@ : "errno=$!" };
prolong_timer('store_mgr', $remaining_time-($dt-alarm(0))); # restart timer
if ($eval_stat ne '') {
- $retval = 0; chomp($eval_stat);
- if (defined $pid) {
- do_log(-1, "%s is taking longer than %d s and will be killed",
- "store_mgr: $archiver", $dt) if $eval_stat eq "timed out";
- kill_proc($pid,$archiver,1,$proc_fh); undef $pid;
- }
+ $retval = 0; chomp $eval_stat;
+ kill_proc($pid,$archiver,1,$proc_fh,$eval_stat) if defined $pid;
+ undef $proc_fh; undef $pid;
do_log(-1, "store_mgr: %s", $eval_stat);
}
$retval; # return the first nonzero status (if any), or 0
@@ -17425,9 +19824,8 @@ __DATA__
# multi-line text will produce several log entries, one for each nonempty line.
# Syntax is explained in the README.customize file.
[?%#D|#|Passed #
-[? [:ccat_maj] |OTHER|CLEAN|TEMPFAIL|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
+[? [:ccat|major] |OTHER|CLEAN|MTA-BLOCKED|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
UNCHECKED|BANNED (%F)|INFECTED (%V)]#
-#([:ccat_maj],[:ccat_min])#
, [? %p ||%p ][?%a||[?%l||LOCAL ]\[%a\] ][?%e||\[%e\] ]%s -> [%D|,]#
[? %q ||, quarantine: %q]#
[? %Q ||, Queue-ID: %Q]#
@@ -17435,20 +19833,21 @@ UNCHECKED|BANNED (%F)|INFECTED (%V)]#
[? %r ||, Resent-Message-ID: %r]#
, mail_id: %i#
, Hits: [:SCORE]#
-#, size: %z#
-#, fwd_to: [:remote_mta]#
+, size: %z#
[~[:remote_mta_smtp_response]|["^$"]||[", queued_as: "]]\
[remote_mta_smtp_response|[~%x|["queued as ([0-9A-Z]+)$"]|["%1"]|["%0"]]|/]#
#[? [:header_field|Subject] ||, Subject: [:dquote|[:header_field|Subject]]]#
#[? [:header_field|From] ||, From: [:uquote|[:header_field|From]]]#
+#[? [:useragent|name] ||, [:useragent|name]: [:uquote|[:useragent|body]]]#
#[? %#T ||, Tests: \[[%T|,]\]]#
-#[? [:AUTOLEARN] ||, autolearn=[:AUTOLEARN]]#
+#[:supplementary_info|SCTYPE|, shortcircuit=%%s]#
+#[:supplementary_info|AUTOLEARN|, autolearn=%%s]#
, %y ms#
]
[?%#O|#|Blocked #
-[? [:ccat_maj] |OTHER|CLEAN|TEMPFAIL|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
+[? [:ccat|major|blocking] |#
+OTHER|CLEAN|MTA-BLOCKED|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
UNCHECKED|BANNED (%F)|INFECTED (%V)]#
-#([:ccat_maj],[:ccat_min])#
, [? %p ||%p ][?%a||[?%l||LOCAL ]\[%a\] ][?%e||\[%e\] ]%s -> [%O|,]#
[? %q ||, quarantine: %q]#
[? %Q ||, Queue-ID: %Q]#
@@ -17456,12 +19855,14 @@ UNCHECKED|BANNED (%F)|INFECTED (%V)]#
[? %r ||, Resent-Message-ID: %r]#
, mail_id: %i#
, Hits: [:SCORE]#
-#, size: %z#
+, size: %z#
#, smtp_resp: [:smtp_response]#
#[? [:header_field|Subject] ||, Subject: [:dquote|[:header_field|Subject]]]#
#[? [:header_field|From] ||, From: [:uquote|[:header_field|From]]]#
+#[? [:useragent|name] ||, [:useragent|name]: [:uquote|[:useragent|body]]]#
#[? %#T ||, Tests: \[[%T|,]\]]#
-#[? [:AUTOLEARN] ||, autolearn=[:AUTOLEARN]]#
+#[:supplementary_info|SCTYPE|, shortcircuit=%%s]#
+#[:supplementary_info|AUTOLEARN|, autolearn=%%s]#
, %y ms#
]
__DATA__
@@ -17475,9 +19876,9 @@ __DATA__
# Long header fields will be automatically wrapped by the program.
#
[?%#D|#|Passed #
-[? [:ccat_maj] |OTHER|CLEAN|TEMPFAIL|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
+#([:ccat|name|main]) #
+[? [:ccat|major] |OTHER|CLEAN|MTA-BLOCKED|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
UNCHECKED|BANNED (%F)|INFECTED (%V)]#
-#([:ccat_maj],[:ccat_min])#
, %s -> [%D|,], Hits: %c#
, tag=[:tag_level], tag2=[:tag2_level], kill=[:kill_level]#
[~[:remote_mta_smtp_response]|["^$"]||\
@@ -17485,9 +19886,10 @@ UNCHECKED|BANNED (%F)|INFECTED (%V)]#
, %0/%1/%2/%k#
]
[?%#O|#|Blocked #
-[? [:ccat_maj] |OTHER|CLEAN|TEMPFAIL|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
+#([:ccat|name|blocking]) #
+[? [:ccat|major|blocking] |#
+OTHER|CLEAN|MTA-BLOCKED|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
UNCHECKED|BANNED (%F)|INFECTED (%V)]#
-#([:ccat_maj],[:ccat_min])#
, %s -> [%O|,], Hits: %c#
, tag=[:tag_level], tag2=[:tag2_level], kill=[:kill_level]#
, %0/%1/%2/%k#
@@ -17502,9 +19904,10 @@ __DATA__
# Long header fields will be automatically wrapped by the program.
#
Subject: [?%#D|Undeliverable mail|Delivery status notification]\
-[? [:ccat_maj] |, CLEAN (other)||, TEMPFAIL\
+[? [:ccat|major] |||, MTA-BLOCKED\
|, OVERSIZED message\
-|, invalid header[?[:ccat_min]||: bad MIME|: unencoded 8-bit character\
+|, invalid header[=explain_badh|1]\
+[?[:ccat|minor]||: bad MIME|: unencoded 8-bit character\
|: improper use of control char|: all-whitespace header field\
|: header line longer than 998 characters|: header field syntax error\
|: missing required header field|: duplicate header field|]\
@@ -17528,7 +19931,7 @@ Our internal reference code for your mes
# ccat_min 0: other, 1: bad MIME, 2: 8-bit char, 3: NUL/CR,
# 4: empty, 5: long, 6: syntax, 7: missing, 8: multiple
-[? %#X ||[? [:ccat_min]
+[? [:explain_badh] ||[? [:ccat|minor]
|INVALID HEADER
|INVALID HEADER: BAD MIME HEADERS OR BAD MIME STRUCTURE
|INVALID HEADER: INVALID 8-BIT CHARACTERS IN HEADER
@@ -17559,7 +19962,7 @@ Our internal reference code for your mes
# ccat_min 0: other, 1: bad MIME, 2: 8-bit char, 3: NUL/CR,
# 4: empty, 5: long, 6: syntax, 7: missing, 8: multiple
-[? %#X ||[? [:ccat_min]
+[? [:explain_badh] ||[? [:ccat|minor]
|# 0: other
|# 1: bad MIME
|# 2: 8-bit char
@@ -17638,10 +20041,10 @@ __DATA__
# The From, To and Date header fields will be provided automatically.
# Long header fields will be automatically wrapped by the program.
#
-Subject: [? [:ccat_maj]
-|Clean message from you (other)\
-|Clean message from you (tempfail)\
+Subject: [? [:ccat|major]
|Clean message from you\
+|Clean message from you\
+|Clean message from you (MTA blocked)\
|OVERSIZED message from you\
|BAD-HEADER in message from you\
|SPAM apparently from you\
@@ -17653,7 +20056,7 @@ Subject: [? [:ccat_maj]
[? %m |#|In-Reply-To: %m]
Message-ID: <VS%i@%h>
-[? [:ccat_maj] |Clean (other)|Clean|TEMPFAIL|OVERSIZED|INVALID HEADER|\
+[? [:ccat|major] |Clean|Clean|MTA-BLOCKED|OVERSIZED|INVALID HEADER|\
spam|SPAM|UNCHECKED contents|BANNED CONTENTS ALERT|VIRUS ALERT]
Our content checker found
@@ -17733,11 +20136,11 @@ __DATA__
#
Date: %d
From: %f
-Subject: [? [:ccat_maj] |Clean (?) mail|Clean mail|TEMPFAIL-ed mail|\
+Subject: [? [:ccat|major] |Clean mail|Clean mail|MTA-blocked mail|\
OVERSIZED mail|INVALID HEADER in mail|spam|SPAM|UNCHECKED contents in mail|\
BANNED contents (%F) in mail|VIRUS (%V) in mail]\
FROM [?%l||LOCAL ][?%a||\[%a\] ][?%s|<>|[?%o|(?)|%s]]
-To: [? %#T |undisclosed-recipients: ;|[<%T>|, ]]
+To: [? %#T |undisclosed-recipients:;|[<%T>|, ]]
[? %#C |#|Cc: [<%C>|, ]]
Message-ID: <VA%i@%h>
@@ -17752,7 +20155,8 @@ Message-ID: <VA%i@%h>
|Scanner detecting a virus: %W
|Scanners detecting a virus: %W
]
-Content type: [:ccat_name] ([:ccat_maj],[:ccat_min])
+Content type: [:ccat|name|main]#
+[? [:ccat|is_blocked_by_nonmain] ||, blocked for [:ccat|name]]
Internal reference code for the message is %n/%i
[? %a |#|[:wrap|78|| |First upstream SMTP client IP address: \[%a\] %g]]
@@ -17790,12 +20194,12 @@ __DATA__
#
Date: %d
From: %f
-Subject: [? [:ccat_maj] |Clean (?) mail|Clean mail|TEMPFAIL-ed mail|\
+Subject: [? [:ccat|major] |Clean mail|Clean mail|MTA-blocked mail|\
OVERSIZED mail|INVALID HEADER in mail|SPAM|SPAM|UNCHECKED contents in mail|\
BANNED contents (%F) in mail|VIRUS (%V) in mail]\
TO YOU from [?%s|<>|[?%o|(?)|%s]]
-To: [? %#T |undisclosed-recipients: ;|[<%T>|, ]]
-[? %#C |#|Cc: [<%C>|, ]]
+[? [:header_field|To] |To: undisclosed-recipients:;|To: [:header_field|To]]
+[? [:header_field|Cc] |#|Cc: [:header_field|Cc]]
Message-ID: <VR%i@%h>
[? %#V |[? %#F ||BANNED CONTENTS ALERT]|VIRUS ALERT]
@@ -17809,6 +20213,8 @@ in an email to you [? %S |from unknown s
%o
[? %S |claiming to be: %s|#]
+Content type: [:ccat|name|main]#
+[? [:ccat|is_blocked_by_nonmain] ||, blocked for [:ccat|name]]
Our internal reference code for your message is %n/%i
[? %a |#|[:wrap|78|| |First upstream SMTP client IP address: \[%a\] %g]]
@@ -17889,11 +20295,12 @@ Date: %d
Date: %d
From: %f
Subject: SPAM FROM [?%l||LOCAL ][?%a||\[%a\] ][?%s|<>|[?%o|(?)|%s]]
-To: [? %#T |undisclosed-recipients: ;|[<%T>|, ]]
+To: [? %#T |undisclosed-recipients:;|[<%T>|, ]]
[? %#C |#|Cc: [<%C>|, ]]
-[? %#B |#|Bcc: [<%B>|, ]]
Message-ID: <SA%i@%h>
+Content type: [:ccat|name|main]#
+[? [:ccat|is_blocked_by_nonmain] ||, blocked for [:ccat|name]]
Internal reference code for the message is %n/%i
[? %a |#|[:wrap|78|| |First upstream SMTP client IP address: \[%a\] %g]]
diff --git a/amavisd-agent b/amavisd-agent
--- a/amavisd-agent
+++ b/amavisd-agent
@@ -5,7 +5,7 @@
# SNMP-like counters updated by amavisd-new.
#
# Author: Mark Martinec <mark.martinec at ijs.si>
-# Copyright (C) 2004 Mark Martinec, All Rights Reserved.
+# Copyright (C) 2004,2007 Mark Martinec, All Rights Reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
@@ -41,14 +41,18 @@
use strict;
use re 'taint';
+use warnings;
+no warnings 'uninitialized';
use Time::HiRes ();
use BerkeleyDB;
-use vars qw($VERSION); $VERSION = 2.401;
+use vars qw($VERSION); $VERSION = 2.530;
use vars qw(%values %virus_by_name);
use vars qw(%virus_by_os %spam_by_os %ham_by_os);
use vars qw(%history $avg_int $uptime);
+
+my($wakeuptime) = 10; # sleep time in seconds, may be fractional
$avg_int = 5*60; # 5 minute interval
@@ -57,13 +61,36 @@ sub p1($$@) {
printf("%-35s %6d %6.0f/h", $k, $values{$k}, $avg*3600);
for my $tot_k (@tot_k) {
if ($values{$tot_k} <= 0) {
- printf(" --- %%")
+ printf(" --- %%")
} else {
- printf(" %6.1f %%", 100*$values{$k}/$values{$tot_k})
+ printf(" %7.1f %%", 100*$values{$k}/$values{$tot_k})
}
print " ($tot_k)";
}
print "\n";
+}
+
+sub p1_size($$@) {
+ my($k,$avg, at tot_k) = @_;
+ my($scale) = 1024*1024;
+ printf("%-35s %6.0fMB %4.0fMB/h", $k, $values{$k}/$scale, $avg*3600/$scale);
+ for my $tot_k (@tot_k) {
+ if ($values{$tot_k} <= 0) {
+ printf(" --- %%")
+ } else {
+ printf(" %5.1f %%", 100*$values{$k}/$values{$tot_k})
+ }
+ print " ($tot_k)";
+ }
+ print "\n";
+}
+
+sub p1_time($$$$) {
+ my($k,$dv,$dcnt,$tot_k) = @_;
+ printf("%-35s %6.0f s %8s s/msg (%s)\n",
+ $k, $values{$k}/1000,
+ $dcnt < 1 ? "---" : sprintf("%7.3f",$dv/1000/$dcnt),
+ $tot_k);
}
sub p2($$$$) {
@@ -74,8 +101,8 @@ sub p2($$$$) {
}
}
-sub enqueue($$$$) {
- my($name,$now,$val,$hold_time) = @_;
+sub enqueue($$$$$) {
+ my($name,$now,$val,$msgcnt,$hold_time) = @_;
if (ref $history{$name} ne 'ARRAY') { $history{$name} = [] }
my($oldest_useful);
for my $j (0..$#{$history{$name}}) {
@@ -86,16 +113,18 @@ sub enqueue($$$$) {
@{$history{$name}} =
@{$history{$name}}[$oldest_useful..$#{$history{$name}}];
}
- push(@{$history{$name}}, [$now,$val]);
- my($average,$dv,$dt); my($n) = scalar(@{$history{$name}});
+ push(@{$history{$name}}, [$now,$val,$msgcnt]);
+ my($average,$dv,$dt,$dcnt); my($n) = scalar(@{$history{$name}});
my($oldest) = $history{$name}->[0];
my($latest) = $history{$name}->[$n-1];
- $dt = $latest->[0] - $oldest->[0]; $dv = $latest->[1] - $oldest->[1];
+ $dt = $latest->[0] - $oldest->[0];
+ $dv = $latest->[1] - $oldest->[1];
+ $dcnt = $latest->[2] - $oldest->[2];
if ($n < 2 || $dt < $hold_time/2) {
- $dt = $uptime; $dv = $val; # average since the start time
+ $dt = $uptime; $dv = $val; $dcnt = $msgcnt; # average since the start time
}
if ($dt > 0) { $average = $dv/$dt }
- ($average, $dv, $dt, $n);
+ ($average, $dv, $dt, $dcnt, $n);
}
sub fmt_ticks($) {
@@ -110,6 +139,12 @@ sub fmt_ticks($) {
# main program starts here
$SIG{INT} = sub { die "\n" }; # do the END code block
+ if (@ARGV > 0) {
+ if (@ARGV == 2 && $ARGV[0] eq '-w' && $ARGV[1] =~ /^\+?\d+(?:\.\d*)?\z/)
+ { $wakeuptime = $ARGV[1] }
+ else
+ { die "Usage: $0 [ -w <wait-interval> ]\n" }
+ }
my($env) = BerkeleyDB::Env->new(
'-Home'=>'/var/amavis/db', '-Flags'=> DB_INIT_CDB | DB_INIT_MPOOL);
defined $env or die "BDB no env: $BerkeleyDB::Error $!";
@@ -118,12 +153,12 @@ sub fmt_ticks($) {
defined $db or die "BDB no dbS 1: $BerkeleyDB::Error $!";
my($cursor);
- $| = 1;
my($stat,$key,$val);
for (;;) {
%values = (); %virus_by_name = ();
%virus_by_os = (); %spam_by_os = (); %ham_by_os = ();
my($now); my($eval_stat,$interrupt); $interrupt = '';
+ $| = 0;
print "\n\n";
{ my($h1) = sub { $interrupt = $_[0] };
local(@SIG{qw(INT HUP TERM TSTP QUIT ALRM USR1 USR2)}) = ($h1) x 8;
@@ -152,7 +187,7 @@ sub fmt_ticks($) {
if ($interrupt ne '') { kill($interrupt,$$) } # resignal
elsif ($eval_stat ne '') { chomp($eval_stat); die "BDB $eval_stat\n" }
for my $k (sort keys %values) {
- if ($values{$k} =~ /^C32 (.*)\z/) {
+ if ($values{$k} =~ /^(?:C32|C64) (.*)\z/) {
$values{$k} = $1;
} elsif ($k eq 'sysUpTime' && $values{$k} =~ /^INT (.*)\z/) {
$uptime = $now - $1; my($ticks) = int($uptime*100);
@@ -164,28 +199,41 @@ sub fmt_ticks($) {
delete($values{$k});
}
}
+ my($msgcnt) = $values{'InMsgs'};
for (sort keys %values) {
- my($avg,$dv,$dt,$n) = enqueue($_, $now, $values{$_}, $avg_int);
+ my($avg,$dv,$dt,$dcnt,$n) =
+ enqueue($_, $now, $values{$_}, $msgcnt, $avg_int);
if (/^OpsDecTyp/) {} # later
elsif (/^CacheHitsVirusMsgs$/) { p1($_,$avg,'ContentVirusMsgs') }
elsif (/^CacheHitsBannedMsgs$/) { p1($_,$avg,'ContentBannedMsgs') }
elsif (/^CacheHitsSpamMsgs$/) { p1($_,$avg,'ContentSpamMsgs') }
- elsif (/^Cache/) { p1($_,$avg,'CacheAttempts') }
- elsif (/^Content/) { p1($_,$avg,'InMsgs') }
- elsif (/^Quar/) { p1($_,$avg,'QuarMsgs') }
- elsif (/^OpsSql/) { p1($_,$avg,'InMsgsRecips') }
- elsif (/^(InMsgs|Ops)/) { p1($_,$avg,'InMsgs') }
- elsif (/^Out/) { p1($_,$avg,'OutMsgs') }
- elsif (/^SqlAddrSender/){ p1($_,$avg,'SqlAddrSender') }
- elsif (/^SqlAddrRecip/) { p1($_,$avg,'SqlAddrRecip') }
- else { p1($_,$avg,undef) }
+ elsif (/^Cache/) { p1($_,$avg,'CacheAttempts') }
+ elsif (/^Content(.*?)Msgs./) { p1($_,$avg,'Content'.$1.'Msgs') }
+ elsif (/^Content/) { p1($_,$avg,'InMsgs') }
+ elsif (/^Quar/) { p1($_,$avg,'QuarMsgs') }
+ elsif (/^OpsSql/) { p1($_,$avg,'InMsgsRecips') }
+ elsif (/^InMsgsSize/) { p1_size($_,$avg,'InMsgsSize') }
+ elsif (/^InMsgsRecips(.*)\z/) { p1($_,$avg,'InMsgs'.$1) }
+ elsif (/^(InMsgs|Ops)/) { p1($_,$avg,'InMsgs') }
+ elsif (/^Out/) { p1($_,$avg,'OutMsgs') }
+ elsif (/^Log/) { p1($_,$avg,'InMsgs') }
+ elsif (/^PenPalsAttempts\z/){ p1($_,$avg,'InMsgsRecipsInboundOrInt') }
+ elsif (/^PenPalsHits\z/) { p1($_,$avg,'ContentCleanMsgsInboundOrInt')}
+ elsif (/^PenPalsHits./) { p1($_,$avg,'PenPalsHits') }
+ elsif (/^PenPals/) { p1($_,$avg,'PenPalsAttempts') }
+ elsif (/^SqlAddrSender/) { p1($_,$avg,'SqlAddrSender') }
+ elsif (/^SqlAddrRecip/) { p1($_,$avg,'SqlAddrRecip') }
+ elsif (/^TimeElapsed/i) { p1_time($_,$dv,$dcnt,'InMsgs') }
+ else { p1($_,$avg,undef) }
}
for (sort { $values{$b}<=>$values{$a} } grep {/^OpsDecTyp/} keys %values) {
- my($avg,$dv,$dt,$n) = enqueue($_, $now, $values{$_}, $avg_int);
+ my($avg,$dv,$dt,$dcnt,$n) =
+ enqueue($_, $now, $values{$_}, $msgcnt, $avg_int);
p1($_,$avg,'InMsgs');
}
for my $href (\%virus_by_name,\%virus_by_os,\%spam_by_os,\%ham_by_os) {
- for (keys %$href) { $href->{$_} = $1 if $href->{$_} =~ /^C32 (.*)\z/ }
+ for (keys %$href)
+ { $href->{$_} = $1 if $href->{$_} =~ /^(?:C32|C64) (.*)\z/ }
}
for my $href (\%virus_by_os,\%spam_by_os,\%ham_by_os) {
for (keys %$href) {
@@ -201,29 +249,34 @@ sub fmt_ticks($) {
my($href,$tot_k) = @$pair;
for (sort {$href->{$b} <=> $href->{$a}} keys %$href) {
if (!$separated) { print "\n"; $separated = 1 }
- my($avg,$dv,$dt,$n) = enqueue($_, $now, $href->{$_}, $avg_int);
+ my($avg,$dv,$dt,$dcnt,$n) =
+ enqueue($_, $now, $href->{$_}, $msgcnt, $avg_int);
p2($_,$avg,$tot_k,$href);
}
}
- $separated = 0;
- for my $href (\%virus_by_os, \%spam_by_os, \%ham_by_os) {
- for (sort {$href->{$b} <=> $href->{$a}} keys %$href) {
+ if (0) { # disabled
+ $separated = 0;
+ for my $href (\%virus_by_os, \%spam_by_os, \%ham_by_os) {
+ for (sort {$href->{$b} <=> $href->{$a}} keys %$href) {
+ if (!$separated) { print "\n"; $separated = 1 }
+ my($avg,$dv,$dt,$dcnt,$n) =
+ enqueue($_, $now, $href->{$_}, $msgcnt, $avg_int);
+ /^[a-zA-Z]+\.byOS\.(.*)\z/; my($os) = $1;
+ p2($_,$avg,"all.byOS.$os",$href);
+ }
+ }
+ $separated = 0;
+ for (sort { $values{$b}<=>$values{$a} }
+ grep {/^all\.byOS\./} keys %values) {
if (!$separated) { print "\n"; $separated = 1 }
- my($avg,$dv,$dt,$n) = enqueue($_, $now, $href->{$_}, $avg_int);
- /^[a-zA-Z]+\.byOS\.(.*)\z/; my($os) = $1;
- p2($_,$avg,"all.byOS.$os",$href);
- }
- }
- $separated = 0;
- for (sort { $values{$b}<=>$values{$a} }
- grep {/^all\.byOS\./} keys %values) {
- if (!$separated) { print "\n"; $separated = 1 }
- my($avg,$dv,$dt,$n) = enqueue($_, $now, $values{$_}, $avg_int);
- p1($_,$avg,'InMsgs');
- }
- sleep 10;
-# Time::HiRes::sleep 0.5;
- }
+ my($avg,$dv,$dt,$dcnt,$n) =
+ enqueue($_, $now, $values{$_}, $msgcnt, $avg_int);
+ p1($_,$avg,'InMsgs');
+ }
+ }
+ $| = 1;
+ Time::HiRes::sleep($wakeuptime);
+ } # forever
END {
if (defined $db) {
diff --git a/amavisd-custom.conf b/amavisd-custom.conf
new file mode 100644
--- /dev/null
+++ b/amavisd-custom.conf
@@ -0,0 +1,281 @@
+package Amavis::Custom;
+use strict;
+
+# Example use of custom hooks, available since amavisd-new-2.5.0
+
+# This code can be placed directly at end of file amavisd.conf,
+# or invoked from there by a call to include_config_files such as:
+# include_config_files('/etc/amavisd-custom.conf');
+# or specified on amavisd command line by using additional -c options.
+#
+# It replaces default hooks in package Amavis::Custom (in file amavisd)
+# with replacement subroutines of the same name, and thus enable them.
+#
+# The code below demonstrates obtaining and displaying some of the
+# more interesting information on each passing mail.
+# The example below also illustrates how to use already provided code in
+# amavisd to interface with a SQL database server (e.g. MySQL or PostgreSQL),
+# allowing for persistent connections and automatic reconnect in case
+# of a connection failure.
+#
+# Modifying recipient address, sending a copy to a mailbox quarantine,
+# or creating and sending a short notification alert is illustrated.
+
+
+#testing database:
+# $ mysqladmin create user_presence
+# $ mysql user_presence
+# CREATE TABLE users (
+# email varchar(255) NOT NULL UNIQUE,
+# present char(1)
+# );
+# INSERT INTO users VALUES ('test at example.com', 'Y');
+# INSERT INTO users VALUES ('absent at example.com', 'N');
+# INSERT INTO users VALUES ('postmaster at example.com', 'Y');
+
+
+# replaces placeholder routines in Amavis::Custom with actual code
+
+use DBI qw(:sql_types);
+use DBD::mysql;
+BEGIN {
+ import Amavis::Conf qw(:platform :confvars c cr ca $myhostname);
+ import Amavis::Util qw(do_log untaint safe_encode safe_decode);
+ import Amavis::rfc2821_2822_Tools;
+ import Amavis::Notify qw(string_to_mime_entity);
+}
+
+# MAIL PROCESSING SEQUENCE:
+#
+# child process initialization
+#*custom hook: new()
+# loop for each mail:
+# receive mail
+# mail checking and collecting results
+# *custom hook: checks() - may inspect or modify checking results
+# deciding mail fate (lookup on *_lovers, thresholds, ...)
+# quarantining
+# sending notifications (to admin and recip)
+# *custom hook: before_send() - may send other notif., quarantine, modify mail
+# forwarding (unless blocked)
+# sending delivery status notification (if needed)
+# issue main log entry, manage statistics (timing, counters, nanny)
+# *custom hook: mail_done() - may inspect results
+# endloop after $max_requests or earlier
+
+# invoked at child process creation time;
+# return an object, or just undef when custom checks are not needed
+sub new {
+ my($class,$conn,$msginfo) = @_;
+ my($self) = bless {}, $class;
+ my($conn_h) = Amavis::Out::SQL::Connection->new(
+ ['DBI:mysql:database=user_presence', 'root', ''] );
+# ['DBI:mysql:database=user_presence;host=127.0.0.1', 'user1', 'passwd1'] );
+ $self->{'conn_h'} = $conn_h;
+ $self; # returning an object activates further callbacks,
+ # returning undef disables them
+}
+
+#sub checks { # may be left out if not needed
+# my($self,$conn,$msginfo) = @_;
+#}
+
+sub before_send {
+ my($self,$conn,$msginfo) = @_;
+ # $self ... whatever was returned by new()
+ # $conn ... object with information about a SMTP connection
+ # $msginfo ... object with info. about a mail message being processed
+
+ my($ll) = 2; # log level (0 is the most important level, 1, 2,... 5 less so)
+ do_log($ll,"CUSTOM: new message");
+
+ # examine some data pertaining to the SMTP connection from client
+ # See methods in Amavis::In::Connection for the full set of available data.
+ #
+ # SMTP client's IP address as a string (IPv4 or IPv6)
+ my($client_ip) = $msginfo->client_addr;
+ # does client IP address match @mynetworks_maps? (boolean)
+ my($is_client_ip_internal) = $msginfo->client_addr_mynets;
+ do_log($ll,"CUSTOM: [%s], is internal IP: %s",
+ $client_ip, $is_client_ip_internal ? 'YES' : 'NO');
+
+ # examine some data pertaining to the message as a whole (not per-pecipient)
+ # See methods in Amavis::In::Message for the full set of available data.
+ #
+ my($log_id) = $msginfo->log_id; # log ID string, e.g. '48262-21-2'
+ my($mail_id) = $msginfo->mail_id; # long-term unique id, e.g. 'yxqmZgS+M09R'
+ my($sender) = $msginfo->sender; # envelope sender address, e.g. 'usr at e.com'
+ my($mail_size) = $msginfo->msg_size; # mail size in bytes
+ my($spam_level)= $msginfo->spam_level; # spam level (without per-recip boost)
+ do_log($ll,"CUSTOM: %d bytes, score: %.2f",
+ $log_id,$mail_id,$mail_size,$spam_level);
+ do_log($ll,"CUSTOM: Return-Path (env. sender): <%s>", $sender);
+
+ # full mail is only stored in file, which may be read if desired (see below);
+ # full mail header is available in ->orig_header;
+
+ # some mail header fields are available through $msginfo->orig_header_fields
+ # these may be multiline, may contain folding whitespace or comments;
+ # alternatively, the whole original mail header is available in ->orig_header
+ my($m_id) = $msginfo->orig_header_fields->{'message-id'}; # e.g. <12 at e.n>
+ my($subj) = $msginfo->orig_header_fields->{'subject'};
+ my($from) = $msginfo->orig_header_fields->{'from'};
+ # e.g.: "=?ISO-8859-1?Q?Ren=E9_van_den_Berg?=" <vd at example.com>
+ my($is_bulk) = $msginfo->orig_header_fields->{'precedence'}; # e.g. List
+ $is_bulk = $is_bulk=~/^[ \t]*(bulk|list|junk)\b/i ? $1 : undef;
+ for ($m_id,$from,$subj) { # RFC2047-decode char. sets in some header fields
+ local($1); chomp; my($str);
+ s/\n([ \t])/$1/sg; s/^[ \t]+//s; s/[ \t]+\z//s; # unfold, trim
+ eval { $str = safe_decode('MIME-Header',$_) }; # to string of logical chr
+ $_ = $str if $@ eq ''; # replace if all ok, otherwise keep unchanged
+ }
+ # $m_id, $from, and $subj are now ready for examination - Perl logical chars
+ do_log($ll,"CUSTOM: Subject: %s",safe_encode('iso-8859-1',$subj)); #as Latin1
+ do_log($ll,"CUSTOM: From: %s", safe_encode('iso-8859-1',$from)); # as Latin1
+ # NOTE: rfc2822 allows multiple addresses in the From field!
+ my($rfc2822_sender) = $msginfo->rfc2822_sender; # undef or scalar
+ my($rfc2822_from) = $msginfo->rfc2822_from; # undef, scalar or listref
+ $rfc2822_from = join(', ',@$rfc2822_from) if ref $rfc2822_from;
+ do_log($ll,"CUSTOM: From (parsed): %s", $rfc2822_from);
+ do_log($ll,"CUSTOM: Sender: %s", $rfc2822_sender) if defined $rfc2822_sender;
+
+ my($tempdir) = $msginfo->mail_tempdir; # working directory for this process
+ # $tempdir/parts/ is a directory where mail parts were extracted to
+ my($mail_file_name) = $msginfo->mail_text_fn;
+ # filename of the original mail, normally $tempdir/email.txt
+ do_log($ll,"CUSTOM: temp.dir: %s", $tempdir);
+ do_log($ll,"CUSTOM: filename: %s", $mail_file_name);
+
+ # full mail header is available in ->orig_header;
+ # some individual header fields are quickly accessible ->orig_header_fields
+
+ # mail body is only stored in file, which may be read if desired
+ my($fh) = $msginfo->mail_text; # file handle of our original mail
+ my($line); my($line_cnt) = 0;
+# $fh->seek(0,0) or die "Can't rewind mail file: $!";
+# for ($! = 0; defined($line = $fh->getline); $! = 0) {
+# $line_cnt++;
+# # examine one $line at a time; (or read by blocks for speed)
+# }
+# defined $line || $!==0 or die "Error reading mail file: $!";
+# do_log($ll,"CUSTOM: %d lines", $line_cnt);
+
+ # examine some data pertaining to the each recipient of the message
+ # See methods in Amavis::In::Message::PerRecip for the full set of data.
+ #
+ my($any_passed) = 0;
+ for my $r (@{$msginfo->per_recip_data}) { # $r contains per-recipient data
+ next if $r->recip_done; # skip recipient that won't receive a message
+ # if all recipients have ->recip_done true, mail will not be passed at all
+ $any_passed++;
+ my($recip) = $r->recip_addr; # recipient envelope address, e.g. rc at ex.com
+ my($is_local) = $r->recip_is_local; # recipient matches @local_domains_maps
+ my($localpart,$domain) = split_address($recip);
+
+ my($spam_level_boost) = $r->recip_score_boost; # per-recip score contrib.
+ # $spam_level + $spam_level_boost is the actual per-recipient spam score
+ my($do_tag) = $r->is_in_contents_category(CC_CLEAN,1); # >= tag_level
+ my($do_tag2) = $r->is_in_contents_category(CC_SPAMMY); # >= tag2_level
+ my($do_kill) = $r->is_in_contents_category(CC_SPAM); # >= kill_level
+ do_log($ll,"CUSTOM: recip: %s, score: %.2f, %s, %s, %s, %s",
+ $recip, $spam_level+$spam_level_boost,
+ $is_local ? 'IS LOCAL' : 'not local',
+ $do_tag ? 'tag' : 'no-tag',
+ $do_tag2 ? 'tag2' : 'no-tag2',
+ $do_kill ? 'kill' : 'no-kill');
+
+ # don't bother with outgoing mail!
+ next if !$is_local;
+
+ # do a SQL lookup
+ my($conn_h) = $self->{'conn_h'};
+ $conn_h->begin_work_nontransaction; # (re)connect if not connected
+ #
+ my($select_clause) =
+ 'SELECT present,email FROM users WHERE users.email=?';
+ # list of actual arguments replacing '?' placeholders
+ my(@pos_args) = ( lc(untaint($recip)) );
+ $conn_h->execute($select_clause, at pos_args); # do the query
+ #
+ my($a_ref); my($user_is_offline);
+ while ( defined($a_ref=$conn_h->fetchrow_arrayref($select_clause)) ) {
+ do_log($ll,"CUSTOM: SQL fields %s", join(", ", @$a_ref));
+ $user_is_offline = 1 if $a_ref->[0] =~ /^(0|N)$/i;
+ }
+ $conn_h->finish($select_clause) if defined $a_ref; # only if not all read
+
+ if ($user_is_offline) {
+ # we have three choices of alerting the recipient:
+ # - redirect his mail to dedicated e-mail address;
+ # - use quarantining code to deliver a copy of the message to
+ # a dedicated address;
+ # - construct and send a notification to a dedicated address
+ #
+ my($choice) = 0;
+ if ($choice == 0) {
+ # ignore
+ } elsif ($choice == 1) {
+ # rewrite address and deliver normally
+ my($new_addr) = $localpart . '+redirect' . $domain;
+ $r->recip_addr_modified($new_addr); # replaces delivery address!
+ } elsif ($choice == 2) {
+ # quarantine (i.e. send a mail copy) to a dedicated mailbox
+ # in addition to delivering normally
+ my($new_addr) = 'alert+' . $localpart . $domain;
+ Amavis::do_quarantine($conn, $msginfo, undef,
+ [$new_addr], 'local:all-%m');
+ } elsif ($choice == 3) {
+ # construct and send a short notification,
+ # in addition to delivering normally
+ my($when) = rfc2822_timestamp($msginfo->rx_time);
+ my($text) = <<"EOD";
+From: Alerting Service <alerter\@$myhostname>
+To: <$recip>
+Subject: New message from $sender
+Message-ID: <AA$log_id\@$myhostname>
+
+A new message just arrived on $when
+from $from (return-path <$sender>)
+Subject: $subj
+EOD
+ my($notification) = Amavis::In::Message->new;
+ $notification->rx_time($msginfo->rx_time); # copy the reception time
+ $notification->log_id($log_id); # copy log id
+ $notification->delivery_method(c('notify_method'));
+ $notification->sender(''); # use null return path to avoid loops
+ $notification->sender_smtp('<>');
+ my($new_addr) = 'alert+' . $localpart . $domain;
+ $notification->recips([$new_addr]);
+ # character set is controlled through $hdr_encoding and $bdy_encoding
+ # config variables, defaults to 'iso-8859-1'
+ $notification->mail_text(
+ string_to_mime_entity(\$text, $msginfo, undef,0,0));
+ Amavis::mail_dispatch($conn, $notification, 1, 0);
+ my($n_smtp_resp, $n_exit_code, $n_dsn_needed) =
+ one_response_for_all($notification, 0); # check status
+ if ($n_smtp_resp =~ /^2/ && !$n_dsn_needed) { # ok
+ } elsif ($n_smtp_resp =~ /^4/) {
+ die "temporarily unable to alert recipient: $n_smtp_resp";
+ } else {
+ do_log(-1, "FAILED to alert recipient: %s", $n_smtp_resp);
+ }
+ }
+ }
+
+ }
+ if (!$any_passed) {
+ do_log($ll,"CUSTOM: mail is blocked for all recipients");
+ } else { # will do delivery
+ do_log($ll,"CUSTOM: being delivered to %d recips", $any_passed);
+ # add a custom header field if desired (for all recipients of this message)
+ # $msginfo->header_edits->add_header('X-Amavis-Example',
+ # sprintf("a custom header field, mail contains %d lines",$line_cnt) );
+ }
+ do_log($ll,"CUSTOM: done");
+};
+
+#sub mail_done { # may be left out if not needed
+# my($self,$conn,$msginfo) = @_;
+#}
+
+1; # insure a defined return
diff --git a/amavisd-nanny b/amavisd-nanny
--- a/amavisd-nanny
+++ b/amavisd-nanny
@@ -5,7 +5,7 @@
# and keep an eye on the health of child processes in amavisd-new.
#
# Author: Mark Martinec <mark.martinec at ijs.si>
-# Copyright (C) 2004 Mark Martinec, All Rights Reserved.
+# Copyright (C) 2004,2007 Mark Martinec, All Rights Reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
@@ -41,24 +41,39 @@
use strict;
use re 'taint';
+use warnings;
+no warnings 'uninitialized';
use POSIX qw(strftime);
+use Errno qw(ESRCH);
+use Time::HiRes ();
use BerkeleyDB;
-use vars qw($VERSION); $VERSION = 1.02;
-
-my($idlettl) = 60*60; # idle children are sent a SIGTERM after this many seconds
-my($activettl) = 10*60; # stuck active children are sent a SIGTERM after this
- # many seconds
+use vars qw($VERSION); $VERSION = 1.200;
+
+my($idlettl) = 3*60*60; # idle children are sent a SIGTERM
+ # after this many seconds
+my($activettl) = 10*60; # stuck active children are sent a SIGTERM
+ # after this many seconds
my($db_home) = '/var/amavis/db'; # DB databases directory
my($dbfile) = 'nanny.db';
-my($wakeuptime) = 2; # seconds
-
-sub fmt_age($$) {
- my($t,$char) = @_;
- my($bar);
- $bar = substr(($char x 9 . ':') x 3 . $char x 5 . '>', 0, $t) if $char ne '';
+my($wakeuptime) = 2; # sleep time in seconds, may be fractional
+
+sub fmt_age($$$) {
+ my($t,$state_bar,$idling) = @_;
+ $t = int($t);
+ my($char) = $idling ? '.' : '=';
+ my($bar_l) = $idling ? $t : length($state_bar);
+ my($bar) = substr( ($char x 9 . ':') x 3 . $char x 5, 0,$bar_l);
+ if (!$idling) {
+ $state_bar = substr($state_bar,0,length($bar)-2) . substr($state_bar,-1,1)
+ . '>' if length($state_bar) > length($bar);
+ for my $j (0 .. length($bar)-1) {
+ substr($bar,$j,1) = substr($state_bar,$j,1)
+ if substr($bar,$j,1) eq '=' && substr($state_bar,$j,1) ne ' ';
+ }
+ }
my($s) = $t % 60; $t = int($t/60);
my($m) = $t % 60; $t = int($t/60);
my($h) = $t % 24; $t = int($t/24);
@@ -69,7 +84,36 @@ sub fmt_age($$) {
};
# main program starts here
- $SIG{INT} = sub { die "\n" }; # do the END code block
+ $SIG{INT} = sub { die "\n" }; # do the END code block when interrupted
+ if (@ARGV > 0) {
+ if (@ARGV == 2 && $ARGV[0] eq '-w' && $ARGV[1] =~ /^\+?\d+(?:\.\d*)?\z/)
+ { $wakeuptime = $ARGV[1] }
+ else {
+ print <<'EOD';
+States legend:
+ A accepted a connection
+ b begin with a protocol for accepting a request
+ m 'MAIL FROM' smtp command started a new transaction in the same session
+ d data transfer from MTA to amavisd
+ = content checking just started
+ D decoding of mail parts
+ V virus scanning
+ S spam scanning
+ P pen pals database lookup and updates
+ r preparing results
+ Q quarantining and preparing/sending notifications
+ F forwarding mail to MTA
+ . content checking just finished
+ sp space indicates idle (elapsed bar is showing dots)
+
+EOD
+ die "Usage: $0 [ -w <wait-interval> ]\n";
+ }
+ }
+ print <<'EOD';
+process-id task-id elapsed in elapsed-bar (dots indicate idle)
+ or state idle or busy
+EOD
my($env) = BerkeleyDB::Env->new(
'-Home'=>$db_home, '-Flags'=> DB_INIT_CDB | DB_INIT_MPOOL);
defined $env or die "BDB no env: $BerkeleyDB::Error $!";
@@ -80,10 +124,11 @@ sub fmt_age($$) {
my(%waittime); # associative array on pid
+ my(%proc_last_timestamp, %proc_state_bars);
for (;;) {
$| = 0;
print "\n";
- my(%proc_timestamp, %proc_task_id);
+ my(%proc_timestamp, %proc_state, %proc_task_id);
my($stat,$key,$val); my($now);
my($eval_stat,$interrupt); $interrupt = '';
{ my($h1) = sub { $interrupt = $_[0] };
@@ -91,13 +136,19 @@ sub fmt_age($$) {
eval {
$cursor = $db->db_cursor; # obtain read lock
defined $cursor or die "db_cursor error: $BerkeleyDB::Error";
- $now = time;
- my($now_utc_iso8601) = strftime("%Y%m%dT%H%M%S",gmtime($now));
+ $now = Time::HiRes::time; local($1,$2);
+ my($now_utc_iso8601) = strftime("%Y%m%dT%H%M%S",gmtime(int($now)));
while ( ($stat=$cursor->c_get($key,$val,DB_NEXT)) == 0 ) {
- if ($val !~ /^(\d+) (.*?) *\z/s) {
+ if ($val !~ /^(\d+(?:\.\d*)?) (.*?) *\z/s) {
print STDERR "Bad db entry: $key, $val\n";
} else {
- ($proc_timestamp{$key}, $proc_task_id{$key}) = ($1,$2);
+ $proc_timestamp{$key} = $1; my($task_id) = $2;
+ $proc_state{$key} = $1 if $task_id =~ s/^([^0-9])//;
+ $proc_task_id{$key} = $task_id;
+ if (!exists $proc_state_bars{$key}) { # new process appeared
+ $proc_last_timestamp{$key} = $proc_timestamp{$key};
+ $proc_state_bars{$key} = '';
+ }
}
}
$stat==DB_NOTFOUND or die "c_get: $BerkeleyDB::Error $!";
@@ -110,27 +161,54 @@ sub fmt_age($$) {
$cursor = undef;
}
}
- if ($interrupt ne '') { kill($interrupt,$$) } # resignal
+ if ($interrupt ne '') { kill($interrupt,$$) } # resignal, ignoring status
elsif ($eval_stat ne '') { chomp($eval_stat); die "BDB $eval_stat\n" }
+ for my $key (keys(%proc_state_bars)) { # remove old entries
+ if (!exists($proc_timestamp{$key})) {
+ delete $proc_timestamp{$key};
+ delete $proc_task_id{$key};
+ delete $proc_state_bars{$key};
+ }
+ }
my(@to_be_removed, at killed);
for my $pid (sort {$a<=>$b} keys %proc_timestamp) {
- my($idling) = $proc_task_id{$pid} =~ /^\.?\z/s;
+ $proc_state{$pid} = ' ' if $proc_state{$pid} eq '';
+ my($idling) = $proc_task_id{$pid} eq '' &&
+ $proc_state{$pid} =~ /^[. ]?\z/s;
+ my($age) = $now - $proc_timestamp{$pid};
+ if ($idling) { $proc_state_bars{$pid} = '' }
+ else {
+ $proc_state_bars{$pid} = ''
+ if $proc_timestamp{$pid} ne $proc_last_timestamp{$pid};
+ my($len) = int($age+0.5);
+ $len = 1 if $len < 1;
+ my($str) = $proc_state_bars{$pid};
+ if ($len > length($str)) { # replicate last character to desired size
+ my($ch) = $str eq '' ? '=' : substr($str,-1,1);
+ $str .= $ch x ($len - length($str));
+ }
+ substr($str,$len-1,1) = $proc_state{$pid};
+ $proc_state_bars{$pid} = $str;
+ }
+ $proc_last_timestamp{$pid} = $proc_timestamp{$pid};
my($ttl) = $idling ? $idlettl : $activettl;
- if (!kill(0,$pid)) { # test if the process is still there
+ my($n) = kill(0,$pid); # test if the process is still there
+ if ($n == 0 && $! != ESRCH) {
+ die "Can't check the process $pid: $!";
+ } elsif ($n == 0) { # ESRCH means there is no such process
printf("PID %s: %-11s went away %s\n",
- $pid, $proc_task_id{$pid},
- fmt_age($now-$proc_timestamp{$pid}, $idling?'.':'=') );
+ $pid, $proc_task_id{$pid} || $proc_state{$pid},
+ fmt_age($age, $proc_state_bars{$pid}, $idling) );
push(@to_be_removed, $pid);
- } elsif ($now <= $proc_timestamp{$pid}+$ttl) { # all ok
+ } elsif ($age <= $ttl) { # all ok
printf("PID %s: %-11s %s\n",
- $pid, $proc_task_id{$pid},
- fmt_age($now-$proc_timestamp{$pid}, $idling?'.':'=') );
+ $pid, $proc_task_id{$pid} || $proc_state{$pid},
+ fmt_age($age, $proc_state_bars{$pid}, $idling) );
} else { # send a SIGTERM
printf("PID %s: %-11s terminated %s\n",
- $pid, $proc_task_id{$pid},
- fmt_age($now-$proc_timestamp{$pid}, $idling?'.':'=') );
-
- if (kill('TERM',$pid)) { push(@killed,$pid) }
+ $pid, $proc_task_id{$pid} || $proc_state{$pid},
+ fmt_age($age, $proc_state_bars{$pid}, $idling) );
+ if (kill('TERM',$pid) || $! == ESRCH) { push(@killed,$pid) }
else { warn "Can't SIGTERM $pid: $!" }
}
}
@@ -151,14 +229,17 @@ sub fmt_age($$) {
}
my($delay) = 1; # seconds
while (@killed) {
- sleep $delay; $delay = 2;
+ Time::HiRes::sleep($delay); $delay = 2;
for my $pid (@killed) {
$waittime{$pid}++;
- printf("PID %s: sending SIGKILL in %d s\n", $pid, 30 - $waittime{$pid});
+ printf("PID %s: sending SIGKILL in %d s\n", $pid, 30-$waittime{$pid});
if ($waittime{$pid} > 30) { # send a SIGKILL
- kill('KILL',$pid) or warn "Can't SIGKILL $pid: $!";
+ kill('KILL',$pid) or $! == ESRCH or warn "Can't SIGKILL $pid: $!";
$waittime{$pid} = 0;
- } elsif (!kill(0,$pid)) { # no longer around
+ } elsif (kill(0,$pid)) { # process is still there
+ } elsif ($! != ESRCH) { # problem?
+ warn "Can't check process $pid: $!";
+ } else { # no longer around
printf("PID %s: %-11s successfully killed\n", $pid);
delete($waittime{$pid});
$pid = undef;
@@ -169,7 +250,7 @@ sub fmt_age($$) {
join(', ', at killed)) if @killed;
}
$| = 1;
- sleep $wakeuptime;
+ Time::HiRes::sleep($wakeuptime);
} # forever
END {
diff --git a/amavisd-new-courier.patch b/amavisd-new-courier.patch
--- a/amavisd-new-courier.patch
+++ b/amavisd-new-courier.patch
@@ -1,20 +1,27 @@
---- amavisd.ori Sat Sep 30 11:25:44 2006
-+++ amavisd Sat Sep 30 11:25:55 2006
-@@ -95,5 +95,5 @@
+--- amavisd.ori Wed Jun 27 12:40:07 2007
++++ amavisd Wed Jun 27 12:40:18 2007
+@@ -97,5 +97,5 @@
# Amavis::In::AMCL
# Amavis::In::SMTP
-#( Amavis::In::Courier )
+# Amavis::In::Courier
+ # Amavis::Out::SMTP::Protocol
# Amavis::Out::SMTP
- # Amavis::Out::Pipe
-@@ -141,5 +141,5 @@
+@@ -155,5 +155,5 @@
fetch_modules('REQUIRED BASIC MODULES', 1, qw(
Exporter POSIX Fcntl Socket Errno Carp Time::HiRes
- IO::Handle IO::File IO::Socket IO::Socket::UNIX IO::Socket::INET
+ IO::Handle IO::File IO::Select IO::Socket IO::Socket::UNIX IO::Socket::INET
IO::Wrap IO::Stringy Digest::MD5 Unix::Syslog File::Basename
- Mail::Field Mail::Address Mail::Header Mail::Internet Compress::Zlib
-@@ -7231,4 +7231,34 @@
+ Compress::Zlib MIME::Base64 MIME::QuotedPrint MIME::Words
+@@ -7891,5 +7891,5 @@
+ ### but before binding to sockets
+ sub post_configure_hook {
+-# umask(0007); # affect protection of Unix sockets created by Net::Server
++ umask(0007); # affect protection of Unix sockets created by Net::Server
+ }
+
+@@ -7902,4 +7902,34 @@
### Net::Server hook
+### This hook takes place immediately after the "->run()" method is called.
@@ -48,8 +55,8 @@
+
+### Net::Server hook
### This hook occurs in the parent (master) process after chroot,
- ### change of user, and change of group has occured. It allows
-@@ -7281,4 +7311,15 @@
+ ### after change of user, and change of group has occured. It allows
+@@ -7953,4 +7983,15 @@
}
Amavis::SpamControl::init_pre_fork() if $extra_code_antispam;
+ if ($courierfilter_shutdown) {
@@ -63,9 +70,9 @@
+ POSIX::close(3);
+ }
+ }
- };
- if ($@ ne '') {
-@@ -7571,5 +7612,7 @@
+ 1;
+ } or do {
+@@ -8253,5 +8294,7 @@
if ($sock->NS_proto eq 'UNIX') { # traditional amavis helper program
if ($suggested_protocol eq 'COURIER') {
- die "unavailable support for protocol: $suggested_protocol";
@@ -74,7 +81,7 @@
+ $courier_in_obj->process_courier_request($sock, $conn, \&check_mail);
} elsif ($suggested_protocol eq 'AM.PDP') {
$amcl_in_obj = Amavis::In::AMCL->new if !$amcl_in_obj;
-@@ -7665,4 +7708,24 @@
+@@ -8343,4 +8386,24 @@
}
+### Net::Server hook
@@ -99,8 +106,8 @@
+
### Child is about to be terminated
### user customizable Net::Server hook
-@@ -9813,4 +9876,9 @@
- Amavis::Conf::read_config(@config_files);
+@@ -11056,4 +11119,9 @@
+ Amavis::Conf::supply_after_defaults();
+# courierfilter shutdown needs can_read_hook, added in Net::Server 0.90
+if ($courierfilter_shutdown && Net::Server->VERSION < 0.90) {
@@ -109,14 +116,14 @@
+
if (defined $desired_user && $daemon_user ne '') {
local($1);
-@@ -10124,4 +10192,6 @@
- log_level => $DEBUG ? 4 : 2,
+@@ -11467,4 +11535,6 @@
+ log_level => ($DEBUG || c('log_level') >= 5) ? 4 : 2,
log_file => undef, # will be overridden to call do_log()
+ # need to set multi_port for can_read_hook
+ multi_port => $courierfilter_shutdown ? 1 : undef,
- },
- }, 'Amavis';
-@@ -12686,5 +12756,413 @@
+ });
+
+@@ -14184,5 +14254,429 @@
use warnings FATAL => 'utf8';
-BEGIN { die "Code not available for module Amavis::In::Courier" }
@@ -133,10 +140,11 @@
+ import Amavis::Conf qw(:platform :confvars ca c);
+ import Amavis::Util qw(do_log am_id untaint debug_oneshot snmp_counters_init
+ switch_to_my_time switch_to_client_time
-+ read_text xtext_encode xtext_decode);
++ read_text orcpt_encode xtext_decode);
+ import Amavis::Lookup qw(lookup);
+ import Amavis::Lookup::IP qw(lookup_ip_acl);
-+ import Amavis::rfc2821_2822_Tools qw(quote_rfc2821_local qquote_rfc2821_local);
++ import Amavis::rfc2821_2822_Tools qw(quote_rfc2821_local qquote_rfc2821_local
++ unquote_rfc2821_local);
+ import Amavis::Timing qw(section_time);
+ import Amavis::TempDir;
+ import Amavis::In::Message;
@@ -170,18 +178,17 @@
+ $policy_bank_changed = $self->change_policy_bank();
+ $self->call_check_mail($conn, $check_mail);
+ $self->process_result();
-+
-+ }; if ($@) {
-+ # An exception occurred
-+ chomp($@);
-+ my $msg = "Error in processing: $@";
++ 1;
++ } or do { # An exception occurred
++ my($eval_stat) = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
++ my $msg = "Error in processing: $eval_stat";
+ do_log(-2, "TROUBLE in process_courier_request: 451 4.5.0 %s", $msg);
+ # Close the mail text file
+ $self->{msginfo}->mail_text->close() if ($self->{msginfo}->mail_text);
+ $self->{msginfo}->mail_text(undef);
+ # Send a temporary failure to Courier
+ $self->{smtp_resp} = "451 4.5.0 $msg";
-+ }
++ };
+
+ # Send the SMTP reponse back to Courier (done outside the eval to ensure that
+ # it always happens exactly once, whether or not there is an exception)
@@ -284,7 +291,10 @@
+ if (/^s ( .*? (?: \[ (?: \\. | [^\]\\] )* \]
+ | [^@"<>\[\]\\\s] )*
+ ) \z/xs) {
-+ $self->{msginfo}->sender($1);
++ my $sender_quoted = $1;
++ my $sender_unquoted = unquote_rfc2821_local($sender_quoted);
++ $self->{msginfo}->sender_smtp('<'.$sender_quoted.'>');
++ $self->{msginfo}->sender($sender_unquoted);
+ }
+
+ # Recipient
@@ -292,7 +302,10 @@
+ | [^@"<>\[\]\\\s] )*
+ ) \z/xs) {
+ $recip = Amavis::In::Message::PerRecip->new;
-+ $recip->recip_addr($1);
++ my $addr_quoted = $1;
++ my $addr_unquoted = unquote_rfc2821_local($addr_quoted);
++ $recip->recip_addr_smtp('<'.$addr_quoted.'>');
++ $recip->recip_addr($addr_unquoted);
+ $recip->courier_control_file($path);
+ $recip->courier_recip_index($rcpt_idx);
+ $recip->recip_destiny(D_PASS); # Default destiny
@@ -373,6 +386,10 @@
+ my $cl_ip_mynets = ($cl_ip eq '' ? undef
+ : lookup_ip_acl($cl_ip, @{ ca('mynetworks_maps') }));
+ $self->{msginfo}->client_addr_mynets($cl_ip_mynets);
++ if (($cl_ip_mynets?1:0) > (c('originating')?1:0)) {
++ $current_policy_bank{'originating'} = $cl_ip_mynets;
++ $policy_changed = 1;
++ }
+ if ($cl_ip_mynets && defined($policy_bank{'MYNETS'})) {
+ Amavis::load_policy_bank('MYNETS');
+ $policy_changed = 1;
@@ -410,10 +427,11 @@
+ $self->{msginfo}->dsn_passed_on(c('forward_method') eq '' ? 1 : 0);
+
+ # Log the message
-+ do_log(1, 'Courier %s %s: <%s> -> %s%s',
++ do_log(1, 'Courier %s %s: %s -> %s%s',
+ $self->{msginfo}->queue_id, $self->{tempdir}->path,
-+ $self->{msginfo}->sender,
-+ join(',', qquote_rfc2821_local(@{ $self->{msginfo}->recips })),
++ $self->{msginfo}->sender_smtp,
++ join(',', map { $_->recip_addr_smtp }
++ @{ $self->{msginfo}->per_recip_data }),
+ join('',
+ !$self->{msginfo}->auth_submitter ||
+ $self->{msginfo}->auth_submitter eq '<>' ? ():
@@ -444,8 +462,7 @@
+}
+
+# courier_in_obj->process_result( )
-+# Processes the result of mail scanning - recipient addition/deletion (for the
-+# time being we do not support this and only put a warning in the log)
++# Processes the result of mail scanning - recipient addition/deletion
+# Before calling this, the SMTP response must be stored in $self->{smtp_resp}
+# and may be altered
+# This does not send the SMTP response back to Courier
@@ -455,15 +472,21 @@
+ if ($self->{smtp_resp} =~ /^25/) {
+ foreach my $r (@{ $self->{msginfo}->per_recip_data }) {
+ my ($addr, $newaddr) = ($r->recip_addr, $r->recip_final_addr);
-+
++
+ if ($r->recip_done) {
-+ $self->delete_recipient($r);
-+
++ # Deleted recipient
++ $self->delete_recipient($r) if defined $addr;
++
++ } elsif (!defined($r->courier_control_file)) {
++ # Newly added recipient
++ $self->add_recipient($newaddr, '', $r->dsn_notify);
++
+ } elsif ($newaddr ne $addr) {
++ # Recipient with address changed
+ $r->recip_smtp_response("251 2.1.5 Amavisd replaced recip with <$newaddr>");
-+ $self->delete_recipient($r);
-+
-+ my $orcpt = $r->dsn_orcpt || 'rfc822;'.xtext_encode(quote_rfc2821_local($_));
++ $self->delete_recipient($r) if defined $addr;
++
++ my $orcpt = $r->dsn_orcpt || orcpt_encode($r->recip_addr_smtp);
+ $self->add_recipient($newaddr, $orcpt, $r->dsn_notify);
+ }
+ }
@@ -531,9 +554,9 @@
+}
1;
---- amavisd.conf-sample.ori Sat Sep 30 11:25:39 2006
-+++ amavisd.conf-sample Sat Sep 30 11:25:55 2006
-@@ -146,4 +146,11 @@
+--- amavisd.conf-sample.ori Wed Jun 27 12:39:59 2007
++++ amavisd.conf-sample Wed Jun 27 12:40:18 2007
+@@ -145,4 +145,11 @@
#$notify_method = $forward_method;
+# COURIER using courierfilter
@@ -558,7 +581,7 @@
+#$unix_socketname = "/var/lib/courier/allfilters/amavisd"; # Courier socket
# (usual setting is $MYHOME/amavisd.sock)
-@@ -2414,4 +2422,7 @@
+@@ -2462,4 +2470,7 @@
#$interface_policy{'SOCK'} = 'AM.PDP-SOCK';
+# Needed for Courier: speak courier protocol on the socket
diff --git a/amavisd-new-qmqpqq.patch b/amavisd-new-qmqpqq.patch
--- a/amavisd-new-qmqpqq.patch
+++ b/amavisd-new-qmqpqq.patch
@@ -1,36 +1,36 @@
---- amavisd.ori Sat Sep 30 11:25:44 2006
-+++ amavisd Sat Sep 30 11:27:48 2006
-@@ -96,4 +96,5 @@
+--- amavisd.ori Wed Jun 27 12:40:07 2007
++++ amavisd Wed Jun 27 12:41:41 2007
+@@ -98,4 +98,5 @@
# Amavis::In::SMTP
#( Amavis::In::Courier )
+# Amavis::In::QMQPqq
+ # Amavis::Out::SMTP::Protocol
# Amavis::Out::SMTP
- # Amavis::Out::Pipe
-@@ -2601,4 +2602,5 @@
+@@ -3006,4 +3007,5 @@
: sprintf(" (%s [%s])", c('myhostname'), $conn->socket_ip) ),
($conn->socket_port eq '' ? 'unix socket' : "port ".$conn->socket_port) );
+ # must not use proto name QMQPqq in 'with'
$s .= "\n with $smtp_proto" if $smtp_proto=~/^(ES|S|L)MTPS?A?\z/i; # rfc3848
$s .= "\n id $id" if $id ne '';
-@@ -6697,4 +6699,5 @@
+@@ -7330,4 +7332,5 @@
$extra_code_sql_lookup $extra_code_ldap
$extra_code_in_amcl $extra_code_in_smtp $extra_code_in_courier
+ $extra_code_in_qmqpqq
$extra_code_out_smtp $extra_code_out_pipe
$extra_code_out_bsmtp $extra_code_out_local $extra_code_p0f
-@@ -6720,4 +6723,5 @@
+@@ -7353,4 +7356,5 @@
# Amavis::In::AMCL, Amavis::In::SMTP and In::Courier objects
use vars qw($amcl_in_obj $smtp_in_obj $courier_in_obj);
+use vars qw($qmqpqq_in_obj); # Amavis::In::QMQPqq object
use vars qw($sql_dataset_conn_lookups); # Amavis::Out::SQL::Connection object
use vars qw($sql_dataset_conn_storage); # Amavis::Out::SQL::Connection object
-@@ -7157,4 +7161,5 @@
+@@ -7828,4 +7832,5 @@
do_log(0,"SMTP-in proto code %s loaded", $extra_code_in_smtp ?'':" NOT");
do_log(0,"Courier proto code %s loaded", $extra_code_in_courier ?'':" NOT");
+ do_log(0,"QMQPqq-in proto code %s loaded", $extra_code_in_qmqpqq ?'':" NOT");
do_log(0,"SMTP-out proto code %s loaded", $extra_code_out_smtp ?'':" NOT");
do_log(0,"Pipe-out proto code %s loaded", $extra_code_out_pipe ?'':" NOT");
-@@ -7590,4 +7595,10 @@
+@@ -8272,4 +8277,10 @@
$amcl_in_obj = Amavis::In::AMCL->new if !$amcl_in_obj;
$amcl_in_obj->process_policy_request($sock, $conn, \&check_mail, 0);
+ } elsif ($suggested_protocol eq 'QMQPqq') {
@@ -41,25 +41,25 @@
+ $qmqpqq_in_obj->process_qmqpqq_request($sock,$conn,\&check_mail);
} else { # defaults to SMTP or LMTP
if (!$extra_code_in_smtp) {
-@@ -7677,4 +7688,5 @@
+@@ -8355,4 +8366,5 @@
do_log(5,"child_finish_hook: invoking DESTROY methods");
undef $smtp_in_obj; undef $amcl_in_obj; undef $courier_in_obj;
+ undef $qmqpqq_in_obj;
undef $sql_storage; undef $sql_wblist; undef $sql_policy; undef $ldap_policy;
undef $sql_dataset_conn_lookups; undef $sql_dataset_conn_storage;
-@@ -7687,4 +7699,5 @@
+@@ -8365,4 +8377,5 @@
# do_log(5,"at the END handler: invoking DESTROY methods");
undef $smtp_in_obj; undef $amcl_in_obj; undef $courier_in_obj;
+ undef $qmqpqq_in_obj;
undef $sql_storage; undef $sql_wblist; undef $sql_policy; undef $ldap_policy;
undef $sql_dataset_conn_lookups; undef $sql_dataset_conn_storage;
-@@ -9666,4 +9679,5 @@
+@@ -10902,4 +10915,5 @@
$extra_code_sql_lookup, $extra_code_ldap,
$extra_code_in_amcl, $extra_code_in_smtp, $extra_code_in_courier,
+ $extra_code_in_qmqpqq,
$extra_code_out_smtp, $extra_code_out_pipe,
$extra_code_out_bsmtp, $extra_code_out_local, $extra_code_p0f,
-@@ -9902,5 +9916,10 @@
+@@ -11145,5 +11159,10 @@
undef $extra_code_in_courier;
}
- if ($needed_protocols_in{'QMQPqq'}) { die "In::QMQPqq code not available" }
@@ -71,7 +71,7 @@
+ }
}
-@@ -12687,4 +12706,270 @@
+@@ -14185,4 +14204,276 @@
BEGIN { die "Code not available for module Amavis::In::Courier" }
+
@@ -197,6 +197,7 @@
+Amavis::Timing::init();
+
+$conn->smtp_proto("QMQPqq"); # the name of the method is too specific
++my($eval_stat);
+eval {
+ # get length of whole package
+ $self->{bytesleft} = $self->getlen;
@@ -232,11 +233,11 @@
+ # comma has to follow
+ $self->getcomma;
+
-+ # get sender
++ # get sender (presumably in unquoted form, really???)
+ $self->getnetstring($sender);
+ section_time('sender receiving');
+
-+ # get recips
++ # get recips (presumably in unquoted form, really???)
+ my $i = 0;
+ while($self->{bytesleft}) {
+ $self->getnetstring($recips[$i++]);
@@ -248,13 +249,16 @@
+ $self->getcomma;
+
+ $msginfo->sender($sender);
++ $msginfo->sender_smtp(qquote_rfc2821_local($sender));
+ $msginfo->recips(\@recips);
+
-+ do_log(1, sprintf("%s:%s:%s %s: <%s> -> %s Received: %s",
++ do_log(1, sprintf("%s:%s:%s %s: %s -> %s Received: %s",
+ $self->{proto},$conn->socket_ip eq $inet_socket_bind ?
+ '' : '['.$conn->socket_ip.']',
+ $conn->socket_port, $self->{tempdir_pers},
-+ $sender, join(',', map{"<$_>"}@recips),
++ $msginfo->sender_smtp,
++ join(',', map { $_->recip_addr_smtp }
++ @{$msginfo->per_recip_data}),
+ join(' ',
+ ($msginfo->msg_size eq '' ? ()
+ : 'SIZE='.$msginfo->msg_size),
@@ -301,16 +305,18 @@
+ else {
+ $self->qmqpqq_resp("Z",$smtp_resp);
+ }
-+ };
++ 1;
++} or do {
++ $eval_stat = $@ ne '' ? $@ : "errno=$!";
++};
+
+$self->{tempdir}->clean;
+alarm(0); do_log(4,"timer stopped after QMQPqq eval");
+
-+if($@ ne '') {
-+ chomp($@);
-+
-+ do_log(0,"QMQPqq: NOTICE: $@");
-+ $self->qmqpqq_resp("Z","Service shutting down, $@");
++if($eval_stat ne '') {
++ chomp $eval_stat;
++ do_log(0,"QMQPqq: NOTICE: $eval_stat");
++ $self->qmqpqq_resp("Z","Service shutting down, $eval_stat");
+ }
+# report elapsed times by section for each transaction
+do_log(2, "%s", Amavis::Timing::report());
@@ -342,18 +348,19 @@
+}
1;
---- amavisd.conf.ori Sat Sep 30 11:25:26 2006
-+++ amavisd.conf Sat Sep 30 11:27:48 2006
-@@ -49,5 +49,6 @@
- $enable_global_cache = 1; # enable use of libdb-based cache if $enable_db=1
-
--$inet_socket_port = 10024; # listen on this local TCP port(s) (see $protocol)
+--- amavisd.conf.ori Wed Jun 27 12:39:44 2007
++++ amavisd.conf Wed Jun 27 12:41:41 2007
+@@ -55,6 +55,6 @@
+ # option(s) -p overrides $inet_socket_port and $unix_socketname
+
+-$inet_socket_port = 10024; # listen on this local TCP port(s)
+-# $inet_socket_port = [10024,10026]; # listen on multiple TCP ports
+$protocol = 'QMQPqq'; # suggested protocol to use on all input sockets
+$inet_socket_port = 10628; # accept connections on this local TCP port(s)
- $unix_socketname = "$MYHOME/amavisd.sock"; # amavisd-release or amavis-milter
- # option(s) -p overrides $inet_socket_port and $unix_socketname
---- amavisd.conf-sample.ori Sat Sep 30 11:25:39 2006
-+++ amavisd.conf-sample Sat Sep 30 11:27:48 2006
+
+ $policy_bank{'MYNETS'} = { # mail originating from @mynetworks
+--- amavisd.conf-sample.ori Wed Jun 27 12:39:59 2007
++++ amavisd.conf-sample Wed Jun 27 12:41:41 2007
@@ -225,8 +225,11 @@
# SMTP SERVER (INPUT) PROTOCOL SETTINGS (e.g. with Postfix, Exim v4, ...)
# (used when MTA is configured to pass mail to amavisd via SMTP or LMTP)
@@ -367,7 +374,7 @@
+
# SMTP SERVER (INPUT) access control
# - do not allow free access to the amavisd SMTP port !!!
-@@ -2353,8 +2356,14 @@
+@@ -2400,8 +2403,14 @@
# ],
# };
+#
diff --git a/amavisd-release b/amavisd-release
--- a/amavisd-release
+++ b/amavisd-release
@@ -32,7 +32,7 @@
# using socket protection (unix socket) or @inet_acl (for inet socket).
#
# Author: Mark Martinec <mark.martinec at ijs.si>
-# Copyright (C) 2005 Mark Martinec, All Rights Reserved.
+# Copyright (C) 2005,2007 Mark Martinec, All Rights Reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
@@ -54,7 +54,7 @@
# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
-# OR BUSINESS INTERRUPTION) HOWEVERREADME.protocol CAUSED AND ON ANY THEORY OF LIABILITY,
+# OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
# OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
@@ -66,12 +66,15 @@
# http://www.ijs.si/software/amavisd/
#------------------------------------------------------------------------------
+use warnings;
+use warnings FATAL => 'utf8';
+no warnings 'uninitialized';
use strict;
use re 'taint';
use IO::Socket;
use Time::HiRes ();
-use vars qw($VERSION); $VERSION = 1.200;
+use vars qw($VERSION); $VERSION = 1.500;
use vars qw($log_level $socketname);
$log_level = 1;
@@ -142,14 +145,21 @@ sub ask_amavisd($$) {
sub release_file($$$@) {
my($sock,$mail_file,$secret_id, at alt_recips) = @_;
- $mail_file =~ m{^ ([^/].*/)? ([A-Z0-9_-]*)
- ([A-Z0-9][A-Z0-9+-]{10}[A-Z0-9]) (\.gz)? \z}xsi
- or die "Invalid quarantine file name: $mail_file";
- my($fn_path,$fn_prefix,$mail_id,$fn_suffix) = ($1,$2,$3,$4);
+ my($fn_path,$fn_prefix,$mail_id,$fn_suffix); local($1,$2,$3,$4);
+ if ($mail_file =~ m{^ ([^/].*/)? ([A-Z0-9][A-Z0-9._-]*[_-])?
+ ([A-Z0-9][A-Z0-9_+-]{10}[A-Z0-9]) (\.gz)? \z}xsi) {
+ ($fn_path,$fn_prefix,$mail_id,$fn_suffix) = ($1,$2,$3,$4);
+ } elsif ($mail_file =~ m{^ ([^/].*/)? () ([A-Za-z0-9$._=+-]+?) (\.gz)?\z}xs){
+ ($fn_path,$fn_prefix,$mail_id,$fn_suffix) = ($1,$2,$3,$4); # old style
+ } else {
+ usage("Invalid quarantine ID: $mail_file");
+ }
my($quar_type) =
- $fn_suffix eq '.gz' ? 'Z' : $mail_id eq $mail_file ? 'Q' : 'F';
+ $fn_suffix eq '.gz' ? 'Z'
+ : $fn_path eq '' && $mail_id eq $mail_file ? 'Q' : 'F';
+ my($request_type) = $0 =~ /\brequeue\z/i ? 'requeue' : 'release';
my(@query) = (
- 'request=release',
+ "request=$request_type",
"quar_type=$quar_type",
"mail_id=$mail_id",
);
@@ -159,6 +169,7 @@ sub release_file($$$@) {
push(@query, map {"recipient=$_"} @alt_recips);
}
my($attr_ref) = ask_amavisd($sock,\@query);
+ $attr_ref && %$attr_ref or die "Invalid response received";
for my $attr_name (keys %$attr_ref) {
for my $attr_val (@{$attr_ref->{$attr_name}})
{ do_log(2,"< $attr_name=$attr_val") }
@@ -170,9 +181,10 @@ sub usage(;$) {
sub usage(;$) {
my($msg) = @_;
print STDERR $msg,"\n\n" if $msg ne '';
- print STDERR "amavisd-release version $VERSION\n";
- die "Usage: \$ amavisd-release mail_file [secret_id [alt_recip1 alt_recip2 ...]]\n".
- " or to read request lines from stdin: \$ amavisd-release -\n";
+ my($prog) = $0; $prog =~ s{^.*/(?=[^/]+\z)}{};
+ print STDERR "$prog version $VERSION\n";
+ die "Usage: \$ $prog mail_file [secret_id [alt_recip1 alt_recip2 ...]]\n".
+ " or to read request lines from stdin: \$ $prog -\n";
}
# Main program starts here
diff --git a/amavisd.conf b/amavisd.conf
--- a/amavisd.conf
+++ b/amavisd.conf
@@ -10,8 +10,9 @@ use strict;
# COMMONLY ADJUSTED SETTINGS:
-# @bypass_virus_checks_maps = (1); # uncomment to DISABLE anti-virus code
-# @bypass_spam_checks_maps = (1); # uncomment to DISABLE anti-spam code
+# @bypass_virus_checks_maps = (1); # controls running of anti-virus code
+# @bypass_spam_checks_maps = (1); # controls running of anti-spam code
+# $bypass_decode_parts = 1; # controls running of decoders&dearchivers
$max_servers = 2; # num of pre-forked children (2..15 is common), -m
$daemon_user = 'vscan'; # (no default; customary: vscan or amavis), -u
@@ -21,7 +22,7 @@ use strict;
# $MYHOME = '/var/amavis'; # a convenient default for other settings, -H
$TEMPBASE = "$MYHOME/tmp"; # working directory, needs to exist, -T
-$ENV{TMPDIR} = $TEMPBASE; # environment variable TMPDIR
+$ENV{TMPDIR} = $TEMPBASE; # environment variable TMPDIR, used by SA, etc.
$QUARANTINEDIR = '/var/virusmails'; # -Q
# $quarantine_subdir_levels = 1; # add level of subdirs to disperse quarantine
@@ -32,10 +33,6 @@ use strict;
# $lock_file = "$MYHOME/var/amavisd.lock"; # -L
# $pid_file = "$MYHOME/var/amavisd.pid"; # -P
#NOTE: create directories $MYHOME/tmp, $MYHOME/var, $MYHOME/db manually
-
- at local_domains_maps = ( [".$mydomain"] );
-# @mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
-# 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 );
$log_level = 0; # verbosity 0..5, -d
$log_recip_templ = undef; # disable by-recipient level-0 log entries
@@ -47,23 +44,59 @@ use strict;
$enable_db = 1; # enable use of BerkeleyDB/libdb (SNMP and nanny)
$enable_global_cache = 1; # enable use of libdb-based cache if $enable_db=1
-
-$inet_socket_port = 10024; # listen on this local TCP port(s) (see $protocol)
+$nanny_details_level = 2; # nanny verbosity: 1: traditional, 2: detailed
+
+ at local_domains_maps = ( [".$mydomain"] ); # list of all local domains
+
+ at mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
+ 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 );
+
$unix_socketname = "$MYHOME/amavisd.sock"; # amavisd-release or amavis-milter
- # option(s) -p overrides $inet_socket_port and $unix_socketname
-
-$interface_policy{'SOCK'}='AM.PDP-SOCK'; # only relevant with $unix_socketname
+ # option(s) -p overrides $inet_socket_port and $unix_socketname
+
+$inet_socket_port = 10024; # listen on this local TCP port(s)
+# $inet_socket_port = [10024,10026]; # listen on multiple TCP ports
+
+$policy_bank{'MYNETS'} = { # mail originating from @mynetworks
+ originating => 1, # is true in MYNETS by default, but let's make it explicit
+ os_fingerprint_method => undef, # don't query p0f for internal clients
+};
+
+# it is up to MTA to re-route mail from authenticated roaming users or
+# from internal hosts to a dedicated TCP port (such as 10026) for filtering
+$interface_policy{'10026'} = 'ORIGINATING';
+
+$policy_bank{'ORIGINATING'} = { # mail supposedly originating from our users
+ originating => 1, # declare that mail was submitted by our smtp client
+ allow_disclaimers => 1, # enables disclaimer insertion if available
+ # notify administrator of locally originating malware
+ virus_admin_maps => ["virusalert\@$mydomain"],
+ spam_admin_maps => ["virusalert\@$mydomain"],
+ warnbadhsender => 1,
+ # forward to a smtpd service providing DKIM signing service
+ forward_method => 'smtp:[127.0.0.1]:10027',
+ # force MTA conversion to 7-bit (e.g. before DKIM signing)
+ smtpd_discard_ehlo_keywords => ['8BITMIME'],
+ bypass_banned_checks_maps => [1], # allow sending any file names and types
+ terminate_dsn_on_notify_success => 0, # don't remove NOTIFY=SUCCESS option
+};
+
+$interface_policy{'SOCK'} = 'AM.PDP-SOCK'; # only applies with $unix_socketname
+
# Use with amavis-release over a socket or with Petr Rehor's amavis-milter.c
# (with amavis-milter.c from this package or old amavis.c client use 'AM.CL'):
-$policy_bank{'AM.PDP-SOCK'} = { protocol=>'AM.PDP' };
+$policy_bank{'AM.PDP-SOCK'} = {
+ protocol => 'AM.PDP',
+ auth_required_release => 0, # do not require secret_id for amavisd-release
+};
$sa_tag_level_deflt = 2.0; # add spam info headers if at, or above that level
-$sa_tag2_level_deflt = 6.31; # add 'spam detected' headers at that level
-$sa_kill_level_deflt = 6.31; # triggers spam evasive actions
+$sa_tag2_level_deflt = 6.2; # add 'spam detected' headers at that level
+$sa_kill_level_deflt = 6.9; # triggers spam evasive actions (e.g. blocks mail)
$sa_dsn_cutoff_level = 10; # spam level beyond which a DSN is not sent
-# $sa_quarantine_cutoff_level = 20; # spam level beyond which quarantine is off
-# $penpals_bonus_score = 4; # (no effect without a @storage_sql_dsn database)
-# $penpals_threshold_high = $sa_kill_level_deflt; # don't waste time on hi spam
+# $sa_quarantine_cutoff_level = 25; # spam level beyond which quarantine is off
+$penpals_bonus_score = 8; # (no effect without a @storage_sql_dsn database)
+$penpals_threshold_high = $sa_kill_level_deflt; # don't waste time on hi spam
$sa_mail_body_size_limit = 400*1024; # don't waste time on SA if mail is larger
$sa_local_tests_only = 0; # only tests which do not require internet access?
@@ -166,14 +199,14 @@ use strict;
qr'^\.(exe-ms|dll)$', # banned file(1) types, rudimentary
# qr'^\.(exe|lha|tnef|cab|dll)$', # banned file(1) types
-### BLOCK THE FOLLOWING, EXCEPT WITHIN UNIX ARHIVES:
+### BLOCK THE FOLLOWING, EXCEPT WITHIN UNIX ARCHIVES:
# [ qr'^\.(gz|bz2)$' => 0 ], # allow any in gzip or bzip2
[ qr'^\.(rpm|cpio|tar)$' => 0 ], # allow any in Unix-type archives
qr'.\.(pif|scr)$'i, # banned extensions - rudimentary
# qr'^\.zip$', # block zip type
-### BLOCK THE FOLLOWING, EXCEPT WITHIN ARHIVES:
+### BLOCK THE FOLLOWING, EXCEPT WITHIN ARCHIVES:
# [ qr'^\.(zip|rar|arc|arj|zoo)$'=> 0 ], # allow any within these archives
qr'^application/x-msdownload$'i, # block these MIME types
@@ -187,7 +220,7 @@ use strict;
# qr'^\.wmf$', # Windows Metafile file(1) type
# block certain double extensions in filenames
- qr'\.[^./]*[A-Za-z][^./]*\.(exe|vbs|pif|scr|bat|cmd|com|cpl|dll)\.?$'i,
+ qr'\.[^./]*[A-Za-z][^./]*\.\s*(exe|vbs|pif|scr|bat|cmd|com|cpl|dll)[.\s]*$'i,
# qr'\{[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}\}?'i, # Class ID CLSID, strict
# qr'\{[0-9a-z]{4,}(-[0-9a-z]{4,}){0,7}\}?'i, # Class ID extension CLSID, loose
@@ -198,6 +231,8 @@ use strict;
# inf|ins|isp|js|jse|lnk|mda|mdb|mde|mdw|mdt|mdz|msc|msi|msp|mst|
# ops|pcd|pif|prg|reg|scr|sct|shb|shs|vb|vbe|vbs|
# wmf|wsc|wsf|wsh)$'ix, # banned ext - long
+# qr'.\.(ani|cur|ico)$'i, # banned cursors and icons filename
+# qr'^\.ani$', # banned animated cursor file(1) type
# qr'.\.(mim|b64|bhx|hqx|xxe|uu|uue)$'i, # banned extension - WinZip vulnerab.
);
@@ -288,10 +323,10 @@ use strict;
['rpm', \&do_uncompress, ['rpm2cpio.pl','rpm2cpio'] ],
['cpio', \&do_pax_cpio, ['pax','gcpio','cpio'] ],
['tar', \&do_pax_cpio, ['pax','gcpio','cpio'] ],
- ['tar', \&do_tar],
['deb', \&do_ar, 'ar'],
# ['a', \&do_ar, 'ar'], # unpacking .a seems an overkill
['zip', \&do_unzip],
+ ['7z', \&do_7zip, ['7zr','7za','7z'] ],
['rar', \&do_unrar, ['rar','unrar'] ],
['arj', \&do_unarj, ['arj','unarj'] ],
['arc', \&do_arc, ['nomarch','arc'] ],
@@ -308,7 +343,7 @@ use strict;
@av_scanners = (
-# ### http://www.vanja.com/tools/sophie/
+# ### http://www.clanfield.info/sophie/ (http://www.vanja.com/tools/sophie/)
# ['Sophie',
# \&ask_daemon, ["{}/\n", '/var/run/sophie'],
# qr/(?x)^ 0+ ( : | [\000\r\n]* $)/, qr/(?x)^ 1 ( : | [\000\r\n]* $)/,
@@ -329,6 +364,7 @@ use strict;
# # this entry; when running chrooted one may prefer socket "$MYHOME/clamd".
# ### http://www.clamav.net/ and CPAN (memory-hungry! clamd is preferred)
+# # note that Mail::ClamAV requires perl to be build with threading!
# ['Mail::ClamAV', \&ask_clamav, "*", [0], [1], qr/^INFECTED: (.+)/],
# ### http://www.openantivirus.org/
@@ -421,7 +457,7 @@ use strict;
['CentralCommand Vexira (new) vascan',
['vascan','/usr/lib/Vexira/vascan'],
"-a s --timeout=60 --temp=$TEMPBASE -y $QUARANTINEDIR ".
- "--vdb=/usr/lib/Vexira/vexira8.vdb --log=/var/log/vascan.log {}",
+ "--log=/var/log/vascan.log {}",
[0,3], [1,2,5],
qr/(?x)^\s* (?:virus|iworm|macro|mutant|sequence|trojan)\ found:\ ( [^\]\s']+ )\ \.\.\.\ / ],
# Adjust the path of the binary and the virus database as needed.
@@ -458,12 +494,21 @@ use strict;
qr/^(?:Info|Virus Name):\s+(.+)/ ],
# NOTE: check options and patterns to see which entry better applies
- ### http://www.f-secure.com/products/anti-virus/ version 4.65
+# ### http://www.f-secure.com/products/anti-virus/ version 4.65
+# ['F-Secure Antivirus for Linux servers',
+# ['/opt/f-secure/fsav/bin/fsav', 'fsav'],
+# '--delete=no --disinf=no --rename=no --archive=yes --auto=yes '.
+# '--dumb=yes --list=no --mime=yes {}', [0], [3,6,8],
+# qr/(?:infection|Infected|Suspected): (.+)/ ],
+
+ ### http://www.f-secure.com/products/anti-virus/ version 5.52
['F-Secure Antivirus for Linux servers',
['/opt/f-secure/fsav/bin/fsav', 'fsav'],
- '--delete=no --disinf=no --rename=no --archive=yes --auto=yes '.
- '--dumb=yes --list=no --mime=yes {}', [0], [3,6,8],
- qr/(?:infection|Infected|Suspected): (.+)/ ],
+ '--virus-action1=report --archive=yes --auto=yes '.
+ '--dumb=yes --list=no --mime=yes {}', [0], [3,4,6,8],
+ qr/(?:infection|Infected|Suspected|Riskware): (.+)/ ],
+ # NOTE: internal archive handling may be switched off by '--archive=no'
+ # to prevent fsav from exiting with status 9 on broken archives
# ### http://www.avast.com/
# ['avast! Antivirus daemon',
@@ -498,13 +543,18 @@ use strict;
'-s -q {}', [0], [1..7],
qr/^... (\S+)/ ],
- ### http://www.nod32.com/, version v2.52 and above
- ['ESET NOD32 for Linux Mail servers',
- ['/opt/eset/nod32/bin/nod32cli', 'nod32cli'],
- '--subdir --files -z --sfx --rtp --adware --unsafe --pattern --heur '.
- '-w -a --action-on-infected=accept --action-on-uncleanable=accept '.
- '--action-on-notscanned=accept {}',
- [0,3], [1,2], qr/virus="([^"]+)"/ ],
+# ### http://www.nod32.com/, version v2.52 and above
+# ['ESET NOD32 for Linux Mail servers',
+# ['/opt/eset/nod32/bin/nod32cli', 'nod32cli'],
+# '--subdir --files -z --sfx --rtp --adware --unsafe --pattern --heur '.
+# '-w -a --action-on-infected=accept --action-on-uncleanable=accept '.
+# '--action-on-notscanned=accept {}',
+# [0,3], [1,2], qr/virus="([^"]+)"/ ],
+
+ ### http://www.eset.com/, version v2.7
+ ['ESET NOD32 Linux Mail Server - command line interface',
+ ['/usr/bin/nod32cli', '/opt/eset/nod32/bin/nod32cli', 'nod32cli'],
+ '--subdir {}', [0,3], [1,2], qr/virus="([^"]+)"/ ],
## http://www.nod32.com/, NOD32LFS version 2.5 and above
['ESET NOD32 for Linux File servers',
@@ -605,6 +655,11 @@ use strict;
# consider also: --all --nowarn --alev=15 --flev=15. The --all argument may
# not apply to your version of bdc, check documentation and see 'bdc --help'
+ ### ArcaVir for Linux and Unix http://www.arcabit.pl/
+ ['ArcaVir for Linux', ['arcacmd','arcacmd.static'],
+ '-v 1 -summary 0 -s {}', [0], [1,2],
+ qr/(?:VIR|WIR):[ \t]*(.+)/ ],
+
# ['File::Scan', sub {Amavis::AV::ask_av(sub{
# use File::Scan; my($fn)=@_;
# my($f)=File::Scan->new(max_txt_size=>0, max_bin_size=>0);
@@ -632,23 +687,24 @@ use strict;
### http://www.f-prot.com/ - backs up F-Prot Daemon
['FRISK F-Prot Antivirus', ['f-prot','f-prot.sh'],
- '-dumb -archive -packed {}', [0,8], [3,6],
- qr/Infection: (.+)|\s+contains\s+(.+)$/ ],
+ '-dumb -archive -packed {}', [0,8], [3,6], # or: [0], [3,6,8],
+ qr/(?:Infection:|security risk named) (.+)|\s+contains\s+(.+)$/ ],
### http://www.trendmicro.com/ - backs up Trophie
['Trend Micro FileScanner', ['/etc/iscan/vscan','vscan'],
'-za -a {}', [0], qr/Found virus/, qr/Found virus (.+) in/ ],
### http://www.sald.com/, http://drweb.imshop.de/ - backs up DrWebD
- ['drweb - DrWeb Antivirus',
+ ['drweb - DrWeb Antivirus', # security LHA hole in Dr.Web 4.33 and earlier
['/usr/local/drweb/drweb', '/opt/drweb/drweb', 'drweb'],
'-path={} -al -go -ot -cn -upn -ok-',
[0,32], [1,9,33], qr' infected (?:with|by)(?: virus)? (.*)$'],
### http://www.kaspersky.com/
['Kaspersky Antivirus v5.5',
- ['/opt/kav/5.5/kav4unix/bin/kavscanner',
- '/opt/kav/5.5/kav4mailservers/bin/kavscanner','kavscanner'],
+ ['/opt/kaspersky/kav4fs/bin/kav4fs-kavscanner',
+ '/opt/kav/5.5/kav4unix/bin/kavscanner',
+ '/opt/kav/5.5/kav4mailservers/bin/kavscanner', 'kavscanner'],
'-i0 -xn -xp -mn -R -ePASBME {}/*', [0,10,15], [5,20,21,25],
qr/(?:INFECTED|WARNING|SUSPICION|SUSPICIOUS) (.*)/ ,
# sub {chdir('/opt/kav/bin') or die "Can't chdir to kav: $!"},
@@ -661,11 +717,12 @@ use strict;
#
# ### http://www.sophos.com/ - backs up Sophie or SAVI-Perl
# ['Sophos Anti Virus (sweep)', 'sweep',
-# '-nb -f -all -rec -ss -sc -archive -cab -tnef --no-reset-atime {}',
+# '-nb -f -all -rec -ss -sc -archive -cab -mime -oe -tnef '.
+# '--no-reset-atime {}',
# [0,2], qr/Virus .*? found/,
# qr/^>>> Virus(?: fragment)? '?(.*?)'? found/,
# ],
-# # other options to consider: -mime -oe -idedir=/usr/local/sav
+# # other options to consider: -idedir=/usr/local/sav
# always succeeds (uncomment to consider mail clean if all other scanners fail)
# ['always-clean', sub {0}],
diff --git a/amavisd.conf-default b/amavisd.conf-default
--- a/amavisd.conf-default
+++ b/amavisd.conf-default
@@ -29,12 +29,16 @@ use strict;
# $pid_file = "$MYHOME/amavisd.pid"; # after-default
# $lock_file = "$MYHOME/amavisd.lock"; # after-default
# $daemon_chroot_dir = undef;
-# $max_servers = 2; # number of pre-forked children
-# $max_requests = 10; # retire a child after that many accepts
+# $max_requests = 20; # retire a child after that many accepts
+# $max_servers = 2; # number of pre-forked children
+# $min_servers = undef; # see Net::Server::Prefork for semantics
+# $min_spare_servers = undef;
+# $max_spare_servers = undef;
# $child_timeout = 8*60;
# $localpart_is_case_sensitive = 0;
# $enable_db = 0;
# $enable_global_cache = 0;
+# $nanny_details_level = 1; # verbosity: 0, 1, 2
# @additional_perl_modules = ();
# @local_domains_maps=(\%local_domains,\@local_domains_acl,\$local_domains_re);
# @mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
@@ -86,7 +90,7 @@ use strict;
## see also $notify_method, $forward_method and $*_quarantine_method
# $localhost_name = 'localhost'; # my EHLO name, and inserted in Received
-# $local_client_bind_address = undef;
+# $local_client_bind_address = undef; # my source IP address as a SMTP client
# $auth_required_out = undef;
# $amavis_auth_user = undef; # for submitting notifications and quarantine
# $amavis_auth_pass = undef;
@@ -97,6 +101,7 @@ use strict;
# $forward_method = 'smtp:[127.0.0.1]:10025';
# $resend_method = undef; # falls back to $forward_method
+# $always_bcc = undef;
# $final_virus_destiny = D_DISCARD; # subj to @viruses_that_fake_sender_maps
# $final_banned_destiny = D_BOUNCE;
@@ -106,6 +111,7 @@ use strict;
## QUARANTINE
# $release_method = undef; # falls back to $notify_method
+# $requeue_method = 'smtp:[127.0.0.1]:25';
# $virus_quarantine_method = 'local:virus-%m';
# $spam_quarantine_method = 'local:spam-%m.gz';
@@ -205,6 +211,7 @@ use strict;
# $allow_fixing_improper_header = 1; # all-white folding lines and long lines
# $allow_fixing_improper_header_folding = 1;
# $append_header_fields_to_bottom = 0;
+# $prepend_header_fields_hdridx = 0;
# $X_HEADER_TAG = 'X-Virus-Scanned'; # after-default
# $X_HEADER_LINE = "$myproduct_name at $mydomain"; # after-default
@@ -215,6 +222,14 @@ use strict;
# $defang_bad_header = undef;
# $defang_undecipherable = undef;
# $defang_all = undef; # mostly for testing
+
+# $allow_disclaimers = undef;
+# $enable_anomy_sanitizer = 0;
+# @anomy_sanitizer_args = (); # a config file or list of var=value pairs
+# $altermime = 'altermime'; # a path to the program
+# @altermime_args_defang = qw(--verbose --removeall);
+# @altermime_args_disclaimer = qw(--disclaimer=/etc/altermime-disclaimer.txt);
+# @disclaimer_options_bysender_maps = ();
# $undecipherable_subject_tag = '***UNCHECKED*** ';
# $sa_spam_subject_tag = undef;
@@ -288,10 +303,10 @@ use strict;
# ['cpio', \&do_pax_cpio, \$cpio],
# ['tar', \&do_pax_cpio, \$pax],
# ['tar', \&do_pax_cpio, \$cpio],
-# ['tar', \&do_tar],
# ['deb', \&do_ar, \$ar],
### ['a', \&do_ar, \$ar], # unpacking .a seems an overkill
# ['zip', \&do_unzip],
+# ['7z', \&do_7zip, ['7zr','7za','7z'] ],
# ['rar', \&do_unrar, \$unrar],
# ['arj', \&do_unarj, \$unarj],
# ['arc', \&do_arc, \$arc],
@@ -314,6 +329,12 @@ use strict;
# $viruses_that_fake_sender_re = undef;
# @viruses_that_fake_sender_maps = (\$viruses_that_fake_sender_re, 1);
+# @virus_name_to_spam_score_maps =
+# (new_RE( [ qr'^(Email|HTML)\.(Phishing|Spam|Scam[a-z0-9]?)\.'i => 0.1 ],
+# [ qr'^(Email|Html)\.Malware\.Sanesecurity\.' => undef ],
+# [ qr'^(Email|Html)(\.[^., ]*)*\.Sanesecurity\.' => 0.1 ],
+# [ qr'^(MSRBL-Images/|MSRBL-SPAM\.)' => 0.1 ],
+# ));
# $banned_namepath_re = undef; # new-style
# $banned_filename_re = undef; # traditional
@@ -330,11 +351,15 @@ use strict;
# @banned_files_lovers_maps = (\%banned_files_lovers, \@banned_files_lovers_acl, \$banned_files_lovers_re);
# @bad_header_lovers_maps = (\%bad_header_lovers, \@bad_header_lovers_acl, \$bad_header_lovers_re);
+# $allowed_header_tests{$_} = 1 for qw(other mime 8bit control empty long
+# syntax missing multiple);
+
## ANTI-SPAM CONTROLS
# $sa_mail_body_size_limit = undef;
# $sa_local_tests_only = 0;
+# $sa_spawned = 0;
# $dspam = undef;
# $sa_timeout = 30;
@@ -358,10 +383,7 @@ use strict;
# @spam_dsn_cutoff_level_maps = (\$sa_dsn_cutoff_level);
# @spam_quarantine_cutoff_level_maps = (\$sa_quarantine_cutoff_level);
-# $penpals_bonus_score = undef;
-# $penpals_halflife = 7*24*60*60; # exponential decay time constant in seconds
-# $penpals_threshold_low = 1.0;
-# $penpals_threshold_high = undef;
+$penpals_bonus_score = 8; # (no effect without a @storage_sql_dsn database)
# @score_sender_maps = ();
@@ -391,8 +413,8 @@ use strict;
# $sql_select_white_black_list =
# 'SELECT wb'.
-# ' FROM wblist LEFT JOIN mailaddr ON wblist.sid=mailaddr.id'.
-# ' WHERE (wblist.rid=?) AND (mailaddr.email IN (%k))'.
+# ' FROM wblist JOIN mailaddr ON wblist.sid=mailaddr.id'.
+# ' WHERE wblist.rid=? AND mailaddr.email IN (%k)'.
# ' ORDER BY mailaddr.priority DESC';
# %sql_clause = (
@@ -463,17 +485,16 @@ use strict;
# CC_SPAM, sub { ca('spam_lovers_maps') },
# CC_BADH, sub { ca('bad_header_lovers_maps') },
# );
-# %defang_by_ccat = (
-# CC_VIRUS, sub { c('defang_virus') || c('defang_all') },
-# CC_BANNED, sub { c('defang_banned') || c('defang_all') },
-# CC_UNCHECKED, sub { c('defang_undecipherable') || c('defang_all') },
-# CC_SPAM, sub { c('defang_spam') || c('defang_all') },
-# CC_SPAMMY, sub { c('defang_spam') || c('defang_all') },
-# # CC_BADH.',3', 1, # NUL or CR character in header
-# # CC_BADH.',5', 1, # header line longer than 998 characters
-# # CC_BADH.',6', 1, # header field syntax error
-# CC_BADH, sub { c('defang_bad_header') || c('defang_all') },
-# CC_CATCHALL, sub { c('defang_all') },
+# %defang_maps_by_ccat = (
+# CC_VIRUS, sub { c('defang_virus') },
+# CC_BANNED, sub { c('defang_banned') },
+# CC_UNCHECKED, sub { c('defang_undecipherable') },
+# CC_SPAM, sub { c('defang_spam') },
+# CC_SPAMMY, sub { c('defang_spam') },
+# # CC_BADH.',3', 1, # NUL or CR character in header
+# # CC_BADH.',5', 1, # header line longer than 998 characters
+# # CC_BADH.',6', 1, # header field syntax error
+# CC_BADH, sub { c('defang_bad_header') },
# );
# %subject_tag_maps_by_ccat = (
# CC_VIRUS, [ '***INFECTED*** ' ],
@@ -506,8 +527,11 @@ use strict;
# CC_SPAM, sub { ca('spam_admin_maps') },
# CC_BADH, sub { ca('bad_header_admin_maps') },
# );
+# %always_bcc_by_ccat = (
+# CC_CATCHALL, sub { c('always_bcc') },
+# );
# %dsn_bcc_by_ccat = (
-# CC_CATCHALL, sub { c('dsn_bcc') },
+# CC_CATCHALL, sub { c('dsn_bcc') },
# );
# %mailfrom_notify_admin_by_ccat = (
# CC_SPAM, sub { c('mailfrom_notify_spamadmin') },
@@ -560,6 +584,7 @@ use strict;
# CC_BADH, sub { ca('addr_extension_bad_header_maps') },
# # CC_OVERSIZED, 'oversized';
# );
+# %addr_rewrite_maps_by_ccat = ( );
## POLICY BANKS
@@ -579,8 +604,9 @@ use strict;
## $policy_bank_name $protocol @inet_acl
## $myhostname $syslog_ident $syslog_facility $syslog_priority
## $log_level $log_templ $log_recip_templ
- ## $forward_method $notify_method $resend_method $release_method
- ## $os_fingerprint_method @smtpd_discard_ehlo_keywords
+ ## $forward_method $notify_method $resend_method
+ ## $release_method $requeue_method
+ ## $os_fingerprint_method $originating @smtpd_discard_ehlo_keywords
## $propagate_dsn_if_possible $terminate_dsn_on_notify_success
## $amavis_auth_user $amavis_auth_pass $auth_reauthenticate_forwarded
## $auth_required_out $auth_required_inp $auth_required_release
@@ -593,14 +619,17 @@ use strict;
## $undecipherable_subject_tag $localpart_is_case_sensitive
## $recipient_delimiter $replace_existing_extension
## $hdr_encoding $bdy_encoding $hdr_encoding_qb
- ## $insert_received_line $append_header_fields_to_bottom
+ ## $allow_disclaimers $insert_received_line
+ ## $append_header_fields_to_bottom $prepend_header_fields_hdridx
## $allow_fixing_improper_header $allow_fixing_improper_header_folding
- ## %allowed_added_header_fields
+ ## %allowed_added_header_fields %allowed_header_tests
## $X_HEADER_TAG $X_HEADER_LINE $notify_xmailer_header
## $remove_existing_x_scanned_headers $remove_existing_spam_headers
## %sql_clause %local_delivery_aliases $banned_namepath_re
## $per_recip_whitelist_sender_lookup_tables
## $per_recip_blacklist_sender_lookup_tables
+ ## @anomy_sanitizer_args @altermime_args_defang
+ ## @altermime_args_disclaimer @disclaimer_options_bysender_maps
##
## @local_domains_maps @mynetworks_maps
## @newvirus_admin_maps @banned_filename_maps
@@ -614,16 +643,17 @@ use strict;
## @message_size_limit_maps @debug_sender_maps
## @bypass_virus_checks_maps @bypass_spam_checks_maps
## @bypass_banned_checks_maps @bypass_header_checks_maps
- ## @viruses_that_fake_sender_maps
+ ## @viruses_that_fake_sender_maps @virus_name_to_spam_score_maps
##
## %final_destiny_by_ccat %lovers_maps_by_ccat
- ## %defang_by_ccat %subject_tag_maps_by_ccat
+ ## %defang_maps_by_ccat %subject_tag_maps_by_ccat
## %quarantine_method_by_ccat %quarantine_to_maps_by_ccat
## %notify_admin_templ_by_ccat %notify_recips_templ_by_ccat
## %notify_sender_templ_by_ccat %warnsender_by_ccat
## %hdrfrom_notify_admin_by_ccat %mailfrom_notify_admin_by_ccat
## %hdrfrom_notify_recip_by_ccat %mailfrom_notify_recip_by_ccat
## %hdrfrom_notify_sender_by_ccat
- ## %admin_maps_by_ccat %dsn_bcc_by_ccat
- ## %warnrecip_maps_by_ccat %addr_extension_maps_by_ccat
+ ## %admin_maps_by_ccat %warnrecip_maps_by_ccat
+ ## %always_bcc_by_ccat %dsn_bcc_by_ccat
+ ## %addr_extension_maps_by_ccat %addr_rewrite_maps_by_ccat
1;
diff --git a/amavisd.conf-sample b/amavisd.conf-sample
--- a/amavisd.conf-sample
+++ b/amavisd.conf-sample
@@ -97,8 +97,7 @@ use strict;
#$lock_file = "$MYHOME/amavisd.lock"; # (default is "$MYHOME/amavisd.lock"), -L
# set environment variables if you want (no defaults):
-$ENV{TMPDIR} = $TEMPBASE; # wise to set TMPDIR, but not obligatory
-#...
+$ENV{TMPDIR} = $TEMPBASE; # used for SA temporary files, by some decoders, etc.
$enable_db = 1; # enable use of BerkeleyDB/libdb (SNMP and nanny)
$enable_global_cache = 1; # enable use of libdb-based cache if $enable_db=1
@@ -156,7 +155,7 @@ use strict;
# master.cf file, like the '2' in the: smtp-amavis unix - - n - 2 smtp
#
$max_servers = 2; # number of pre-forked children (default 2), -m
-$max_requests = 20; # retire a child after that many accepts (default 10)
+$max_requests = 20; # retire a child after that many accepts (default 20)
$child_timeout=5*60; # abort child if it does not complete its processing in
# approximately n seconds (default: 8*60 seconds)
@@ -170,8 +169,9 @@ use strict;
# For more refined controls leave the following two lines commented out,
# and see further down what these two lookup lists really mean.
#
-# @bypass_virus_checks_maps = (1); # uncomment to DISABLE anti-virus code
-# @bypass_spam_checks_maps = (1); # uncomment to DISABLE anti-spam code
+# @bypass_virus_checks_maps = (1); # controls running of anti-virus code
+# @bypass_spam_checks_maps = (1); # controls running of anti-spam code
+# $bypass_decode_parts = 1; # controls running of decoders&dearchivers
#
# Any setting can be changed with a new assignment, so make sure
# you do not unintentionally override these settings further down!
@@ -325,43 +325,45 @@ use strict;
# $log_templ = <<'EOD';
# [?%#D|#|Passed #
-# [? [:ccat_maj] |OTHER|CLEAN|TEMPFAIL|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
+# [? [:ccat|major] |OTHER|CLEAN|MTA-BLOCKED|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
# UNCHECKED|BANNED (%F)|INFECTED (%V)]#
-# #([:ccat_maj],[:ccat_min])#
# , [? %p ||%p ][?%a||[?%l||LOCAL ]\[%a\] ][?%e||\[%e\] ]%s -> [%D|,]#
# [? %q ||, quarantine: %q]#
# [? %Q ||, Queue-ID: %Q]#
# [? %m ||, Message-ID: %m]#
# [? %r ||, Resent-Message-ID: %r]#
# , mail_id: %i#
-# , Hits: [:SCORE][~%c|["[+-].*[+-]"]|[" (%c)"]]#
+# , Hits: [:SCORE]#
# , size: %z#
-# #, fwd_to: [:remote_mta]#
# [~[:remote_mta_smtp_response]|["^$"]||[", queued_as: "]]\
# [remote_mta_smtp_response|[~%x|["queued as ([0-9A-Z]+)$"]|["%1"]|["%0"]]|/]#
# [? [:header_field|Subject] ||, Subject: [:dquote|[:header_field|Subject]]]#
# [? [:header_field|From] ||, From: [:uquote|[:header_field|From]]]#
+# [? [:useragent|name] ||, [:useragent|name]: [:uquote|[:useragent|body]]]#
# [? %#T ||, Tests: \[[%T|,]\]]#
-# [? [:AUTOLEARN] ||, autolearn=[:AUTOLEARN]]#
+# [:supplementary_info|SCTYPE|, shortcircuit=%%s]#
+# [:supplementary_info|AUTOLEARN|, autolearn=%%s]#
# , %y ms#
# ]
# [?%#O|#|Blocked #
-# [? [:ccat_maj] |OTHER|CLEAN|TEMPFAIL|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
+# [? [:ccat|major|blocking] |#
+# OTHER|CLEAN|MTA-BLOCKED|OVERSIZED|BAD-HEADER|SPAMMY|SPAM|\
# UNCHECKED|BANNED (%F)|INFECTED (%V)]#
-# #([:ccat_maj],[:ccat_min])#
# , [? %p ||%p ][?%a||[?%l||LOCAL ]\[%a\] ][?%e||\[%e\] ]%s -> [%O|,]#
# [? %q ||, quarantine: %q]#
# [? %Q ||, Queue-ID: %Q]#
# [? %m ||, Message-ID: %m]#
# [? %r ||, Resent-Message-ID: %r]#
# , mail_id: %i#
-# , Hits: [:SCORE][~%c|["[+-].*[+-]"]|[" (%c)"]]#
+# , Hits: [:SCORE]#
# , size: %z#
# #, smtp_resp: [:smtp_response]#
# [? [:header_field|Subject] ||, Subject: [:dquote|[:header_field|Subject]]]#
# [? [:header_field|From] ||, From: [:uquote|[:header_field|From]]]#
+# [? [:useragent|name] ||, [:useragent|name]: [:uquote|[:useragent|body]]]#
# [? %#T ||, Tests: \[[%T|,]\]]#
-# [? [:AUTOLEARN] ||, autolearn=[:AUTOLEARN]]#
+# [:supplementary_info|SCTYPE|, shortcircuit=%%s]#
+# [:supplementary_info|AUTOLEARN|, autolearn=%%s]#
# , %y ms#
# ]
# EOD
@@ -568,9 +570,6 @@ use strict;
# actually passed to recipients ($final_*_destiny=D_PASS, or *_lovers*).
# Bounces or rejects produce non-delivery status notification regardless.
#
-# Notify sender of banned files?
-#$warnbannedsender = 1; # (defaults to false (undef))
-#
# Notify sender of syntactically invalid header containing non-ASCII chars?
#$warnbadhsender = 1; # (defaults to false (undef))
@@ -720,6 +719,10 @@ use strict;
#$virus_quarantine_method = $spam_quarantine_method =
# $banned_files_quarantine_method = $bad_header_quarantine_method = 'sql:';
+# Send copy of every mail to an archival mail address:
+#$archive_quarantine_method = $notify_method;
+#@archive_quarantine_to_maps = ( 'collector at example.com' );
+
# When using the 'local:' quarantine method (default), the following applies:
#
@@ -865,9 +868,8 @@ use strict;
# set $bypass_decode_parts to true if you only do spam scanning, or if you
# have a good virus scanner that can deal with compression and recursively
# unpacking archives by itself, and save amavisd the trouble.
-# Disabling decoding also causes banned_files checking to only see
-# MIME names and MIME content types, not the content classification types
-# as provided by the file(1) utility.
+# Disabling decoding also causes banned_files checking NOT to see MIME types
+# and content classification types as provided by the file(1) utility.
# It is a double-edged sword, make sure you know what you are doing!
#
#$bypass_decode_parts = 1; # (defaults to false)
@@ -930,14 +932,14 @@ use strict;
qr'^\.(exe-ms|dll)$', # banned file(1) types, rudimentary
# qr'^\.(exe|lha|tnef|cab|dll)$', # banned file(1) types
-### BLOCK THE FOLLOWING, EXCEPT WITHIN UNIX ARHIVES:
+### BLOCK THE FOLLOWING, EXCEPT WITHIN UNIX ARCHIVES:
# [ qr'^\.(gz|bz2)$' => 0 ], # allow any in gzip or bzip2
[ qr'^\.(rpm|cpio|tar)$' => 0 ], # allow any in Unix-type archives
qr'.\.(pif|scr)$'i, # banned extensions - rudimentary
# qr'^\.zip$', # block zip type
-### BLOCK THE FOLLOWING, EXCEPT WITHIN ARHIVES:
+### BLOCK THE FOLLOWING, EXCEPT WITHIN ARCHIVES:
# [ qr'^\.(zip|rar|arc|arj|zoo)$'=> 0 ], # allow any within these archives
qr'^application/x-msdownload$'i, # block these MIME types
@@ -951,7 +953,7 @@ use strict;
# qr'^\.wmf$', # Windows Metafile file(1) type
# block certain double extensions in filenames
- qr'\.[^./]*[A-Za-z][^./]*\.(exe|vbs|pif|scr|bat|cmd|com|cpl|dll)\.?$'i,
+ qr'\.[^./]*[A-Za-z][^./]*\.\s*(exe|vbs|pif|scr|bat|cmd|com|cpl|dll)[.\s]*$'i,
# qr'\{[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}\}?'i, # Class ID CLSID, strict
# qr'\{[0-9a-z]{4,}(-[0-9a-z]{4,}){0,7}\}?'i, # Class ID extension CLSID, loose
@@ -962,6 +964,8 @@ use strict;
# inf|ins|isp|js|jse|lnk|mda|mdb|mde|mdw|mdt|mdz|msc|msi|msp|mst|
# ops|pcd|pif|prg|reg|scr|sct|shb|shs|vb|vbe|vbs|
# wmf|wsc|wsf|wsh)$'ix, # banned ext - long
+# qr'.\.(ani|cur|ico)$'i, # banned cursors and icons filename
+# qr'^\.ani$', # banned animated cursor file(1) type
# qr'.\.(mim|b64|bhx|hqx|xxe|uu|uue)$'i, # banned extension - WinZip vulnerab.
);
@@ -1004,7 +1008,7 @@ use strict;
# ^ (.*\t)? T=(exe|lha|tnef|cab|dll) (\t.*)? $'xm, # banned file(1) types
-### BLOCK THE FOLLOWING, EXCEPT WITHIN UNIX ARHIVES:
+### BLOCK THE FOLLOWING, EXCEPT WITHIN UNIX ARCHIVES:
# # within traditional gzip and bzip2 allow any name and type
# [ qr'(?#rule-3) ^ (.*\t)? T=(gz|bz2) (\t.*)? $'xmi => 0 ], # allow
@@ -1020,7 +1024,7 @@ use strict;
# qr'(?#rule-5) ^ (.*\t)? T=zip (\t.*)? (.*\n)+ .* $'xmi,
-### BLOCK THE FOLLOWING, EXCEPT WITHIN ARHIVES OR CRYPTED:
+### BLOCK THE FOLLOWING, EXCEPT WITHIN ARCHIVES OR CRYPTED:
# # within PC archives allow any types or names at any depth
# [ qr'(?#rule-7) ^ (.*\t)? T=(zip|rar|arc|arj|zoo) (\t.*)? $'xmi => 0 ], # ok
@@ -1044,11 +1048,12 @@ use strict;
# qr'(?#No Metafile MIME) ^(.*\t)? M=application/x-msmetafile (\t.*)? $'xmi,
# qr'(?#No Metafile MIME) ^(.*\t)? M=image/x-wmf (\t.*)? $'xmi,
# qr'(?#No Metafile file) ^(.*\t)? T=wmf (\t.*)? $'xm,
+# qr'(?#No animated cursors) ^(.*\t)? T=ani (\t.*)? $'xm,
# block certain double extensions in filenames
qr'(?# BLOCK DOUBLE-EXTENSIONS )
- ^ (.*\t)? N= [^\t\n]* \. [^./\t\n]* [A-Za-z] [^./\t\n]* \.
- (exe|vbs|pif|scr|bat|cmd|com|cpl|dll) \.? (\t.*)? $'xmi,
+ ^ (.*\t)? N= [^\t\n]* \. [^./\t\n]* [A-Za-z] [^./\t\n]* \. \ *
+ (exe|vbs|pif|scr|bat|cmd|com|cpl|dll) [. ]* (\t.*)? $'xmi,
[ qr'(?# BLOCK EMPTY MIME PART APPLICATION/OCTET-STREAM )
^ (.*\t)? M=application/(octet-stream|x-msdownload|x-msdos-program)
@@ -1090,6 +1095,9 @@ use strict;
# inf|ins|isp|js|jse|lnk|mda|mdb|mde|mdw|mdt|mdz|msc|msi|msp|mst|
# ops|pcd|pif|prg|reg|scr|sct|shb|shs|vb|vbe|vbs|
# wmf|wsc|wsf|wsh) (\t.*)? $'xmi,
+
+# qr'(?# BLOCK CURSOR AND ICON NAME EXENSIONS )
+# ^ (.*\t)? N= [^\t\n]* \. (ani|cur|ico) (\t.*)? $'xmi,
# # banned filename extensions anywhere - WinZip vulnerability (pre-V9)
# qr'(?# BLOCK WinZip VULNERABILITY EXENSIONS )
@@ -1692,10 +1700,10 @@ use strict;
['rpm', \&do_uncompress, ['rpm2cpio.pl','rpm2cpio'] ],
['cpio', \&do_pax_cpio, ['pax','gcpio','cpio'] ],
['tar', \&do_pax_cpio, ['pax','gcpio','cpio'] ],
- ['tar', \&do_tar],
['deb', \&do_ar, 'ar'],
# ['a', \&do_ar, 'ar'], # unpacking .a seems an overkill
['zip', \&do_unzip],
+ ['7z', \&do_7zip, ['7zr','7za','7z'] ],
['rar', \&do_unrar, ['rar','unrar'] ],
['arj', \&do_unarj, ['arj','unarj'] ],
['arc', \&do_arc, ['nomarch','arc'] ],
@@ -1737,11 +1745,11 @@ use strict;
# undef disables this feature and is a default;
# see also $sa_quarantine_cutoff_level above, which only controls quarantining
-# $penpals_bonus_score = 4; # (positive) score by which spam score is lowered
+# $penpals_bonus_score = 5; # (positive) score by which spam score is lowered
# when sender is known to have previously received mail from our
# local user from this mail system; zero or undef disables penpals
# lookups in SQL; default: undef
-# $penpals_halflife = 7*24*60*60; # exponential decay time constant in seconds;
+# $penpals_halflife = 10*24*60*60; #exponential decay time constant in seconds;
# penpal bonus is halved for each halflife period from the last mail
# sent by a local user to a current mail's sender; default: 7 days
# $penpals_threshold_low = 1.0; # no need for pen pals lookup on low spam score
@@ -1800,6 +1808,10 @@ use strict;
#@spam_subject_tag_maps = ('[possible-spam:_SCORE_] ');
#@spam_subject_tag2_maps = ('***SPAM*** _SCORE_ (_REQD_) ');
#@spam_subject_tag3_maps = ('***BLATANT*SPAM**** _SCORE_ (_REQD_) ');
+# another examples, using _maps_by_ccat:
+#$subject_tag_maps_by_ccat{+CC_CLEAN} = [
+# { lc('TestUser at example.net') =>
+# '**TEST:_U_,hits=_SCORE_,req=_REQD_,amid=_TASKID_,mid=_MAILID_**' } ];
#$sa_spam_modifies_subj = 1; # in @spam_modifies_subj_maps, default is true
@@ -1816,14 +1828,20 @@ use strict;
# @av_scanners is a list of n-tuples, where fields semantics is:
# 1. av scanner plain name, to be used in log and reports;
-# 2. scanner program name; this string will be submitted to subroutine
+# 2a.scanner program name; this string will be submitted to subroutine
# find_external_programs(), which will try to find the full program path
# name during startup; if program is not found, this scanner is disabled.
# Besides a simple string (full program path name or just the basename
# to be looked for in PATH), this may be an array ref of alternative
# program names or full paths - the first match in the list will be used;
-# As a special case for more complex scanners, this field may be
-# a subroutine reference, and the whole n-tuple is passed to it as args.
+# 2b.alternatively, this second field may be a subroutine reference,
+# and the whole n-tuple entry is passed to it as args; it should return
+# a triple: ($scan_status,$output,$virusnames_ref), where:
+# - $scan_status is: true if a virus was found, 0 if no viruses,
+# undef if scanner was unable to complete its job (failed);
+# - $output is an optional result string to appear in logging and macro %v;
+# - $virusnames_ref is a ref to a list of detected virus names (may be
+# undef or a ref to an empty list);
# 3. command arguments to be given to the scanner program;
# a substring {} will be replaced by the directory name to be scanned, i.e.
# "$tempdir/parts", a "*" will be replaced by base file names of parts;
@@ -1835,6 +1853,7 @@ use strict;
# quick partial scanners such as jpeg checker);
# 5. an array ref of av scanner exit status values, or a regexp (to be
# matched against scanner output), indicating VIRUSES WERE FOUND;
+# a value undef may be used and it never matches (for consistency with 4.);
# Note: the virus match prevails over a 'not found' match, so it is safe
# even if the no. 4. matches for viruses too;
# 6. a regexp (to be matched against scanner output), returning a list
@@ -1879,7 +1898,7 @@ use strict;
@av_scanners = (
-# ### http://www.vanja.com/tools/sophie/
+# ### http://www.clanfield.info/sophie/ (http://www.vanja.com/tools/sophie/)
# ['Sophie',
# \&ask_daemon, ["{}/\n", '/var/run/sophie'],
# qr/(?x)^ 0+ ( : | [\000\r\n]* $)/, qr/(?x)^ 1 ( : | [\000\r\n]* $)/,
@@ -1900,6 +1919,7 @@ use strict;
# # this entry; when running chrooted one may prefer socket "$MYHOME/clamd".
# ### http://www.clamav.net/ and CPAN (memory-hungry! clamd is preferred)
+# # note that Mail::ClamAV requires perl to be build with threading!
# ['Mail::ClamAV', \&ask_clamav, "*", [0], [1], qr/^INFECTED: (.+)/],
# ### http://www.openantivirus.org/
@@ -1992,7 +2012,7 @@ use strict;
['CentralCommand Vexira (new) vascan',
['vascan','/usr/lib/Vexira/vascan'],
"-a s --timeout=60 --temp=$TEMPBASE -y $QUARANTINEDIR ".
- "--vdb=/usr/lib/Vexira/vexira8.vdb --log=/var/log/vascan.log {}",
+ "--log=/var/log/vascan.log {}",
[0,3], [1,2,5],
qr/(?x)^\s* (?:virus|iworm|macro|mutant|sequence|trojan)\ found:\ ( [^\]\s']+ )\ \.\.\.\ / ],
# Adjust the path of the binary and the virus database as needed.
@@ -2029,12 +2049,21 @@ use strict;
qr/^(?:Info|Virus Name):\s+(.+)/ ],
# NOTE: check options and patterns to see which entry better applies
- ### http://www.f-secure.com/products/anti-virus/ version 4.65
+# ### http://www.f-secure.com/products/anti-virus/ version 4.65
+# ['F-Secure Antivirus for Linux servers',
+# ['/opt/f-secure/fsav/bin/fsav', 'fsav'],
+# '--delete=no --disinf=no --rename=no --archive=yes --auto=yes '.
+# '--dumb=yes --list=no --mime=yes {}', [0], [3,6,8],
+# qr/(?:infection|Infected|Suspected): (.+)/ ],
+
+ ### http://www.f-secure.com/products/anti-virus/ version 5.52
['F-Secure Antivirus for Linux servers',
['/opt/f-secure/fsav/bin/fsav', 'fsav'],
- '--delete=no --disinf=no --rename=no --archive=yes --auto=yes '.
- '--dumb=yes --list=no --mime=yes {}', [0], [3,6,8],
- qr/(?:infection|Infected|Suspected): (.+)/ ],
+ '--virus-action1=report --archive=yes --auto=yes '.
+ '--dumb=yes --list=no --mime=yes {}', [0], [3,4,6,8],
+ qr/(?:infection|Infected|Suspected|Riskware): (.+)/ ],
+ # NOTE: internal archive handling may be switched off by '--archive=no'
+ # to prevent fsav from exiting with status 9 on broken archives
# ### http://www.avast.com/
# ['avast! Antivirus daemon',
@@ -2069,13 +2098,18 @@ use strict;
'-s -q {}', [0], [1..7],
qr/^... (\S+)/ ],
- ### http://www.nod32.com/, version v2.52 and above
- ['ESET NOD32 for Linux Mail servers',
- ['/opt/eset/nod32/bin/nod32cli', 'nod32cli'],
- '--subdir --files -z --sfx --rtp --adware --unsafe --pattern --heur '.
- '-w -a --action-on-infected=accept --action-on-uncleanable=accept '.
- '--action-on-notscanned=accept {}',
- [0,3], [1,2], qr/virus="([^"]+)"/ ],
+# ### http://www.nod32.com/, version v2.52 and above
+# ['ESET NOD32 for Linux Mail servers',
+# ['/opt/eset/nod32/bin/nod32cli', 'nod32cli'],
+# '--subdir --files -z --sfx --rtp --adware --unsafe --pattern --heur '.
+# '-w -a --action-on-infected=accept --action-on-uncleanable=accept '.
+# '--action-on-notscanned=accept {}',
+# [0,3], [1,2], qr/virus="([^"]+)"/ ],
+
+ ### http://www.eset.com/, version v2.7
+ ['ESET NOD32 Linux Mail Server - command line interface',
+ ['/usr/bin/nod32cli', '/opt/eset/nod32/bin/nod32cli', 'nod32cli'],
+ '--subdir {}', [0,3], [1,2], qr/virus="([^"]+)"/ ],
## http://www.nod32.com/, NOD32LFS version 2.5 and above
['ESET NOD32 for Linux File servers',
@@ -2175,6 +2209,11 @@ use strict;
qr/(?:suspected|infected): (.*)(?:\033|$)/ ],
# consider also: --all --nowarn --alev=15 --flev=15. The --all argument may
# not apply to your version of bdc, check documentation and see 'bdc --help'
+
+ ### ArcaVir for Linux and Unix http://www.arcabit.pl/
+ ['ArcaVir for Linux', ['arcacmd','arcacmd.static'],
+ '-v 1 -summary 0 -s {}', [0], [1,2],
+ qr/(?:VIR|WIR):[ \t]*(.+)/ ],
# ['File::Scan', sub {Amavis::AV::ask_av(sub{
# use File::Scan; my($fn)=@_;
@@ -2220,8 +2259,14 @@ use strict;
# }; @r}, @_) },
# ["{}/*"], undef, [1], qr/^(bad jpeg: .*)$/ ],
+# ### an example/testing/template virus scanner (external), wastes 3 seconds
# ['wasteful sleeper example',
-# '/bin/sleep', '3',
+# '/bin/sleep', '3', # calls external program
+# undef, undef, qr/no such/ ],
+
+# ### an example/testing/template virus scanner (internal), does nothing
+# ['null',
+# sub {}, ["{}"], # supplies its own subroutine, no external program
# undef, undef, qr/no such/ ],
);
@@ -2246,23 +2291,24 @@ use strict;
### http://www.f-prot.com/ - backs up F-Prot Daemon
['FRISK F-Prot Antivirus', ['f-prot','f-prot.sh'],
- '-dumb -archive -packed {}', [0,8], [3,6],
- qr/Infection: (.+)|\s+contains\s+(.+)$/ ],
+ '-dumb -archive -packed {}', [0,8], [3,6], # or: [0], [3,6,8],
+ qr/(?:Infection:|security risk named) (.+)|\s+contains\s+(.+)$/ ],
### http://www.trendmicro.com/ - backs up Trophie
['Trend Micro FileScanner', ['/etc/iscan/vscan','vscan'],
'-za -a {}', [0], qr/Found virus/, qr/Found virus (.+) in/ ],
### http://www.sald.com/, http://drweb.imshop.de/ - backs up DrWebD
- ['drweb - DrWeb Antivirus',
+ ['drweb - DrWeb Antivirus', # security LHA hole in Dr.Web 4.33 and earlier
['/usr/local/drweb/drweb', '/opt/drweb/drweb', 'drweb'],
'-path={} -al -go -ot -cn -upn -ok-',
[0,32], [1,9,33], qr' infected (?:with|by)(?: virus)? (.*)$'],
### http://www.kaspersky.com/
['Kaspersky Antivirus v5.5',
- ['/opt/kav/5.5/kav4unix/bin/kavscanner',
- '/opt/kav/5.5/kav4mailservers/bin/kavscanner','kavscanner'],
+ ['/opt/kaspersky/kav4fs/bin/kav4fs-kavscanner',
+ '/opt/kav/5.5/kav4unix/bin/kavscanner',
+ '/opt/kav/5.5/kav4mailservers/bin/kavscanner', 'kavscanner'],
'-i0 -xn -xp -mn -R -ePASBME {}/*', [0,10,15], [5,20,21,25],
qr/(?:INFECTED|WARNING|SUSPICION|SUSPICIOUS) (.*)/ ,
# sub {chdir('/opt/kav/bin') or die "Can't chdir to kav: $!"},
@@ -2275,11 +2321,12 @@ use strict;
#
# ### http://www.sophos.com/ - backs up Sophie or SAVI-Perl
# ['Sophos Anti Virus (sweep)', 'sweep',
-# '-nb -f -all -rec -ss -sc -archive -cab -tnef --no-reset-atime {}',
+# '-nb -f -all -rec -ss -sc -archive -cab -mime -oe -tnef '.
+# '--no-reset-atime {}',
# [0,2], qr/Virus .*? found/,
# qr/^>>> Virus(?: fragment)? '?(.*?)'? found/,
# ],
-# # other options to consider: -mime -oe -idedir=/usr/local/sav
+# # other options to consider: -idedir=/usr/local/sav
# always succeeds (uncomment to consider mail clean if all other scanners fail)
# ['always-clean', sub {0}],
@@ -2359,7 +2406,7 @@ use strict;
#
#$policy_bank{'AM.PDP-SOCK'} = {
# protocol => 'AM.PDP', # Amavis policy delegation protocol
-# auth_required_release => 0, # don't require secret_id for amavisd-release
+# auth_required_release => 0, # do not require secret_id for amavisd-release
#};
#
#$policy_bank{'AM.PDP-INET'} = {
@@ -2373,6 +2420,7 @@ use strict;
#
# $terminate_dsn_on_notify_success = 1;
# $policy_bank{'MYNETS'} = { # mail originating from @mynetworks
+# originating => 1, # is true in MYNETS by deflt, but let's make it explicit
# terminate_dsn_on_notify_success => 0,
# spam_kill_level_maps => 6.9,
# syslog_facility => 'LOCAL4', # tell syslog to log to a separate file
@@ -2428,5 +2476,10 @@ use strict;
# }
#}
+
+# invoke custom hooks or additional configuration files:
+# include_config_files('/etc/amavisd-custom.conf');
+
+
#-------------
1; # insure a defined return
diff --git a/debian/changelog b/debian/changelog
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@ amavisd-new (1:2.4.3-2) unstable; urgenc
+amavisd-new (1:2.5.2-1) unstable; urgency=low
+
+ * New upstream release.
+ * Depend on unrar-free | unrar and use unrar-free in configuration
+ (Closes: #442010)
+
+ -- Ondřej Surý <ondrej at debian.org> Thu, 13 Sep 2007 10:16:45 +0200
+
amavisd-new (1:2.4.3-2) unstable; urgency=low
* Don't remove the amavisd user (Closes: #431853)
diff --git a/debian/control b/debian/control
--- a/debian/control
+++ b/debian/control
@@ -12,7 +12,7 @@ Provides: amavis
Provides: amavis
Conflicts: amavis
Replaces: amavis
-Suggests: spamassassin (>= 3.1.0a), clamav, clamav-daemon, lha, arj, unrar, zoo, nomarch, cpio, lzop, cabextract, apt-listchanges (>= 2.35), libnet-ldap-perl (>= 1:0.32), libauthen-sasl-perl, libdbi-perl (>= 1.43)
+Suggests: spamassassin (>= 3.1.0a), clamav, clamav-daemon, lha, arj, unrar-free | unrar, zoo, nomarch, cpio, lzop, cabextract, apt-listchanges (>= 2.35), libnet-ldap-perl (>= 1:0.32), libauthen-sasl-perl, libdbi-perl (>= 1.43)
Description: Interface between MTA and virus scanner/content filters
AMaViSd-new is a script that interfaces a mail transport agent (MTA) with
zero or more virus scanners, and spamassassin (optional).
diff --git a/debian/etc/conf.d/01-debian b/debian/etc/conf.d/01-debian
--- a/debian/etc/conf.d/01-debian
+++ b/debian/etc/conf.d/01-debian
@@ -34,7 +34,7 @@ use strict;
$arc = ['nomarch', 'arc'];
$unarj = ['arj', 'unarj'];
#$unrar = ['rar', 'unrar']; #disabled (non-free, no security support)
-$unrar = undef;
+$unrar = 'unrar-free';
$zoo = 'zoo';
#$lha = 'lha'; #disabled (non-free, no security support)
$lha = undef;
diff --git a/debian/patches/30_conf.d_support_builtin.dpatch b/debian/patches/30_conf.d_support_builtin.dpatch
--- a/debian/patches/30_conf.d_support_builtin.dpatch
+++ b/debian/patches/30_conf.d_support_builtin.dpatch
@@ -6,10 +6,10 @@
## DP: in the main legacy config file.
@DPATCH@
-diff -urNad unstable~/amavisd unstable/amavisd
---- unstable~/amavisd 2006-11-04 00:27:10.077559871 -0300
-+++ unstable/amavisd 2006-11-04 00:27:10.197545289 -0300
-@@ -2181,6 +2181,25 @@
+diff -urNad amavisd-new-2.5.2~/amavisd amavisd-new-2.5.2/amavisd
+--- amavisd-new-2.5.2~/amavisd 2007-06-27 12:43:00.000000000 +0200
++++ amavisd-new-2.5.2/amavisd 2007-09-13 10:21:11.000000000 +0200
+@@ -2394,6 +2394,25 @@
Amavis::Util::read_text("$dir/template-spam-admin.txt", $file_chset);
}
@@ -35,7 +35,7 @@ diff -urNad unstable~/amavisd unstable/a
#use CDB_File;
#sub tie_hash($$) {
# my($hashref, $filename) = @_;
-@@ -9811,7 +9830,11 @@
+@@ -11050,7 +11069,10 @@
Amavis::Lookup::RE->new(@$Amavis::Conf::map_full_type_to_short_type_re);
# default location of the config file if none specified
@@ -44,7 +44,6 @@ diff -urNad unstable~/amavisd unstable/a
+ @config_files = Amavis::Util::find_config_files('/usr/share/amavis/conf.d',
+ '/etc/amavis/conf.d');
+}
-+
- # Read/execute the config file, which may override default settings
- Amavis::Conf::read_config(@config_files);
-
+ # Read and evaluate config files, which may override default settings
+ Amavis::Conf::include_config_files(@config_files);
+ Amavis::Conf::supply_after_defaults();
diff --git a/debian/patches/40_fix_paths.dpatch b/debian/patches/40_fix_paths.dpatch
--- a/debian/patches/40_fix_paths.dpatch
+++ b/debian/patches/40_fix_paths.dpatch
@@ -5,9 +5,45 @@
## DP: Fix references to paths that are different in Debian
@DPATCH@
-diff -urNad unstable~/README_FILES/README.chroot unstable/README_FILES/README.chroot
---- unstable~/README_FILES/README.chroot 2006-11-04 00:32:39.000000000 -0300
-+++ unstable/README_FILES/README.chroot 2006-11-04 00:32:48.798409618 -0300
+diff -urNad amavisd-new-2.5.2~/amavisd-agent amavisd-new-2.5.2/amavisd-agent
+--- amavisd-new-2.5.2~/amavisd-agent 2007-05-22 14:53:55.000000000 +0200
++++ amavisd-new-2.5.2/amavisd-agent 2007-09-13 10:32:50.000000000 +0200
+@@ -146,7 +146,7 @@
+ { die "Usage: $0 [ -w <wait-interval> ]\n" }
+ }
+ my($env) = BerkeleyDB::Env->new(
+- '-Home'=>'/var/amavis/db', '-Flags'=> DB_INIT_CDB | DB_INIT_MPOOL);
++ '-Home'=>'/var/lib/amavis/db', '-Flags'=> DB_INIT_CDB | DB_INIT_MPOOL);
+ defined $env or die "BDB no env: $BerkeleyDB::Error $!";
+ my($db) = BerkeleyDB::Hash->new(
+ '-Filename'=>'snmp.db', '-Flags'=>DB_RDONLY, '-Env'=>$env );
+diff -urNad amavisd-new-2.5.2~/amavisd-nanny amavisd-new-2.5.2/amavisd-nanny
+--- amavisd-new-2.5.2~/amavisd-nanny 2007-03-29 12:26:48.000000000 +0200
++++ amavisd-new-2.5.2/amavisd-nanny 2007-09-13 10:32:50.000000000 +0200
+@@ -56,7 +56,7 @@
+ my($activettl) = 10*60; # stuck active children are sent a SIGTERM
+ # after this many seconds
+
+-my($db_home) = '/var/amavis/db'; # DB databases directory
++my($db_home) = '/var/lib/amavis/db'; # DB databases directory
+ my($dbfile) = 'nanny.db';
+ my($wakeuptime) = 2; # sleep time in seconds, may be fractional
+
+diff -urNad amavisd-new-2.5.2~/amavisd-release amavisd-new-2.5.2/amavisd-release
+--- amavisd-new-2.5.2~/amavisd-release 2007-06-11 14:51:09.000000000 +0200
++++ amavisd-new-2.5.2/amavisd-release 2007-09-13 10:32:50.000000000 +0200
+@@ -79,7 +79,7 @@
+
+ $log_level = 1;
+ # $socketname = '127.0.0.1:9998';
+- $socketname = '/var/amavis/amavisd.sock';
++ $socketname = '/var/run/amavis/amavisd.sock';
+
+ sub sanitize_str {
+ my($str, $keep_eol) = @_;
+diff -urNad amavisd-new-2.5.2~/README_FILES/README.chroot amavisd-new-2.5.2/README_FILES/README.chroot
+--- amavisd-new-2.5.2~/README_FILES/README.chroot 2006-01-14 02:02:09.000000000 +0100
++++ amavisd-new-2.5.2/README_FILES/README.chroot 2007-09-13 10:32:50.000000000 +0200
@@ -23,6 +23,9 @@
If you have Postfix, check its chroot setup script for further hints:
postfix-xxx/examples/chroot-setup/<YOUR-OS> and BASIC_CONFIGURATION_README.
@@ -18,9 +54,9 @@ diff -urNad unstable~/README_FILES/READM
exit # This is NOT an automatic script!!!
# Don't execute commands without knowing what they will do!!!
-diff -urNad unstable~/README_FILES/README.old.scanners unstable/README_FILES/README.old.scanners
---- unstable~/README_FILES/README.old.scanners 2006-11-04 00:32:39.000000000 -0300
-+++ unstable/README_FILES/README.old.scanners 2006-11-04 00:32:48.798409618 -0300
+diff -urNad amavisd-new-2.5.2~/README_FILES/README.old.scanners amavisd-new-2.5.2/README_FILES/README.old.scanners
+--- amavisd-new-2.5.2~/README_FILES/README.old.scanners 2005-04-25 01:16:20.000000000 +0200
++++ amavisd-new-2.5.2/README_FILES/README.old.scanners 2007-09-13 10:32:50.000000000 +0200
@@ -89,7 +89,7 @@
otherwise your logfiles don't show which file(s) is/are infected.
@@ -39,9 +75,9 @@ diff -urNad unstable~/README_FILES/READM
# End of perl script
-diff -urNad unstable~/README_FILES/README.performance unstable/README_FILES/README.performance
---- unstable~/README_FILES/README.performance 2006-11-04 00:32:39.000000000 -0300
-+++ unstable/README_FILES/README.performance 2006-11-04 00:32:48.799409497 -0300
+diff -urNad amavisd-new-2.5.2~/README_FILES/README.performance amavisd-new-2.5.2/README_FILES/README.performance
+--- amavisd-new-2.5.2~/README_FILES/README.performance 2005-08-12 16:15:59.000000000 +0200
++++ amavisd-new-2.5.2/README_FILES/README.performance 2007-09-13 10:32:50.000000000 +0200
@@ -16,7 +16,7 @@
Hopefully hardware matches expectations,
fast disks and enough memory are paramount.
@@ -51,9 +87,9 @@ diff -urNad unstable~/README_FILES/READM
where amavisd does mail unpacking.
| is there any suggested configuration for this
-diff -urNad unstable~/README_FILES/README.sendmail unstable/README_FILES/README.sendmail
---- unstable~/README_FILES/README.sendmail 2006-11-04 00:32:39.000000000 -0300
-+++ unstable/README_FILES/README.sendmail 2006-11-04 00:32:48.799409497 -0300
+diff -urNad amavisd-new-2.5.2~/README_FILES/README.sendmail amavisd-new-2.5.2/README_FILES/README.sendmail
+--- amavisd-new-2.5.2~/README_FILES/README.sendmail 2005-04-25 01:16:34.000000000 +0200
++++ amavisd-new-2.5.2/README_FILES/README.sendmail 2007-09-13 10:32:50.000000000 +0200
@@ -212,7 +212,7 @@
/var/spool/mqueue and /var/spool/mqamavis is owned by amavis.
@@ -75,9 +111,9 @@ diff -urNad unstable~/README_FILES/READM
write access to.
NOTE: As sendmail will perform most tasks as user amavis now, it may
-diff -urNad unstable~/README_FILES/README.sendmail-dual unstable/README_FILES/README.sendmail-dual
---- unstable~/README_FILES/README.sendmail-dual 2006-11-04 00:32:39.000000000 -0300
-+++ unstable/README_FILES/README.sendmail-dual 2006-11-04 00:32:48.800409375 -0300
+diff -urNad amavisd-new-2.5.2~/README_FILES/README.sendmail-dual amavisd-new-2.5.2/README_FILES/README.sendmail-dual
+--- amavisd-new-2.5.2~/README_FILES/README.sendmail-dual 2006-09-15 22:36:47.000000000 +0200
++++ amavisd-new-2.5.2/README_FILES/README.sendmail-dual 2007-09-13 10:32:50.000000000 +0200
@@ -427,8 +427,8 @@
- Mail handling is I/O-intensive. For better performance one may place
@@ -89,39 +125,3 @@ diff -urNad unstable~/README_FILES/READM
- One of the important arguments for choosing the dual-MTA setup is to be
able to keep the number of content filtering processes under control,
-diff -urNad unstable~/amavisd-agent unstable/amavisd-agent
---- unstable~/amavisd-agent 2006-11-04 00:32:39.000000000 -0300
-+++ unstable/amavisd-agent 2006-11-04 00:32:48.797409740 -0300
-@@ -111,7 +111,7 @@
- # main program starts here
- $SIG{INT} = sub { die "\n" }; # do the END code block
- my($env) = BerkeleyDB::Env->new(
-- '-Home'=>'/var/amavis/db', '-Flags'=> DB_INIT_CDB | DB_INIT_MPOOL);
-+ '-Home'=>'/var/lib/amavis/db', '-Flags'=> DB_INIT_CDB | DB_INIT_MPOOL);
- defined $env or die "BDB no env: $BerkeleyDB::Error $!";
- my($db) = BerkeleyDB::Hash->new(
- '-Filename'=>'snmp.db', '-Flags'=>DB_RDONLY, '-Env'=>$env );
-diff -urNad unstable~/amavisd-nanny unstable/amavisd-nanny
---- unstable~/amavisd-nanny 2006-11-04 00:32:39.000000000 -0300
-+++ unstable/amavisd-nanny 2006-11-04 00:32:48.797409740 -0300
-@@ -51,7 +51,7 @@
- my($activettl) = 10*60; # stuck active children are sent a SIGTERM after this
- # many seconds
-
--my($db_home) = '/var/amavis/db'; # DB databases directory
-+my($db_home) = '/var/lib/amavis/db'; # DB databases directory
- my($dbfile) = 'nanny.db';
- my($wakeuptime) = 2; # seconds
-
-diff -urNad unstable~/amavisd-release unstable/amavisd-release
---- unstable~/amavisd-release 2006-11-04 00:32:39.000000000 -0300
-+++ unstable/amavisd-release 2006-11-04 00:32:48.797409740 -0300
-@@ -76,7 +76,7 @@
-
- $log_level = 1;
- # $socketname = '127.0.0.1:9998';
-- $socketname = '/var/amavis/amavisd.sock';
-+ $socketname = '/var/run/amavis/amavisd.sock';
-
- sub sanitize_str {
- my($str, $keep_eol) = @_;
diff --git a/p0f-analyzer.pl b/p0f-analyzer.pl
--- a/p0f-analyzer.pl
+++ b/p0f-analyzer.pl
@@ -44,7 +44,7 @@
use re 'taint';
use Socket;
use vars qw($VERSION);
- $VERSION = '1.000';
+ $VERSION = '1.310';
# Example usage:
# p0f -i bge0 -l 'tcp dst port 25' 2>&1 | p0f-analyzer.pl 2345
@@ -67,16 +67,24 @@ EOD
my($port) = untaint($ARGV[0]);
- my(@inet_acl) = qw( 127.0.0.1 ); # list of IP addresses from which queries
- # will be accepted, others are ignored
+# my($bind_addr) = '0.0.0.0'; # bind to all IPv4 interfaces
+ my($bind_addr) = '127.0.0.1'; # bind just to a loopback interface
+
+ my(@inet_acl) = qw( 127.0.0.1 ); # list of IP addresses from which queries
+ # will be accepted, others are ignored
my($retention_time) = 10*60; # time to keep collected information in cache
- my($debug) = 0; # nonzero enables log messages to STDERR
+ my($debug) = 0; # nonzero enables log messages to STDERR
printf STDERR ("p0f-analyzer version %s starting.\n", $VERSION) if $debug;
printf STDERR ("listening on UDP port %s, allowed queries from: %s\n",
$port, join(", ", at inet_acl)) if $debug;
socket(S, PF_INET, SOCK_DGRAM, getprotobyname('udp')) or die "socket: $!";
- bind(S, sockaddr_in($port,INADDR_ANY)) or die "bind: $!";
+
+ my($packed_addr);
+ $packed_addr = inet_aton($bind_addr)
+ or die "inet_aton: bad IP address [$bind_addr]: $!";
+ bind(S, sockaddr_in($port,$packed_addr))
+ or die "binding to [$bind_addr] failed: $!";
my($fn_sock) = fileno(S); my($fn_stdin) = fileno(STDIN);
my($rin,$rout); $rin = '';
vec($rin,$fn_sock,1) = 1; vec($rin,$fn_stdin,1) = 1;
@@ -94,40 +102,55 @@ EOD
if (!defined($clientaddr)) {
printf STDERR ("query from unknown client\n") if $debug;
} elsif (!grep {$_ eq $clientaddr} @inet_acl) {
- printf STDERR ("query from non-approved client: %s\n",
- $clientaddr) if $debug;
- } elsif ($inbuf !~ /^(\d+\.\d+\.\d+\.\d+) ([^ ]*)$/) {
- printf STDERR ("invalid query syntax from %s\n",
- $clientaddr) if $debug;
+ printf STDERR ("query from non-approved client: %s:%s\n",
+ $clientaddr,$port) if $debug;
+ } elsif ($port < 1024 || $port == 2049 || $port > 65536) {
+ printf STDERR ("query from non-approved port: %s:%s\n",
+ $clientaddr,$port) if $debug;
+ } elsif ($inbuf !~ /^(\d+\.\d+\.\d+\.\d+) (.*)$/s) {
+ printf STDERR ("invalid query syntax from %s:%s\n",
+ $clientaddr,$port) if $debug;
} else {
- printf STDERR ("query from %s: %s\n", $clientaddr,$inbuf) if $debug;
- my($src_ip,$nonce) = ($1,$2); my($resp) = ''; my($timestamp);
- if (exists($src{$src_ip})) {
- for my $e (@{$src{$src_ip}}) {
- $timestamp = $e->{t};
- if ($resp eq '') { $resp = $e->{d} }
- elsif ($e->{d} eq $resp) {}
- else { # keep the longest common string head
- my($j); my($resp_l) = length($resp);
- for ($j=0; $j<$resp_l; $j++)
- { last if substr($e->{d},$j,1) ne substr($resp,$j,1) }
- if ($j < $resp_l) {
-# printf("TRUNCATED to %d: %s %s => /%s/\n",
-# $j, $resp, $e->{d}, substr($resp,0,$j));
- $resp = substr($resp,0,$j);
+ my($src_ip,$nonce) = ($1,$2);
+ if (length($nonce) > 1024) {
+ printf STDERR ("invalid query from %s:%s, nonce too long: %d chrs\n",
+ $clientaddr,$port,length($nonce)) if $debug;
+ } elsif ($nonce !~ /^([\040-\177].*)\z/s) {
+ printf STDERR ("invalid query from %s:%s, forbidden char in nonce\n",
+ $clientaddr,$port) if $debug;
+ } else {
+ printf STDERR ("query from %s:%s: %s\n",
+ $clientaddr,$port,$inbuf) if $debug;
+ my($resp) = ''; my($timestamp);
+ if (exists($src{$src_ip})) {
+ for my $e (@{$src{$src_ip}}) {
+ $timestamp = $e->{t};
+ if ($resp eq '') { $resp = $e->{d} }
+ elsif ($e->{d} eq $resp) {}
+ else { # keep the longest common string head
+ my($j); my($resp_l) = length($resp);
+ for ($j=0; $j<$resp_l; $j++)
+ { last if substr($e->{d},$j,1) ne substr($resp,$j,1) }
+ if ($j < $resp_l) {
+# printf STDERR ("TRUNCATED to %d: %s %s => /%s/\n",
+# $j, $resp, $e->{d}, substr($resp,0,$j));
+ $resp = substr($resp,0,$j);
+ }
}
+ last;
}
- last;
- }
- }
- $resp = $src_ip.' '.$nonce.' '.$resp;
- printf STDERR ("response to %s: %s\n", $clientaddr,$resp) if $debug;
- defined(send(S, $resp."\015\012", 0, $paddr)) or die "send: $!";
+ }
+ $resp = $src_ip.' '.$nonce.' '.$resp;
+ printf STDERR ("response to %s:%s: %s\n",
+ $clientaddr,$port,$resp) if $debug;
+ defined(send(S, $resp."\015\012", 0, $paddr)) or die "send: $!";
+ }
}
}
if (vec($rout,$fn_stdin,1)) {
my($line); my($nbytes) = sysread(STDIN,$line,1024);
defined $nbytes or die "Read: $!";
+ last if $nbytes == 0; # eof
chomp($line); $cnt_since_cleanup++;
$line =~ /^(\d+\.\d+\.\d+\.\d+):(\d+)[ -]*(.*)
\ ->\ (\d+\.\d+\.\d+\.\d+):(\d+)\s*(.*)$/x or next;
@@ -135,31 +158,34 @@ EOD
($1,$2,$3,$4,$5,$6);
my($descr) = "$src_t, $src_d";
if (!exists($src{$src_ip})) {
-# printf("first: %s %s %.70s\n", $src_ip, $src_port, $descr);
+ printf STDERR ("first: %s %s %.70s\n",
+ $src_ip, $src_port, $descr) if $debug >= 2;
$src{$src_ip} = [ { t=>$now, p=>$src_port, c=>1, d=>$descr } ]
} else {
my($found) = 0;
for my $e (@{$src{$src_ip}}) {
if ($e->{d} eq $descr) {
$e->{c}++; $e->{p} = '*'; $e->{t} = $now, $found = 1;
-# printf("deja-vu: %s %d, cnt=%d %.70s\n",
-# $src_ip,$src_port,$e->{c},$descr);
+ printf STDERR ("deja-vu: %s %d, cnt=%d %.70s\n",
+ $src_ip,$src_port,$e->{c},$descr) if $debug >= 2;
last;
}
}
if (!$found) {
push(@{$src{$src_ip}}, { p=>$src_port, c=>1, d=>$descr });
-# printf("new: %s %d %.70s\n", $src_ip,$src_port,$descr);
+ printf STDERR ("new: %s %d %.70s\n",
+ $src_ip,$src_port,$descr) if $debug >= 2;
}
}
if ($cnt_since_cleanup > 50) {
for my $ip (keys %src) {
my(@kept) = grep { $_->{t} + $retention_time >= $now } @{$src{$ip}};
if (!@kept) {
-# printf("EXPIRED: %s, age = %d s\n", $ip, $now - $src{$ip}[0]{t});
+ printf STDERR ("EXPIRED: %s, age = %d s\n",
+ $ip, $now - $src{$ip}[0]{t}) if $debug >= 2;
delete $src{$ip};
} elsif (@kept != @{$src{$ip}}) {
-# printf("SHRUNK: %s\n", $ip);
+ printf STDERR ("SHRUNK: %s\n", $ip) if $debug >= 2;
@{$src{$ip}} = @kept;
}
}
@@ -167,6 +193,8 @@ EOD
}
}
}
+ print STDERR "normal termination\n" if $debug;
+ exit 0;
# Return untainted copy of a string (argument can be a string or a string ref)
sub untaint($) {
diff --git a/test-messages/sample.tar.gz.compl b/test-messages/sample.tar.gz.compl
index 10f105d6a45e10d982af97229532e65e68494642..1c6269f74afe8d8f21266724cfbe619581b3fcdb
GIT binary patch
literal 14391
zc$|H<Q*<TJ)8-u;C#T~R+qP}1<Bn~+W81dXv6GH%qhs6F{MW2mv*!Eeedl6dJoVJx
z^;=bSv3FdapNq$)J3l|7a}(o|GUuXlW4?co6a&;bEx%EbST$#$tRP3rdm`%UUf+A(
zL7Tp!vDJ2Jr0;zYlHqVsOqhXF7FjXK23eI8KQ&5{&nlYz>u9_(t95oJU}`-}QLbn*
z5oHXZg&^|B_PLHs3m_zIc>(2+TxVQQujy2*kLgF2TV<xX?q~Ik3r&!TUSL22Mo;A%
zvhd|hN*u6CA$1opu at +%63<q>D8|h?pWU>@ia(yT)I_0`zP?dl2U8QYgY;AZdGJ{qu
zbyg}(I8i{4 at m7dVELh^1C4qS)2L{wH8ZOs0p}A5yXkqsXqLV8<&B7C6dIu5MM1)y@
zl1-WaIx0d4U(&HBj8qh~0-7*)B#e@~p%o_AaH5#1vapJ~(ull at 8n_WE-oz-TRORq_
zAhma*gB6AKnvm#)DEb2#4AtyT{xv9yNt_1iN*PgPPKAC+*=C~DO6UwVQs$BlV4x}=
zfWQt7h0!!z)o_fRZ!ccxMI)*X6-*93CfUHZp_ZVgEDW!t2+>ypMtlMl$m%eF7i^nb
zrNe3N1^A8}w?JT2iCB%cs#HrR3c6F%X~@sVi-9#=<CDs8S43_*2I1l1&B{HPXmR2z
z`q%BM;q?J~#lZ{PVc66k*$}B_g-9r7{T);I)Wx)b9*BW at fE*zeJ;kOYXouY#>hVYm
zgx-YoinCHHaCz~J4rqrD418!Z_BJ*GTU<%mU)>N`H*!-BhhXSR&-l$oE6=A)M6ms{
z_YjdLMz&cLx)sbwVsT7JqR;`zc12miwLA3kVG2(KVuO>m at l+Q7TBGy~JG`kP;QN at E
zW3Wko0`17eDPU|#fJu@^rin6Rrp2PwoIHlgg;W{h6k<S_1u{TY`N)Vh!V;Wo`|ef3
z#S--8ET+|_5|2@#?P(WNvML`ww(^5m8dd3`e=j+=ebg(#rFj;JCAqi7;;y&(Dt~cc
z=rq}T{XKZ`w3;$Si|eQ{$AP^|(RJnkvM9!|KYp5v7e9n{CLe7~^9Hf!5mD<l-Se(H
zcm6FIGP2z6NmCI)Ftuboy*{GA<lbdan(FQb>N=xfzY?3Eqd=~iTlI&bFr$Wr8 at PIc
zMs{t<6fs!~>1wP~R&HKeq}7i#VBr_nejEB3&<7S=o&(JhJBE?eseNL3t3SW`fTUW-
z+|Yuc_I at M9w%f7;J<!tP`AC6m?Ah1_m!X(lB&I3$*->GYC~4j*9%UI at FX{@_3=Kbe
zO7e-JG=P`J>998rxggXjU}(pCZz?%ZbZ{b)dcdH1%m&Ul at +k*1gt?9)Qb=VN3+P&_
z5~<5NwT^siG&kL+U*Zw|Li2B_E4_uu-Jam at 9^|(b#nz}psM5*{zHPoH>d!Q>VVBdE
zk)>U=eRzg-5 at Nn8W$r4=V4{Ig*=g&#dOMK|k?UvXkmBx>LYOgUGaq#&@C9REyGljK
z?HWt6RL#^VE$RfrKA?d(tldcX1Jy02V~5Wwm1P=+{HEWGtM6jfei_M;c1dC}2njN(
zncRG~PY%wBN>?-!CQZtU8$WgtF=KRv@|W^xn_aRz&M#71FA_p`E!*&x#ix9&r8%eL
z!d-c?k_Sxu5(4bW$lReHtRu!g<RZ{BvxZR4X8_2<AB(B7Abce;9f79<w`MZZV{G9j
zlczexjdK2>pw|~j-zg1qS at G_RX5BWbwkBqDbWgSrH1@D?$h?C3K4AvpV0g!X>DDc>
zEJ^*@W#TIu6lz0GZ_c<P7EuDvc5fK*`@4<`2gua05=O&4)@I(n4p78Wq?FpR3Dljq
z30q9aSZ9TL_A4@}ZrKKfW9bIXAX?WI*`H0otb7-k3}y~Hv+e|AI4({`b04+^EeV^_
zCIxEfT(vDGgb*c^hLkt{bGf@=SXHne?zHxwNuf*V8Krd5Sy`0F2y_xr6l+NZGF<g&
zcfk=f5_z+1l?jueqkdy&b00iq1Lc5`7pN&W4%JbEBF)+!u}~Rc%#3XkYe8`3nK}Cv
zYj9BqVk7UU+!m9E+M?T~{NUM-J_R^+j at IHO601$&HM43!Xi;GqvGWm*E{8Fn7g-Fu
zxvn->AMgQ4js8y at 2_R>-80zmo)ykwt(`Xx%kHO^PpunY}#B8e_9f)~SI>|`WS!>sH
zS1nh0F(r{lrdcD6;p~(ZD$FhsjI8Dwv?*%|`_7^uhU5w2l?aRB!{UM%U>HR))5^TC
z;isjtY8ASC!M2rP9~&ZbF7&5N(i(K at 0J#ki?&_`nSfgIwjF7R7)r#($<ykop6I)kf
zQ)}={xY*^w!Pmp)0Xf5(TQBtI)#0R_2-7~ItOWoZyd4)3f;J!AlIjq2?agWtQnp$=
zem`^C1vBCyKwgu|W1(iQYGAZT^)Isqg5}b|Lw2L5V`X{FG$eu=m8njcO}K<f*v#8B
zLp82R=LprK3~pc)<#f|4*AS`ej`uYG)t0RhIh<OoVkC}uspo63Q^Z~a)WtfL#HvCv
z8K0 at 3qk|huNRdA%NJd-eCQ(M(SJH_PqI(MHCRU~@j|10p#e*^1+Ct;F at Pd*2Q^+x6
zBfZ&j-9LO+sK)|ZEU)`((L|+|X|<HlyQ0?BZO}&{+%~=0O3BQ0qVa#EII_15?E}QM
zx9R^O509fB%W?6rMf394F8xy;a1UumlXcbgF^r~QeVo0lzq=3Ba?KI1Iyy0+XUT2S
z_Z-x~_zl~TRnz}(Z<oc0(UNycweUI^q`dNwsMdg3dw*P8kqCX|nX8G54fR{MI5 at GE
zcD!sT|ChKmatMf@!F9Hxb9t>}UWlK+Vjtca8m*K?J;Vu!AT9YI5`w_fKQ6^&6COx;
z- at vg>q86Ny?#!C4Ui49`o4akLj}xZ8GCMrp_b*;~V>4>6(Ls}LX8EJm2?DxwY+a<l
zwPN!IQ-e$vAGN9&dbdGOdU8XDG;9qeu4B*WtpPrJl at U5(?ok0LNf<=`SB4HRJApk-
zZ4%LFa&VcX+Bfo;4AjTM1V=u;Umdh{RBLZiRgImV^!#K}p-0*lz?UaREhZDHFQvpH
zOnWa^P50Ln{ZWYaUqcIEb|mO{fkeAxwBY&WX>oj0#eBcawb*uo9n-FY3HIDmvjxWh
z-%TKaAvIe>c?gBLt#Oi9O<b4=V}jHUt!qHR4xF;_;gS&+VuVMQTWIugqOW6=yy{VU
zjINgEvT_m*kniV3;EBhPy7%Qm7C?YVPle>-CuV|SEm40s_Fm&{nFjcwf(C6SM}0pg
zk6Lb?6s`jkcN<Op?iOSFI3YO_u57Y4Tbg1Z*G(j;k0Lot-{fx7#?$!;f1b@`Iyj3$
zILmR`-)qYtN-Vh-N`rhjJ1k;(fN%Vg;wl1$xjVBZRFxZDuL@~S;-y)>yQAK{WwPCo
ziZLdKKv8UffZIwwhHZ{FJV=_3yU#&_V8#8Sz8q0U`#hta6Kg?tgVo2+`PQ%aMBex+
z-QoBzUB6T6KE^J_uY2xX&Iu1q<Z*P<E7BwMdCG<mc?mQ|Jc7f67^KP&48_>V%)){V
z6^Ld7yqg|jxbe!So{AB2QzVls7z}Dx;{p}7L{>PIDstzrq`6dW1XIyk_658 at XeC*Z
z&7A_}``JtKblv$h%DRj(ny2=>1aDg+__*!Ge~q*2jxL-VOato<$bXF;7;GZCDY(0l
zpDN7h4E-xyr)FVa7rd^nBukT8ze5+-0WsI0U<_$;phv2iL2yv6E{r{F^g%zl(L?c+
z88}CbnNK74Lq#@G=X%(x<P=>u63Fru%og~DkhP0~P at o)d=0xS5;TAT*!6!|@i0j~V
zMAT8_xtczdxJ9E0*iJTo$OEbOFp6=IYB<#P;`3pDte=ne&}Fua2i!)H4Ou8ln+v^)
zC;B^&7zI2=k%d?!#9Q^gg}hi$fBBn$H9A6?6;{lG@;^idap=5+?))tv96%h%(Z7h~
z<iJLP5uukEp>_)qMXDk8v5W#i;;j6G at s|n_2KjDOCEXCsLswmaH28ZWAWb^hA{>pW
z12y?2{sEmfXlsY$JqEG7*xhWoMpBY<eEOVeu&@DT8v1l)<%5NdOsf9XzQ*AI=7n&U
ztUxFFlL69Wfo!c!4PIX8=`HKuyTd$xal at +bMRMeIWVaZJ1v6+#H|XMc$fq?d9_d)1
z2~+?s-1RFEgu$vVH?xWK4u^x)2PpoS<4>Zm1s_U&RD#Kb<EV#WcmG=&&?g7~e**u9
zAK0vf^8ZW!-xA8YK<@u~?0-r4Wf2_T|GU)xl)&l`;OBphLMT&*DRcb)03H2*fa3nQ
zpw0z01fog*jcHuqgCsBTUz0|MG`#$moa3nEnuv$S7I*&*DP3}0!PtL8TCXG!O7Y*c
ztq##tkZ3-D3yBPnW^<Gi{pZe?eg_?B5{nHDPhG+crWt`J^9L5C+0e9!d$xpTqE+-_
zf|y(k671U_i~%0fQgj&758G=F5q^}Ubg~ue!4k<<8sJL0Q%GEv1W|5O9RLY_f^fSX
zMiGn(hXI}js1Fo@>T at Bg+oc2>3@)eE8I#wBLcl1`qHQ80 at U``(R43W9ymp!Kt at +}?
zkBfR<wyS!6N?!f~gY9GGgY+3*<*9K2q9upJ at ivb9dz8&GL^6WduobJmm9ivSB5t&E
z#LwJkhhxEK0imw!QpH;@63rEmF&GsM)K?>|8WAlOXk|1G+=4EIng-NmYX@#|(E-A3
zqFMu<$)tE-3`A0{nld?NgXT(*h4 at Zq27{*YW01QjCATkTI2`^`*u^|}#>^OmM(hx9
z?p(teUHz`3V7J&HTg87w2X#Aq{w`tr?5jOfeeP0yuA0Y_W4g<1o}_k_0~zR<Fy)Sm
zgoFVh(>c*1D5<%*<O$OhYrPPAKG-m>8&~l2j~?-cfq2(VdadsT_-eSWRda6MC(W4}
z&=&;$&@b(Ws~PWqs54}t$wjxN!CPC8L5L9kSwAznmE3k;E#6fb$3HfaMgP8|h*wFR
zvVrcJ`7kFO8$3=Y5KKjZ59pMRrzELh8KJ8py2Uu%9g9U^+;?<5{W7-6pSK-FDdZEI
zPfiz(al8*n&56oqUg*g&`ke%>;G>w3CM>1 at 86fZh`4n3isf!JfWTrj<PwR~YG3c<+
z2Izp24S5Wv<hgCB6Qf<C7M&G-xKCm5I{K*8a!6T_1e4nc?CzY8$79%bQ{tH2_rT<T
zFmBrX3oX8(ouj}BftU^Xg7*gu)3PKrXKvd8453hv3SCIh0(ObkU$|=iVMgf_Az!^h
zdj|o`;J9unSY*vwy~9uGDUR0#VU`wtORSZ-j}Nzy*PHy>?b2 at aD&p!gDDR|pspn)X
z=Y&8dmB6uG;pCO7a|OBR-~Ht?4u6`I9|BGWUa+N^AW)65IjzRs#@!S>zco)8Q|aUw
zc%nJ3eiF}5*>pTTiz at OPyxWp-0c69HW8P=l at 5;JeK>w5RUX{_)*l)hMaq*IC9%c}N
zI(122A%q at +&5U&nWwIfDtcsec$q6rU#N#MYf`D_rP1vVY(ZDuy7l1H+bb6nQ&~IX*
zz=Wbd+*>5s<GTb9RMv4sI^zy<+%euXz*%w#zk2*}73Y`wtpNWBhdd86 at qoQJoe+0Q
z5-Z0*O!B24>?y{b!NQyso6pFRr%;1**Hw(~J6DjT*z!m=daQ|AbRlHr`T2bRaw^Lt
z$2 at ZoDX^4!Bgto0SLiRT#)afbI7==^dzq}^XhKya*Cufcl9mxhx)ZutHB`!iVeZIm
z&c at iEA$Nq#igR$ZlCxxLu9$-1R12i^5pc)Fo2!Gk*T=&mc%f?v2v{VFQa0B*=&H1f
zUTJuVC~bCE%@EE`lb|_gtXd0ZdX_)?fP7c`{_<2z<rF8)%uw3$(_JV^EKO&PFpce;
z at fI;2=(H1*JEuU`i#2UkR1?|$M)zoxg(IQ3xEf1ipiYRn?5U6{A6P1#HfCkS2`lXx
zYE)ij&sel%0u3p8lf<4#Q$TnSLzEdJOCd3+)9g4#hAu&C43C3gbQVnM&_1N%tZ}PE
z5d=Vf!$2gY)bQaOakQKmsfkrwgiRl)^-QU2VAR-EkVIuZAA%EUf<RGtHXA##B at uBP
zx4L#6Ob8S;Knkq{L17vVLn?fnQ20u-o65iioH&69cpr;iqAeUh&5w;<9V^`Ry5DWA
zl&c>mW!ey5NGNIi1_2>Ui=9q4cY_U{!_w{7w!V0j){XGTV-Hc^HOG&g#$Jv=U3 at Md
z$TeLrO70|QQ<;zwu;awQe3q||aps8j{3Nk8NxwaVdKf<X06FOYtHP>1KIL8vvXSs3
zo9a^AYNls%j{^=?0jK1V2!_^S`gebmSc`G{B>FLTgD$hfguVcW+%Nr#ba}PKZWV0#
zR<4e_MD6ew;=35K3~pB#3c+mqCE#w5Fedq3m)MTOczlxTNJnw0;xF%&e!jBGAxZcK
zqKLylzEmQ)pvhL7%5n)LtWK5gZ&K%&UO6s6;Q=67K|tV&$o7*#gM&qJ!uyY;5Pq*r
z*c!hy1HMnHH4cZem)=PH6v>aUky+BZ9rZ{|EehK7c{I<^2u%|!D=|LbgwTkKQCW?o
zMR&owJFU3#ZhWf*5ojD&^<r&;2$Ufci{?;7IU6{|v;%pQ52<>k-R63h8ByF%^NIV3
z$6pM_qq+;}EAi4Iq^5((c{6HGUT_I6wL*5v>L|x7 at oa-Kwob}n^A$#FP>aE#gC?|}
zGI}#Ne{i{h%U`zxnM|e0M%s*qltg5F(CtKnIq at +kOd`-CXQT-s!n<UKqCgU+eTV|)
zC?!h-51B;&mGr=I(%bf}-s7bk(gmypoC^*KBXKk88BjvBut^=-VK(v4G$Zg^B)MUR
z-3)DE#*1<&zo8}^8{pwF;D at xeV|a0GwaM~}b-4HvASAdH3uxl2#STd*T)Inokf~A#
zrz!)O9nkI=l(zx%)w(T$A%t<_UM7iiSaM~)oUW*_1XbRhfGT<UmiR7zX{5&OEZT17
z(&Tm!267_T0*o=au)VFPKRFd*h2ud_*U>w~<;=s7(HvlPjsw4u2t9v^h^rd`BFk)h
z4MrOXvFc}d`bOa<lZm1qYFkhB-YfhBxmYE~=qUs)p&gHM3=X<+2uvp=so;Eq8-fy^
zZx<BA at 3!%r#B__tqf{?JX;=P}@4ZV-JFIqd2>lg?N0WhN!hlFYO;7p?xO!*lqWBxy
z$~(ykfgkNH$q&jh^_ME4tEG3L>!XP at lpf4bTK$p-%KL$<zz{goNdgD(h-5x!XS&M%
z4~H{Z&xcWI at Fa3LBA?g<7~vsQ5US-Yk;u*`RWQWLl9Cz=FmJ`9Ler(=pJ81|aKdfT
zZuN^v#C?I%Hvu!}PW;5#Q)@6tU9DZq!8B9S;>&LmPhXy}9y-<ULiqf{PVpFB4Y^`R
z#Tafc^JV=ZDgwJ?b0%e-)pXzV0>k*eV at 9@F>biHnF2Q=#<vXSDn~DneMXbMGzcDiS
zPbQ9;u_=(6*cvyA7^PYbv^wL@!_X2MA3m}DUUZ0?#4w!~ThI~w*I6D3CPSfsz}RE>
zoZIZH(~;=H;;brR`iPI03^|9Snbqx<D=?rI1I^BbL>aKqsfY$394NE4PYZXT%E;;W
zOED(T$8-QsQ>0||DIAQHC^CA(!=NC=fpL#aqSh!)PH9MUDGeVDCU!C|AE(Skr_I2{
z{MU(wH|g9KxH))ptk12{#)|aNAwG(1_6r43*TSf#9M9T-{I;q at 0tM5o2%Ec5roLgd
z%N*Na8b=+cx?^w9A6t}oWE^d=ai@^GKY$y30e_n<l6{Y?z$mexOv)s%9tnmyH)1HO
zKky*9kxJmq7^an~4ycV)cZD+?#5KoSuss0704tiD*G%fc7>IRQl(r!JldmFd2-|#H
z)ggi+p$!6GJ;*19T8Tg8i+!Ij42>dv-ITcn6C-t1FrvwW#15tEVBbbpx0l3arE8%p
zo0k!Gil5y~iut;faw6(D9NQ%$g4xS at E32@HFU{aOkUEe2$1=a at A5V8hgi`OuHYegx
zC>;a%!%=rsi6Vz<WkVF^D~+SRhsaF42aK={dfh-Jr(g*glz|;1H%Mr_HX#5c#O$+A
zShg~lVcTziR49Y0xyt8!71Z^f1uLE at aPaaatN(mk;p&`=O?7JjNKRkCnTj@?a>(Z;
zjMc*CK?CGxkbybIlrRAQB?OU|!&<T5LhQ4AsC$qE^JtbO=Lr;X<BwF{TD6k;TwWj=
zIi#Czk%Dve9EHu(4sg6=w0xJnQWL}c`?O2%Cxp+%{ICmd4tg at -H8TrIonhLxQC_(g
zjk&S69(!0<L5Sp|b`bF2j1WY3iCuV$punwA*JJ0oaudBw6^B8FZF$42p3;tf08bv-
zTA_yk3}*K24~ZF>9Fa2d90R4R`yY$2A=<pyt>&M66S3X1aeY3WKMO3pll9vlMS1Oi
z>QXV$oDA;Hp6x0G1rR8Fdf6hP)O`o4|K%-GYew*azeW_(PF??tpM}QcbFwZ7HjY{s
z<SCk?>4lrFZ*e1-&j}R?8doS;+(UpU&+QHQdn$zZJAE^d`-`_(X3cnyFqJS!cN6Sr
z_NV1KK`)igb^($VSmvy~x9bM at 9%Zk4kje^ekH!EbP16pC8cPOWu{+F$Al)#6ga41^
z*^TRsTGat=unCfYBN4=R2sV!;&9<R!2C|o3S>A_%!YY_%3EArn_HtqodlV_Yf5sLA
zi&VJKkudg(lt`2oc{hM4QOy-yc3*eV{&zaL&6p7PiJJC#dB260KOdYABj=sqrlj;Q
zfUkss;2~?q#26WrhFMszGUX$l2^fS;r-zKBP~x{y9X}_4TadPQ?C6y_xt8A>C}_4`
z5b?mjeB?2*0Di*tHok6|MVfkb6nHR2OAevcjEEDrB4q-jR!M4b?f!43op)oph4a?&
z3}2x3zOCuf&R&&ZWou49X?Oi}|0RyZqt`<vO(FEF<(5|v8r<=H!lgxW!>x(K7doJw
zV8zw#<XTXu<>K&aQV}J~=+z at VX<FC*58GXLZ&ZXLd81%mpn*Bt`mx)eoj%>`Q$?!@
z`IY7K*84Zp%edg>m+1#O_o(gszDPevt+9Q0fmt at as^4!iA4yanak0j}Tf~>6En`&W
z<!PT)a8^g9bMGLUQjd<&QYerVAO1|+#l}12!myqA1|x^*^!hTS0$ZMYRKwque+a5)
z3#4Jx<GQgqrkSNE7sL&yYuOGX%0uOgx+EfJZFB|NAI7l^0DlI;5g(mA9&d`!Z|SyV
z><TKg6Gk~NlbiE_?HE=;GVBaC;JcwX{9i_w22j5_wPWaB8HtPgTYus+wd-taRy*O2
zv%S>DIk^;4_X9uMLrsc}#RlQ?v+?qo3(rG)=5H+`uVsbC5oDC>A~t#ciB{zO^w&Z8
zR)MzdyTV=0y)Zq8sino{Qk`EyAzxG3dRNh3B6QKqF=0<w at 1I-u`-jtqu(Dw<lGpr9
zSHun5@{Vf9&Os3<ZcMq2+r>oA`_D7K(7hRMeXySoH#x_5L}!N5$RALNgef#Iv9D~r
zG#vELd3S}yaa5>sv!EA*lQ?Hh-h)o~-e>nyeX-QGS>4S(SnlwlXs01Q2Fl`4#Ew)+
z*&b{=c7>^J#YWfZz-Oi*&d}{prkV=<Rk}`H%!LCS&Dh=1&05&4Gutw_ at 6YrSQ1*=X
z6JLJWj}-D&faV$7XTJ31H(-y`?d}EkJ4Jc9p?pxU&hB2bJO8qs+qzx+e$X=MD&a`H
zSPgCL(uQcAn#`EUE#V}t13d(t$?;PtyKw3@(Jpk*wq|2?{C$_?!9vK}nXgNnrght&
zZ|wAr<HurD!Q~nDH!&TXP8?*Pe*yRKySf~yqgpBLG&<jRPon2Raq{vxTAd at z`m+6-
zl0iGeyouFN_G8y}VlByF&EBF1`>HUk7Qf)Lb9QBB>ZTy(mv7M*ON)dE<JQiv<sLvc
zi9XIug2=`z#aM>7oxJl(X4LH(`Hc5_@`l8S8PHeM*B;v1ml~f6o6rgAGLZH`&H2yH
z;}wK8KDrs#K!fY|F|d0kO@)yw<!h|l``dp{SH=^wXOoPS=VdRI_3iJAb?N6Ap(5DB
z*3`E3&pTm{^}E1cQbo!4H`PDr*A#g;u(nqedF?M0JBzPTM}X~;Xct9`9?qAipgDG{
zv~JJX{Tah2>llI=L7+W%2R$oMq+6-;t-xF1VB_x`KJ5u&JChmT?TsDwomq|%Ejx!-
z*gr7lx&k&^&wIsN6Sr3|e;~>RIFI(!kv at x_OeQY13180X#P3p3|Cl*UUuAsp?_==!
zW+tHG9)6}$uO5uvuRO-+vRkb3eaLi?)??*-ijfylZFi7*B*_iuDSsWC4Zm#+Y;%8p
zb-zZj2RVD~3l~PyXYCsNnha9Gv5nd))tjD|Mf8Af`;(ZLziGVXJv1U|kyFm(KY&BN
z(5b<&Odt8(bBZk>jM~Nd3V#K+1uB*7%2HAAKM+XFv|v*}{28D9+ncfvVz>7ES!VG$
z^jo%}m4t6YwqhoEf91DjDEmv#yIt@$UC-9wFOHvIw~eU3AhT_~LhN7v_%B%Gd=llI
zD^BT*g95-VYev*sotdMz1rk^Yy;{)5LWR6ko<1jSy8L`JrZb-i%f5Fa3sq;EAU2%R
zP@?5w?+;%uGUk8OG+Ks4bIjCaw`be5+s~9D7w^zb^Fl4-A}(*>UH#E(jQWi at 8T#Ds
zf>U`%kgfF0>{(=&;|lXsl^zUhsqGaB%*$^C&61C;LhryULpCLDw1AO9FM<N(2Bl+b
zyYUfa`{y at e&)mEA-$djfysx|8TCH!D;Ud7`oj9$3>;2?Qh-cbehpIfYH*;lmc6N7$
zL?lS?J$j-^+XC;_x1pSq#^-wE_j>#_MLiBN5=fAZ>@ljZ`{-iRfA<42?QC@`KzxjI
zNULcRb0^-kpSl#yHdj`*1yA-bnkoa{)GCm^8|Fb?g`WpMMJ`rv)9axFjgwy-IX2hm
z{#{k4tpg%W8&eav#(~WUGc78*?AoE(lX;{@9(`xOKJA*vQ?Yt_+ZN#nXR-LRTK at v#
zQz^Yh=pp3jXx1j|s%m&9TlHq_LiSR%2*kVV^gjQ0^o?U at A;j*Th_Ji1;F&=HEa$|%
zvdhMc>~Tg%Au at e_GB20y*mNTJFTbmeBBA1PDcP?PJ0`b*3Wli0#*#c83Ew>^FO_R+
zk;5Gq9;jcGxck0;F0(1~)b;Z2bp7JIa^W3z4fy`D&M0?n>8(5ApcMr at U*$=?g3sL!
zx57pqd#$rnDfSvz!<tdfIS(`AIe~INH^R?CpP8gHz9?J^^gD97#9oRS>*{p2Y#HAv
zn8=3;TO>2R8`xQQ`7jJO+=Pb|XPmmGWtA2!ynUj&x3I{ucEftYMn~GeB=yQrr+*&&
z5+vmF1(<%;VE44+5=3g`>(mz#`E?88f0Bi}KL_3`8dEA}qx$*J0(k~Pr^U7+f{j)b
zVmapp(tpAnAf8iXt#Q~$A|+YJZTk-^SS}`aT=Y7VSVH4#l_z=H&foKT3M*)d29jNo
zomD4Eb>#jun^$2WL0h$|RRS<IA&U+cE_)Tv_p8VTX8a153jr7I6JM5HQzt<463HA_
zPG%;_GO<d5!w4X&@db+h0cA>wmK8Nol`2pDDNfPbA?DbMR#TEYa!Ot=Awwi(AoHvm
zUk{{E#^skMh`*!{%+_|4^3f;Wn*;D-1_f2OEaw*DLm9DTEl--j$KY|O?MpHH=L&<;
z5jOV*JFI4%_W2-{hQavwgw$fB%Bz&oy3 at h4nZEMFz-O&aV-_sfP=XEvaOf8-*mdY2
z$zX^KV)u{~j=^;g8RtoY_3n3e`(WCLo7Sfd?!j*BUPdxd5fHDw3-Xc)K1VS=At6c$
z`}lq{wtc7Rnd78T<Od-`<qn!V19|IU6ovM8VMEI>CTv_D*|wQ+CDI?}G9B<hDA6}h
zAi=GfM&rS5&5l>U4QX=&c0$vBvb=yz-4`5t7!7RLxe@=Uf2?&pantDmHF2FsYs9y<
zJ)R=V3EPnWS+K*2-Jl_9N9DA;fW6OPWW)_|85Y*Dj4YEhLGZXcDWmqIVpHHYiyh+W
z3gTz<x+l88%G-lDk6_lVcR{->_LqtCQECj9!&4C}HUsH1JC=z{YEZo1?Qy;!CC{+f
zauPhr9f^%i;zm+(TouautFIZHq-F^LHTB;*H>p*MH!hxpzmP9d<Bh%W7HcH<P&<8-
ztceX!E85913iIqKOByL1m9U)UnU%G at HBdT{9hIkp$W}ii)XHJ&%NfMWo4Zu0dz)Lf
z0 at -FLxBT0Yqb7)akivi5xzWxXUE|Jg!E(<mu{RKS_J0exDx-isfaCFk52881f6tPG
zRgKAmEzflsg0X|-<6#{}p}=Gz{3qe!allzf!OEb(>hQr at wIvtAeZl@g?v$d<gM%TV
z_+Larh+WK`3RSDX?#bXI9uXdYN{WT-C31Gf5j3e7{JWpE^nbn@;6iIGO4gv0ou%f5
zTz~?nHxWWv?lWUw_iCK7g1LRpSzZS<ExQ-pUh9;%LLtQ0IhJ9(rQvIHGv%e6Kn2`Z
zB)jJ<t?IljAueVG)PrJlgZsXuw8$ICJq_;f22<<1;UP{3fIi^?O(^dD>>DuWkSE|M
zC73()-#~Z!zyw5qr8r=Sl9uk at js9wc5BEA*N5xabJ{4GG3RF{n>$)a)l-a%q^pXUr
zAzY;BGYcqYG`1&pUwz(zNUv2bXvpkRGRxB;1S-rNnImW_x*@<YywXoO1EvxBcO(Li
zkf8&H_H7I%Q#W|@W36Xsm5L?&@edqWj#R8h=TiFL<~R5OFE`sgX>)FJyaEfEIcWr5
z;C%{18-6G6TP&JQ4p~}ke;;b8X=^L89<TeN2RKc)*uFDGL at td4xiuhJL-4$&P545v
z(#T&beqYDx15hEua3u7{KM10-9l0LYc3rwyQ4}ArA&6rsAd3iVh6lbce^uGmo!Dwj
zfZf~+*SC4)QovEpmJ%w-w>bV|Yh6I4s4B6PJfAJVxVvG)o7 at q&KPvsPWPtaH8oSG2
z5agIRZuqZ)yv*xDvgl0?*zv5imNd<+LC9#?Z>|`JD<o)rsf$-2i^`hn2t%Mor&g?y
zqN*2RRj*b}bADRj_R*0%W({pWzr69ZBv)6ly7FyBV_i|6dTw?56{+ZyB2tj3o+I<|
z1t%9ta2bqiNUeP))}=vF)$?ANQBeNT30P5a)LRm{C1acqyy<%UB0P~4mX~j$hXZkz
zq;qYt7<c`?<vwy)-G+}`ctn%DLGar9ajT#svF at S$9AIYZ?{KhhPkByOL>W#0J~R}9
zdW)Nu2dH5+L?^UcL at r>A#_IZ4=*+tX;-(D9uEDf?rYtj?LyrA>UcVuThhx>Iu)wIQ
z5sdn^DKtfD%(D{s!~n6dONd3wKJZ%*D^>w`wZhM_<-ie=kr&KhH|)QzIG9FWC$A9H
zK_^K6QJw!jDgMy2e>QC2ef+t*+sK&E-Bx?6$a0mGk}@dn!}T)rk$yp0+i8}l4iE-7
zgacZG&%!VCrR%i_k)XtDPSJlUfw5Eb^k}^J2ME$@Yy|7B%u6%8C#G6b=3YeUEE3Dd
z$Q7In9v%(%aj+ZSRy+N6-MeSmm|*Qv>e~~@0-4w6Q;d9`9 at kTvV-?eZ4UTPYP*`H)
z&pCCJd7|-+(KtvgzB@|kJg)`sgy5{&lST!7ib+1Kp7P(P4PtIA`tiU2R1p^OLXqwE
zZQs7|h1J|M(W0wg%(Bh<`((T}bX5|n6(5o%f9oM at p}9hRL0=CamQ8zwDg~}U?zzIh
z8gVrD8T{sx^}-;MW&!*yYu8npcLl5s`J!sK%+m~_-fZ|6 at xs^oxw at HrKa@icg>93x
zHy!ohOpJrWkugS2!JT~={`kn<75)|@Nmtj|God}ipKVg1`BaX1QLMC-g6Jx0sRohJ
zR#Q)ODMpgfR_Un;s2n6m!PQt25{f%Jd-}2d&|4Cz?cyn?o+2_d@&LnkLDgadFc4pA
z7U7Y$UYmG4e6TFZ9aqo~f<=we&4O{=?J&la42;0hf^(}+nb4;-ci!rL3Eb4M+bik(
zbC|>AKCXwF&5Jf{2NBE|bD#EU^$Gx$IEt)7zg#3=BWc@|ypLepcD7*sN+e-z^oSt6
zeePsf7d#Guf9UQZV=&qvCHe=wJf=n4N-kX2{U)0N&U%*z6lc!=puZgS6!bHl`t;S4
zR9tZ*usOfSqV|qRz*PP>i(;jsx4?+B-q{}39sdFXk^79-FX at NFckz9v3bxZP-A)t!
zTJ<lK7k=0YwzE^V^=5r1j=`**TJ{}L!TRfe_8PXT2R*X=$4>NKz>*wX4C__Is)>I8
zx!Xywa45sd5+}ylga5YTQBWqxmB79h!|j}B+mRSD??Z5)VL9pUG>Xb4rh8~;J4cbS
zrj21~CLC at n@-9+Qv}9u@@u9fy1zzA!h;luHKf^W8(<FW><S_iVU;W6?3^U7tFP|x6
z>g2~zO5f(Y!fjWQR_3b?Sy~g59nUo3%;$_kj3&iqvM$4qRQ3}LftlZV>!?o)@2usY
zOoeC2UiV_STP0ckFx^UW>K;R^T;)X$WDv0GF1g?~5I$hXpLFHQt<8-(gcXXSO+kbN
zhY`eUN1Ez*F9odv>IO#{e|vIg5;yo?5_|67B|oCO|H?ej$>-9wt8XgP^Gz&$_T9EH
zTqON&0o;9!(TJCIEF+&=Pnb(w$S0(&bhxyQ6j^>{9$hHC_uNp%<44B!)c at q-UApLJ
z<chtum1Mk*jhx;bMpn5$<w at vyV)Rfx7(Wla4-xu^^T;~Sjcv#0H9y;&NTJ|HSi6p!
zFb(v|+>8so*I%coe;#-LmK-z_{04XZeI at Tdgf*}b<gN4~W(D(;<)I-!&$j*hsHlUL
za=`hul!}tnJg+1!a!Uu-A^DnZCjO=bjEEIXIdK*dOss4>AK#xf1Lgv}EeQNzjLJ+3
z642`hX1BV0!4GE6lDU(SzSEl;#{AqOyAoheNCvaDvrE8`=!5n~)fAju5B>_p&1i;2
zjK;=P-n+wumB-Z;FU&tbsXmt at Z#ZxpG#{u_W5rv4nG-EZlc)uwj4&(YszO8$Mtw+<
zK at TVf1K(P0*@*!|IjKP-*s2sF$4o`ER<BZ71c2AHRP_~HW2}p_l~0xh9Q?zPLM==H
zPe(1heInA1aRbCSW-a{V57!Xk4X0=qZsH#w4aE}+;-#7h?I+5FI)I&%X$rC+?GR>T
zL5G=>a0#-Y`OhIgAi{-1DTHnlZo-@<!iD7wA9jG9BdHc<gYNiGhzxcv*a2n^TM1VT
zWQZ?D3Rx at 6#_%5lC>TC0%o)xB$PsP=3;myWwV at w{+17ByNR5#SD-7|(R7HIO^0}NK
zk<ipXY>1UCiGWu{=)_@$W>D5t7tqrR)e7CW>V!@OQ6`S<t{I(tEq8c?(}abt%A0k4
zuDg&e>fc<qp)G7B)t!V?LR4@?S3>#$Noa)?Hnq7D5V5DXbE(@i?y9d&DXd=WXqRIm
z_+_dA=h6gc*lew!7%orA^M$1s<;FQ^LLY-cPzG~UPH^FBQ$c=S0Gl_%gfq;8$n%_m
z<0?RWQ>G0<_iS^a=u`sFZdv)ntSl_?kEu+<0z>93vbu{ZIm;2zdZK#>{e at B$_aJ<s
z4q}jDsBO*tY4^rq<{Iwdu@<YSuyeX^&D>aoH+7V5Grsh)7!)@#i3U8Ln|o=4Svi9#
zJG)(@Tya0?Pkmhd*w~j*im4f(vH0R^-kpfTDV=FSp1CZYnEAIYYHy7xZU4$*>zsHa
zYxyuePOoPna<SVKQ|Oc#V4Xck&}KOk-npQ5xU5%Bf8CQ%iBTz$&?jyo>J%Zgg<kf2
zbpJRfaM(w=%-W5<b`mdwHJqXchhuN at FsWNt6!*ClC&o{R0fitQ<5{249KgDk`F7wI
z07pV;uw|q@%(+d3i+@`{%0p_Y+nk;_!Wk?fKKYmZDqpZYv1_}?9DTZKcQ6q}%)(tZ
zlry_kujCT9y%w0s-zy)s*k;;xn5|d4(kpQ3e#hb{>O<oLBPk$e!R!uSuSZFabNzYw
zbd;07R55)!>cS<@wNnFrJXfL+Xet2<CxS5XvJ-1bU)!DI-<+mauNRDI43H0mR1L~|
zw<JgQ>qNYC(Erh^qh`H+L~17^U at wXCZns6Q=@Dhs at vy$1<AfaRYaZRsk7nl0>p9AK
zfV;ZBP}$PeHpb+63#B49C9FU?&|z9eTgR~-8|`yyd2~evo~1&UczcxZ!&rIA^T!aU
zVP4E*&Q5yq_$Sjyl^ZUDzOnNw12x2x!{iahtN0eon|L!_2q&<0MBW96oEsixrd%`p
zk9NZw|Fu38C5T~{$e#V?#cqgwsktmE-`1X#hnR70O2K`D(Dq4J3qi+QCOUKL#N%>`
zPa2sv&iDtBBbDA=f*8E{9-JbzA^5G-jwM)hBQFzV+&<O=ELfChFJUiTcl%#RA;H*q
z7@=M~J;Bi-2zT?be%}rLf3Di~1-rIE&^7sF;NG8VN at eN`s4pzgFrH{53{oNri+dU$
z#wS83xMurAT6CD$7<|U$lVq7ZR)!GxC$A at 5p-zC%4Z;|*q&r#Aw|#H_H at z;g0leuz
zE!)%;fhXWl`sl|@rQ7{3A{O=9_HMp`tsIg9hbJ?8;g)tI^40Gr^2^ojF|{{X(zz|_
z5Cyq9+|$|ca}c%Dk?&?bk{6|>4%(3t^rex!7ss8I+MpUjSQp%Epw6=L@@+o6=!_4J
z=+AHo(KkON>91G^g{ne`2-%-qQCsSR>16n5M>Fa%DMo2Nx^E`SGA<QtWw>H*3`@2E
z+Hu1E=?~y3nzmSHvRA|eS;BSk at X1M)J1+Ix$Nk*(q0NGV2!NHK%c)Cq?4%N<Jy}w(
z%8d4<SZeEOJ&9N9mtBVa_r;IDE9FMkP*85~cG?Pv7$HLCmreAfpNk|W at UGz>w*#t{
zKm_d6`*nHg^@;@-d8;nSfsQ9BvgD+D?=*O;01M0}vz++bKowR9(r2MxRl-7FfEBk5
zl4r7DE7HQQmh%%x`vE|(>EEDh#P<Wgu?Z~_uA(2m=w->U8q+;h2uG=<acFzg{z%F{
zJnknk5hkQ2wQhlDm>7~v(WxkOW0D8AU#{P-E at LF8OkqX7^3_BVZxN|6CT7HwYGn?G
zs~OWT%8&)xZvvWSOuEf)J#qtW<oTWB3Ug|ltkoM_7wakWPadpz2PPyjBiY0Z*R<%0
z_~4Yef^tn)7F50M4Kmo!@wM!LG*i*FW9jq|qULG{CYUJ>sp>IOo7z at RVt-=zz#kv)
zJs?GE=f$KGX-I38MvpQd*h<v<_bS^V1FX|<EP1-OipijL!Rf7r;E*6ydE&fib*fv4
zDolDX=W_3<^zmfK3Q|X?l3AR4_i$8`DC+K|$#|SF$Jy7T`ya~NDX&>^qln2*&_~G{
z^nOgbNEC`#S-J=&dHQi^_&6~vOI39IOT3BJ%`-qyLknaU)1VEDjoo>i=SZ$UO^L>k
zRG$VVyhLb at 9XPW!cH at W|MF~kQ`ihb}axb^!BZ`X!nBg^j$VA+<OAFJGbY>7uk<c(d
zi{qqOUOS7?1gt3!2^2C+s|IQEa!No4SsG3xJHe~t5}QIwO8>m&BOjv^D;_-&7R837
zNZY_3i_E=pg%Hsp$xz9wsA^VIF_xOtH$3JfyOK+^E_acsInt^T7RhI>!GR@?UKQz)
zB5!d~frr=nb3}=bFNxXA9gv7q%>!M(;?AdTay!K1DIzBIaL15G-XUWzW~L#<=E^Fe
z;*IAT?O$us3TGQSc$UJ3S*X7yYaP&T5MfCoqv;3-V_8K6ELJmu&j%I`+L+87|A1LM
z7fm#8e00+k%=4ZeixRz6{5m+eho$lmTciioWZH5#OI7F}wLk%xn+?X0i$y}}%_rOz
z;?@w_EAYZDzV|G$$Y|gsqK79j4gz at NnWLFr*+Yt at 2)<0I)aGRo-WZ|?LrXSs<Ss^v
zau&UVl$mpLnW+a at 6~r!0z&$J3_9n!-eoQ2Ua>w#&iJe>6oh0On5h>Bh_<{2DeP|!r
zy{M05g!1&@0IAm9mY8goQkY3vH)EseaGj13aJbeMR=o^TK5Mc-1!|G9iFApl-Il{!
zXGev3tdG|rW%rV_I~oxvO#(ZGNC?U`V-Q_BckC!G))9xN$a9oAJeBZ{o&c`)(sIk%
zAT+rdtwt-RMATm$^NR?jOe5=`&^{&ofwduI&yL+PhP;^vjz$df!W(<4kjHGubK3Ry
zdA}IA;#F2G6$?aVEn0d;_uYCFz132SR78cwLb}?O#XU*1Lv_AHA}S1Lo<ME})_wiM
zZ?Y=2FE3tO%gaU2N at Zzf(1b}aMsll<6w@=F*8<ofinWa9e<vxi`6G)1kj-h^*DTcj
zY9iSrMP5{Wuj0GKv{puPsEAJ2W%nH0GS^)MUw^S|?uFjHji*mKS_zG0J+qMB!Z0&@
zP1XMus$nDDAMVUUUGN^4uaVt0RsY at dzJwI5jZXu<7H@%KWsCY2P at nXi$5=)<!t<9x
zxlL(k2=cjgRCd~NR($VFF$Apt(~_sKld$K{Df`14`5?4J_51T5JNF?$BL7MsqT)VC
zDi|;k=+<B`d2mgNGpK4Xe|UewV;pTTFnw?^{B;4TSrS2PY8a6N*vTGfbY|v<Sy5De
Q1Hqd?$cKXOjIZzi0lwN?!~g&Q
More information about the Amavisd-new-debian-devel
mailing list