sendmail and starttls failing

[Michael Grant]

After an update today, sendmail is refusing to accept mail. I'm
seeing this in the logs:

STARTTLS=read, info: fds=9/4, err=2

Here's the full log from when I try to send a message through my
server with authentication:

Jun 30 11:42:59 bottom sm-mta[18852]: NOQUEUE: connect from [1.2.3.4]
Jun 30 11:42:59 bottom sm-mta[18852]: AUTH: available mech=DIGEST-MD5 CRAM-MD5, allowed mech=EXTERNAL
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: Milter (clamav): init success to negotiate
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: Milter (spamassassin): init success to negotiate
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: Milter (opendkim): init success to negotiate
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: Milter: connect to filters
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: milter=clamav, action=connect, continue
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: milter=spamassassin, action=connect, continue
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: milter=opendkim, action=connect, continue
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 220 bottom.networkguild.org ESMTP Sendmail 8.17.1.9/8.17.1.9/Debian-2+deb12u2; Sun, 30 Jun 2024 11:42:59 -0400; (No UCE/UBE) logging access from: [1.2.3.4](FAIL)-[1.2.3.4]
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: <-- EHLO [1.2.3.4]
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: milter=spamassassin, action=helo, continue
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250-bottom.networkguild.org Hello [1.2.3.4], pleased to meet you
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250-ENHANCEDSTATUSCODES
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250-PIPELINING
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250-EXPN
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250-VERB
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250-8BITMIME
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250-SIZE
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250-STARTTLS
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250-DELIVERBY
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 250 HELP
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: <-- STARTTLS
Jun 30 11:42:59 bottom sm-mta[18852]: engine=(null), path=(null), ispre=0, pre=0, initialized=0
Jun 30 11:42:59 bottom sm-mta[18852]: tls_srv_features=(null), relay=[1.2.3.4] [1.2.3.4]
Jun 30 11:42:59 bottom sm-mta[18852]: tls_srv_features=empty, stat=0, relay=[1.2.3.4] [1.2.3.4]
Jun 30 11:42:59 bottom sm-mta[18852]: 45UFgx2h018852: --- 220 2.0.0 Ready to start TLS
Jun 30 11:42:59 bottom sm-mta[18852]: STARTTLS=server, info: fds=9/4, err=2
Jun 30 11:43:00 bottom sm-mta[18852]: STARTTLS=server, get_verify: 0 get_peer: 0x0
Jun 30 11:43:00 bottom sm-mta[18852]: STARTTLS=server, relay=[1.2.3.4], version=TLSv1.2, verify=NOT, cipher=ECDHE-RSA-AES256-GCM-SHA384, bits=256/256
Jun 30 11:43:00 bottom sm-mta[18852]: STARTTLS=server, cert-subject=, cert-issuer=, verifymsg=ok
Jun 30 11:43:00 bottom sm-mta[18852]: AUTH: available mech=DIGEST-MD5 CRAM-MD5 LOGIN PLAIN, allowed mech=EXTERNAL
Jun 30 11:43:00 bottom sm-mta[18852]: STARTTLS=read, info: fds=9/4, err=2
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2h018852: <-- EHLO [1.2.3.4]
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: milter=spamassassin, action=helo, continue
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: --- 250-bottom.networkguild.org Hello [1.2.3.4], pleased to meet you
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: --- 250-ENHANCEDSTATUSCODES
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: --- 250-PIPELINING
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: --- 250-EXPN
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: --- 250-VERB
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: --- 250-8BITMIME
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: --- 250-SIZE
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: --- 250-DELIVERBY
Jun 30 11:43:00 bottom sm-mta[18852]: 45UFgx2i018852: --- 250 HELP
Jun 30 11:43:00 bottom sm-mta[18852]: STARTTLS=read, info: fds=9/4, err=2

My cert for bottom.networkguild.org is still valid. Err=2 is generaly
some sort of file-not-found error, but what file or file descriptor
went bad?

[Michael Grant]

Post by Michael Grant
Jun 30 11:43:00 bottom sm-mta[18852]: AUTH: available mech=DIGEST-MD5 CRAM-MD5 LOGIN PLAIN, allowed mech=EXTERNAL

Update here, it's not apparently an STARTTLS error, it's an AUTH
error. Something in the update last night altered my list of
available AUTH mechanisms.

I manually updated sendmail.cf and updated this line:

O AuthMechanisms=EXTERNAL DIGEST-MD5 CRAM-MD5 NTLM LOGIN PLAIN

by adding "DIGEST-MD5 CRAM-MD5 NTLM LOGIN PLAIN" and now it accepts
mail from my desktop.

I don't see where this is configured. /etc/sasl2/Sendmail.conf which
is a link to /etc/mail/sasl/Sendmail.conf.2, but this file looks good,
I don't know where it's getting the AuthMechanisms from (yet).

[Tim Woodall]

Post by Michael Grant

Post by Michael Grant
Jun 30 11:43:00 bottom sm-mta[18852]: AUTH: available mech=DIGEST-MD5 CRAM-MD5 LOGIN PLAIN, allowed mech=EXTERNAL

Update here, it's not apparently an STARTTLS error, it's an AUTH
error. Something in the update last night altered my list of
available AUTH mechanisms.
O AuthMechanisms=EXTERNAL DIGEST-MD5 CRAM-MD5 NTLM LOGIN PLAIN
by adding "DIGEST-MD5 CRAM-MD5 NTLM LOGIN PLAIN" and now it accepts
mail from my desktop.
I don't see where this is configured. /etc/sasl2/Sendmail.conf which
is a link to /etc/mail/sasl/Sendmail.conf.2, but this file looks good,
I don't know where it's getting the AuthMechanisms from (yet).

I think this is configured in sasl.m4

and I suspect it's something to do with the "sm_version_math" stuff but
exactly what has changed to break this for you I don't know

ifelse(eval(sm_version_math >= 526848), `1', `dnl
ifelse(sm_enable_auth, `yes', `dnl
dnl #
dnl # Set a more reasonable timeout on negotiation
dnl #
define(`confTO_AUTH', `2m')dnl # , def=10m
dnl #
dnl # Do not touch anything above this line...
dnl #
dnl # Available Authentication methods
dnl #
define(`confAUTH_MECHANISMS',dnl
`DIGEST-MD5 CRAM-MD5 PLAIN LOGIN')dnl
dnl #
dnl # These, we will trust for relaying
dnl #
TRUST_AUTH_MECH(`DIGEST-MD5 CRAM-MD5 PLAIN LOGIN')
dnl #
dnl # for 8.12.0+, add EXTERNAL as an available & trusted mech (w/STARTTLS)
dnl # and allow sharing of /etc/sasldb(2) file, allow group read/write
dnl #
ifelse(eval(sm_version_math >= 527360), `1', `dnl
define(`confAUTH_MECHANISMS',dnl
`EXTERNAL 'defn(`confAUTH_MECHANISMS'))dnl
TRUST_AUTH_MECH(`EXTERNAL')
define(`confDONT_BLAME_SENDMAIL',dnl
defn(`confDONT_BLAME_SENDMAIL')`,GroupReadableSASLDBFile,GroupWritableSASLDBFile')dnl
')dnl

[Tim Woodall]

Post by Michael Grant
After an update today, sendmail is refusing to accept mail. I'm

Hmmm, this update seems to have done a lot of odd things.

MSP Queue status...
/var/spool/mqueue-client (2 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
45U9e1iI018145 30770 Sun Jun 30 10:40 MAILER-DAEMON
(Deferred: 421 4.5.0 Bare carriage return (CR) not allowed)
root
45U5Qnln008885 28799 Sun Jun 30 06:26 root
7BIT (Deferred: 421 4.5.0 Bare carriage return (CR) not allowed)
root
Total requests: 2
MTA Queue status...
/var/spool/mqueue is empty
Total requests: 0



That's the cron email telling me about the update.

It's not at all clear to me what it's complaining about.
***@dirac:/var/spool/mqueue-client# od -t x1 qf45U* | grep 0d
***@dirac:/var/spool/mqueue-client#

Unless it's the bare CR in the body of the email - which should be fine!

Moving the queue files from mqueue-client to mqueue and fixing up the
owner and perms and they delivered fine.

[Michael Grant]

Post by Tim Woodall

Post by Michael Grant
After an update today, sendmail is refusing to accept mail. I'm

Hmmm, this update seems to have done a lot of odd things.
MSP Queue status...
/var/spool/mqueue-client (2 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
45U9e1iI018145 30770 Sun Jun 30 10:40 MAILER-DAEMON
(Deferred: 421 4.5.0 Bare carriage return (CR) not allowed)
root
45U5Qnln008885 28799 Sun Jun 30 06:26 root
7BIT (Deferred: 421 4.5.0 Bare carriage return (CR) not allowed)
root
Total requests: 2
MTA Queue status...
/var/spool/mqueue is empty
Total requests: 0
That's the cron email telling me about the update.
It's not at all clear to me what it's complaining about.
Unless it's the bare CR in the body of the email - which should be fine!
Moving the queue files from mqueue-client to mqueue and fixing up the
owner and perms and they delivered fine.

Yeah I'm seeing this too! Identical in fact. This is what I did to
fix this: I added this to my /etc/mail/access file for my local
server that sends this messages to me:

SRV_Features:127.0.0.1 L U G

Specifically, I added the U and G features, (I already had the L
feature disabled for localhost). Uppercase letter disables the
feature, lowercase enables it.

I found the U and G mentioned here:

https://forums.oracle.com/ords/apexds/post/solaris-11-4-sendmail-issue-after-sendmail-8-18-1-update-7312

I did not try this suggestion to use U2 and G2 that he mentioned. If
you do let me know.

[Tim Woodall]

Post by Michael Grant
Yeah I'm seeing this too! Identical in fact. This is what I did to
fix this: I added this to my /etc/mail/access file for my local
SRV_Features:127.0.0.1 L U G
Specifically, I added the U and G features, (I already had the L
feature disabled for localhost). Uppercase letter disables the
feature, lowercase enables it.
https://forums.oracle.com/ords/apexds/post/solaris-11-4-sendmail-issue-after-sendmail-8-18-1-update-7312
I did not try this suggestion to use U2 and G2 that he mentioned. If
you do let me know.

Thanks!

I've just added u2 g2 and it seems to work. My quick test had bare LF
removed and bare CR replaced by space which isn't what I expected but is
good enough...

[Tim Woodall]

Post by Tim Woodall

Post by Michael Grant
Yeah I'm seeing this too! Identical in fact. This is what I did to
fix this: I added this to my /etc/mail/access file for my local
SRV_Features:127.0.0.1 L U G
Specifically, I added the U and G features, (I already had the L
feature disabled for localhost). Uppercase letter disables the
feature, lowercase enables it.
https://forums.oracle.com/ords/apexds/post/solaris-11-4-sendmail-issue-after-sendmail-8-18-1-update-7312
I did not try this suggestion to use U2 and G2 that he mentioned. If
you do let me know.

Thanks!
I've just added u2 g2 and it seems to work. My quick test had bare LF
removed and bare CR replaced by space which isn't what I expected but is
good enough...

Actually, in bookworm this only seems to work with cr. mail wasn't
sending a lf and my email client was not displaying ^f

This works for testing cr and lf.
echo -ne 'Subject: test\n\ncr\rcr/lf\nlf' | /usr/sbin/sendmail -i -- root

[Tim Woodall]

Post by Tim Woodall

Post by Tim Woodall

Post by Michael Grant
Yeah I'm seeing this too! Identical in fact. This is what I did to
fix this: I added this to my /etc/mail/access file for my local
SRV_Features:127.0.0.1 L U G
Specifically, I added the U and G features, (I already had the L
feature disabled for localhost). Uppercase letter disables the
feature, lowercase enables it.
https://forums.oracle.com/ords/apexds/post/solaris-11-4-sendmail-issue-after-sendmail-8-18-1-update-7312
I did not try this suggestion to use U2 and G2 that he mentioned. If
you do let me know.

Thanks!
I've just added u2 g2 and it seems to work. My quick test had bare LF
removed and bare CR replaced by space which isn't what I expected but is
good enough...

Actually, in bookworm this only seems to work with cr. mail wasn't
sending a lf and my email client was not displaying ^f
This works for testing cr and lf.
echo -ne 'Subject: test\n\ncr\rcr/lf\nlf' | /usr/sbin/sendmail -i -- root

This is what I see in sendmail logs:
Jul 1 17:06:55 dirac sm-mta[21391]: 461G6tQr021391: collect: relay=localhost, from=<***@dirac.home.woodall.me.uk>, info=Bare carriage return (CR) not allowed, where=body, status=replaced

I don't think bare LF are a problem for sendmail which is why I suspect
they're not being replaced.

[Tim Woodall]

Post by Tim Woodall

Post by Michael Grant
After an update today, sendmail is refusing to accept mail. I'm

Hmmm, this update seems to have done a lot of odd things.

***@dirac:~# mail root
Cc:
Subject: test cr
this
is^Ma test
.
***@dirac:~# mailq
MSP Queue status...
/var/spool/mqueue-client (1 request)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
45ULV1xk014043 15 Sun Jun 30 22:31 ***@dirac.home.woodall.me.uk
(Deferred: 421 4.5.0 Bare carriage return (CR) not allowed)
root
Total requests: 1
MTA Queue status...
/var/spool/mqueue is empty
Total requests: 0



According to this
https://support.trustwave.com/kb/KnowledgebaseArticle10016.aspx

bare CRs aren't allowed in emails but this has always worked.

I'm only likely to have cron generating emails like this.

Strange that this would have been changed in a stable release. It
doesn't seem to have been a security update.

[Greg Wooledge]

Post by Tim Woodall
According to this
https://support.trustwave.com/kb/KnowledgebaseArticle10016.aspx
bare CRs aren't allowed in emails but this has always worked.
I'm only likely to have cron generating emails like this.
Strange that this would have been changed in a stable release. It
doesn't seem to have been a security update.

It looks like it's coming from this change:

https://metadata.ftp-master.debian.org/changelogs//main/s/sendmail/sendmail_8.17.1.9-2+deb12u2_changelog

* Fix CVE-2023-51765 (Closes: #1059386):
sendmail allowed SMTP smuggling in certain configurations.
Remote attackers can use a published exploitation
technique to inject e-mail messages with a spoofed
MAIL FROM address, allowing bypass of an SPF protection
mechanism. This occurs because sendmail supports
<LF>.<CR><LF> but some other popular e-mail servers
do not. This is resolved with 'o' in srv_features.

I don't know the details of how this leads to a security hole.

[Tim Woodall]

Post by Greg Wooledge

Post by Tim Woodall
According to this
https://support.trustwave.com/kb/KnowledgebaseArticle10016.aspx
bare CRs aren't allowed in emails but this has always worked.
I'm only likely to have cron generating emails like this.
Strange that this would have been changed in a stable release. It
doesn't seem to have been a security update.

https://metadata.ftp-master.debian.org/changelogs//main/s/sendmail/sendmail_8.17.1.9-2+deb12u2_changelog
sendmail allowed SMTP smuggling in certain configurations.
Remote attackers can use a published exploitation
technique to inject e-mail messages with a spoofed
MAIL FROM address, allowing bypass of an SPF protection
mechanism. This occurs because sendmail supports
<LF>.<CR><LF> but some other popular e-mail servers
do not. This is resolved with 'o' in srv_features.
I don't know the details of how this leads to a security hole.

It might be - but the wording suggested that this is blocking bare <LF>
which isn't my problem - and also I'd assume this is header related.

The thing I'm seeing is <CR> in the body of the email - I had no idea
this was illegal - and I'm surprised that tools like cron don't do
something to avoid sending "illegal" emails. Indeed, even mail will do
so happily.

[Mark Fletcher]

Post by Tim Woodall
The thing I'm seeing is <CR> in the body of the email - I had no idea
this was illegal - and I'm surprised that tools like cron don't do
something to avoid sending "illegal" emails. Indeed, even mail will do
so happily.
cron isn’t a mail sending tool — not the right place to police something

like this. Seems to me that sendmail is.

Mark

[Tim Woodall]

Post by Mark Fletcher

Post by Tim Woodall
The thing I'm seeing is <CR> in the body of the email - I had no idea
this was illegal - and I'm surprised that tools like cron don't do
something to avoid sending "illegal" emails. Indeed, even mail will do
so happily.
cron isn?t a mail sending tool ? not the right place to police something

like this. Seems to me that sendmail is.
Mark

Sendmail now polices it - so cron emails get stuck if they contain a
bare cr. Presumably every mta is now doing something similar.

There may be a cron setting that I'm missing to avoid this, I haven't
looked yet.

It is, of course, possible to run the output of every cron job through a
filter too.

my sendmail is now replacing bare cr with space so cron emails are
delivered.

[Greg Wooledge]

cron isn’t a mail sending tool — not the right place to police something
like this. Seems to me that sendmail is.

There are two possible layers here. First, a cron job (typically a
shell command, or a shell script) might invoke mailx(1) or mail(1)
directly. In this case, it's the responsibility of mailx or mail to
format and transmit the message correctly when it invokes the
/usr/sbin/sendmail program.

The second case, which is much more common, is that cron(8) itself
invokes /usr/sbin/sendmail to inject a message whenever a job writes
output to either stdout or stderr. This output is captured by cron,
and then emailed to the job's owner.

hobbit:~$ strings /usr/sbin/cron | grep sendmail
/usr/sbin/sendmail

Looks like cron is doing what I would expect. In this case, it's cron's
responsibility to make sure the message is correctly formatted.

If cron is injecting mail that /usr/sbin/sendmail is rejecting due to
bare LF or bare CR (for whichever implementation of /usr/sbin/sendmail
is installed), then this is a bug in cron.

If your cron job is calling mailx or mail directly, and one of those
tools is injecting a message that gets rejected due to bare LF or bare CR,
then this is a bug in mailx or mail, and should be reported as such.
The same applies to any CLI MUA -- mutt, nail, s-nail, Mail, etc. Also,
note that mailx has multiple implementing packages in Debian, at least
two that I know of: bsd-mailx and mailutils. Make sure you file your
bug reports against the correct packages.

[Jeffrey Walton]

Post by Greg Wooledge

Post by Tim Woodall
According to this
https://support.trustwave.com/kb/KnowledgebaseArticle10016.aspx
bare CRs aren't allowed in emails but this has always worked.
I'm only likely to have cron generating emails like this.
Strange that this would have been changed in a stable release. It
doesn't seem to have been a security update.

https://metadata.ftp-master.debian.org/changelogs//main/s/sendmail/sendmail_8.17.1.9-2+deb12u2_changelog
sendmail allowed SMTP smuggling in certain configurations.
Remote attackers can use a published exploitation
technique to inject e-mail messages with a spoofed
MAIL FROM address, allowing bypass of an SPF protection
mechanism. This occurs because sendmail supports
<LF>.<CR><LF> but some other popular e-mail servers
do not. This is resolved with 'o' in srv_features.
I don't know the details of how this leads to a security hole.

Take a look at the blog at
<https://sec-consult.com/blog/detail/smtp-smuggling-spoofing-e-mails-worldwide/>.

Jeff

[Jeffrey Walton]

Post by Tim Woodall

Post by Tim Woodall

Post by Michael Grant
After an update today, sendmail is refusing to accept mail. I'm

Hmmm, this update seems to have done a lot of odd things.

Subject: test cr
this
is^Ma test
.
MSP Queue status...
/var/spool/mqueue-client (1 request)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
(Deferred: 421 4.5.0 Bare carriage return (CR) not allowed)
root
Total requests: 1
MTA Queue status...
/var/spool/mqueue is empty
Total requests: 0
According to this
https://support.trustwave.com/kb/KnowledgebaseArticle10016.aspx
bare CRs aren't allowed in emails but this has always worked.
I'm only likely to have cron generating emails like this.
Strange that this would have been changed in a stable release. It
doesn't seem to have been a security update.

New SMTP smuggling attack,
<https://www.openwall.com/lists/oss-security/2023/12/21/6>.

The short of it is, non-conforming emails and sloppy parsing have led
to a litany of problems including mail spoofing. It has been going on
for years, but now things are changing.

Jeff