Line (block of text) longer than 990 characters will invalidate DKIM

I notice that when i send html email from Usermin it doesnt sign correctly DKIM. Header from Gmail:

Delivered-To: *****@gmail.com
Received: by 10.36.29.88 with SMTP id 85csp1509037itj;
        Sun, 3 May 2015 22:25:42 -0700 (PDT)
X-Received: by 10.180.78.65 with SMTP id z1mr16393149wiw.14.1430717141359;
        Sun, 03 May 2015 22:25:41 -0700 (PDT)
Return-Path: <*****@*****.com>
Received: from leeroy.*****.com (leeroy.*****.com. [xxx.xxx.xxx.xxx])
        by mx.google.com with ESMTP id wz10si20961872wjc.143.2015.05.03.22.25.40
        for <*****@gmail.com>;
        Sun, 03 May 2015 22:25:41 -0700 (PDT)
Received-SPF: pass (google.com: domain of *****@*****.com designates xxx.xxx.xxx.xxx as permitted sender) client-ip=xxx.xxx.xxx.xxx;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of *****@*****.com designates xxx.xxx.xxx.xxx as permitted sender) smtp.mail=*****@*****.com;
       dkim=neutral (body hash did not verify) header.i=@;
       dmarc=pass (p=REJECT dis=NONE) header.from=*****.com
Received: from leeroy.*****.com (localhost.localdomain [127.0.0.1])
by leeroy.*****.com (Postfix) with SMTP id 838C586006C
for <*****@gmail.com>; Mon,  4 May 2015 07:25:40 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=*****.com;
s=jenkins; t=1430717140;
bh=XYQ21n+dowazCPt9preYO1GrA7JP+pUbQstpl2GZhfs=;
h=From:Subject:To:Date;
b=Fwf1oZHWvWF5wV3RxzFwBt384sWdqaETaADeqAA04Z0/h1x1iA8CjPEC7MZl52MOV
e1zCY+/cO/EbaUscAUDyVRh+AvPcLIqa8wEjTTUIPH+6kjmlJzJuatBG8GoMcCUPOS
vu/I/2dpXAtl7nTZ/6VrOlcZiYUtdiJHHBmmeW+P0BDDZ1IVCZveCcVclkZMBZNHvG
AoqNE7pDRKyXfkyJ/aBmUwT4owl2E7oGaKFzg/bX7cK23zLa2Gy7bYH9N6twpTWGtw
QG+vkekdWU/KFvc0KhsJepy3aeim4KSVxM7shRycKKv0YjkNBQylAPnFWKG0mC33WE
N1C0eUAJN1pzQ==
From: "*****.com" <*****@*****.com>
Subject: Unlock The Inbox
To: *****@gmail.com
Message-Id: <1430717140.15475@*****.com>
X-Mailer: Usermin 1.650
Date: Mon, 04 May 2015 07:25:40 +0200 (CEST)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="bound1430717140"
X-Antispam: clean, score=10
X-Antivirus: avast! (VPS 150503-1, 03.05.2015), Inbound message
X-Antivirus-Status: Clean

This is a multi-part message in MIME format.

--bound1430717140
Content-Type: multipart/alternative; boundary="altsbound1430717140"
Content-Transfer-Encoding: 7bit


--altsbound1430717140
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
....................

How you can see "dkim=neutral (body hash did not verify) header.i=@;" and this is happening only for emails what i send as HTML, but if i send very same email as TXT then DKIM is validated correctly. I try to search for the solution but looks like this problem is with Usermin and not Postfix or DKIM. This problem is going on for long time as first time i encounter it was before 3-4 months. Any advice?

Centos 7 full updated, last Virtualmin/Webmin.

Status: 
Closed (fixed)

Comments

Do you have Usermin configured to send email via an SMTP connection? You can check by looking for a line like send_mode=localhost in /etc/usermin/mailbox/config

Diabolico's picture
Submitted by Diabolico on Mon, 05/04/2015 - 21:25

I have "send_mode=localhost". Should i change this to something else?

No, that should be fine. Do you also see this if you send HTML email from another mail client, like Thunderbird or Roundcube?

Diabolico's picture
Submitted by Diabolico on Tue, 05/05/2015 - 23:14

I try with Thunderbird and no problem: dkim=pass header.i=@mydomain.com;. But i did this just to test as i will not use external mail client on any of my domains. For Roundcube i dont want to install because Usermin is more than enough for what i need.

It maybe worth installing Roundcube as a test - it is a regular PHP app that can run on your website in a sub-directory, and won't interfere with Usermin.

Diabolico's picture
Submitted by Diabolico on Thu, 05/07/2015 - 00:27

Test with Roundcube was ok and html email didnt show any problems. Can we now focus on Usermin and what or where could be a problem?

Received-SPF: pass...
dkim=pass header.i=@mydomain.com;
dmarc=pass...

This problem is old several months, well at least when i saw it first time. I already told you that something must be wrong with Usermin as everything is pointing in that direction. I check several times my config files and suggestion what i found with other people with same problem it was always a faulty software used to send emails and (usually) failing with body hash. I think when i tested last time the problem was with Usermin inserting empty space in DKIM and actually breaking it but i lost too much time with this problem and i'm not sure anymore. Problem show up in Centos 6 and 7 so is not something specific for one OS.

This is odd, because Usermin doesn't do the DKIM signing - it just sends an HTML email via the SMTP connection. A separate DKIM server then appends the signature, which it should base on the original un-signed contents.

One possible cause is that the HTML message is formatted in a way that cannot be DKIM signed. Does your email contain any non-ASCII characters, like ä ?

Diabolico's picture
Submitted by Diabolico on Fri, 05/08/2015 - 00:45

No it was text in pure english with html formatting, same one i sent with Thunderbird, Roundcube and Usermin. Is there anything how Usermin works and what could trigger this problem with postfix? I mean something different how Usermin send email compared to Roundcube?

EDIT: Here is my "main.cf" file: http://pastebin.com/9zEevbWp

One possible cause is that Usermin is sending the HTML email with an encoding that the DKIM signer cannot handle. I can fix this, but it will require a new Usermin release - there isn't any simple work-around.

Diabolico's picture
Submitted by Diabolico on Sat, 05/09/2015 - 16:41

Hey Jamie looks like is nothing else to do but to check Usermin. Whatever i try to do Usermin have problems with DKIM and html emails. Here is log from one of many test i did with html emails:

May  9 23:28:00 leeroy postfix/cleanup[11935]: BF6518626E7: message-id=<1431206880.12032@mydomain.com>
May  9 23:28:00 leeroy opendkim[11827]: BF6518626E7: DKIM-Signature field added (s=mail, d=mydomain.com)
May  9 23:28:00 leeroy postfix/qmgr[11923]: BF6518626E7: from=<myemail@mydomain.com>, size=58859, nrcpt=1 (queue active)
May  9 23:28:00 leeroy postfix/smtpd[11931]: disconnect from localhost.localdomain[127.0.0.1]
May  9 23:28:02 leeroy postfix/smtp[11941]: BF6518626E7: to=<mygmail@gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.206.26]:25, delay=1.3, delays=0.11/0/0.38/0.84, dsn=2.0.0, status=sent (250 2.0.0 OK 1431206882 gs3si4931936wib.29 - gsmtp)
May  9 23:28:02 leeroy postfix/qmgr[11923]: BF6518626E7: removed

How you can see from the logs opendkim did sign the email but could be that Usermin have some problems with formatting html emails and as consequences we have this bug. Maybe the solution is not so hard. Looking around usually the problem is when mail software/script breaks dkim after the email was signed, e.g. empty space. I'm not saying that this is the case here but maybe is worth to check and doesnt require too much time to repair.

My only guess is that Usermin's encoding of HTML email is triggering a bug in OpenDKIM, causing it to generate incorrect signatures. All I can do in the next release is making the encoding method for base64 emails more flexibile, in an attempt to work around this.

Diabolico's picture
Submitted by Diabolico on Fri, 05/15/2015 - 17:31

After last update of Usermin this problem is still present. I took html email and forward to my gmail and this is what i got - from maillog:

May 16 00:06:07 leeroy postfix/smtpd[25443]: connect from localhost.localdomain[127.0.0.1]
May 16 00:06:07 leeroy postfix/smtpd[25443]: 9086A860068: client=localhost.localdomain[127.0.0.1]
May 16 00:06:07 leeroy postfix/cleanup[25446]: 9086A860068: message-id=<1431727567.25442@mydomain.com>
May 16 00:06:07 leeroy opendkim[537]: 9086A860068: DKIM-Signature field added (s=mail, d=mydomain.com)
May 16 00:06:07 leeroy postfix/qmgr[807]: 9086A860068: from=<vminemail@mydomain.com>, size=67471, nrcpt=1 (queue active)
May 16 00:06:07 leeroy postfix/smtpd[25443]: disconnect from localhost.localdomain[127.0.0.1]
May 16 00:06:09 leeroy postfix/smtp[25450]: 9086A860068: to=<mymail@gmail.com>, relay=gmail-smtp-in.l.google.com[173.194.67.27]:25, delay=1.6, delays=0.25/0.02/0.4/0.93, dsn=2.0.0, status=sent (250 2.0.0 OK 1431727569 bq3si5076485wjc.50 - gsmtp)
May 16 00:06:09 leeroy postfix/qmgr[807]: 9086A860068: removed

Gmail header: dkim=neutral (body hash did not verify) header.i=@;

And if i send over Roundcube then the message is ok and gmail gives ok to DKIM. So we are at beginning with this problem and new update didnt sort this problem at all.

Try editing the file /etc/usermin/mailbox/config , and at the end adding the line html_base64=2

Diabolico's picture
Submitted by Diabolico on Fri, 05/15/2015 - 23:49

Still nothing. I try with html_base64=2 and html_base64=1 (orig. was html_base64=0), restart Umin and postfix but nothing. DKIM fails every time. Something happens after DKIM is signed to the email. Maybe Umin is making changes what then breaks DKIM.

Diabolico's picture
Submitted by Diabolico on Mon, 05/18/2015 - 05:47

I sent you two emails, from Usermin and Roundcube, and i put in subject what email is sent from what software. If is not clear what is going on i can give you access to my VPS.

Thanks for the emails - I confirmed that on my system DKIM signature verification also failed. However, I don't see anything clearly wrong with the message from Usermin that could cause this :-(

It is unfortunately difficult to debug this because the software doing the DKIM signing wasn't developed by us, and I can't re-produce the issue on a test system.

Diabolico's picture
Submitted by Diabolico on Tue, 05/19/2015 - 06:17

I sent you new email please check and see if this new info can help you to find the problem with Usermin.

Thanks, but those files don't really have much usable debug info in them :-(

Another thing you should try is sending a plain-text email, but with an HTML file selected as an attachment. I'd be interested to know if that fails the DKIM check.

Diabolico's picture
Submitted by Diabolico on Wed, 05/20/2015 - 13:40

I made simple HTML page and sent as attachment to Gmail and no problem.

EDIT: Ok i made two test, one with small html page (600 bytes) and with heavier html page (13 KB) and no problem with any of this two emails. The problem is when HTML is inside the email, e.g. email is composed as html.

That is odd, because an HTML email sent by Usermin is really just a mail with a single HTML attachment!

Diabolico's picture
Submitted by Diabolico on Fri, 05/22/2015 - 08:58

There must be a way how to check what is going on. For what i try until now Usermin is the only one who fail to properly deliver html emails. Using something else because Usermin is not working i dont find as good solution nor to have a chance for my emails to end in spam folder or to lower IP/domain reputation because of this.

You are probably best off filing a bug against the openDKIM developers - there isn't a lot we can do to debug it. I agree that likely something Usermin is doing that triggers the bug, but even if we can identify that the solution would be at best a work-around.

Diabolico's picture
Submitted by Diabolico on Wed, 05/27/2015 - 11:57

Sorry Jamie but this doesnt make sense. Usermin is faulty here but you want me to open bug report with OpenDKIM? I would understand if this problem is present with other variation but it is not. OpenDKIM is signing the emails but then something happens in the way how Usermin handle this emails and as result we have broken DKIM.

I will go and open bug report with OpenDKIM but i expect them to turn me back here.

OpenDKIM should be able to properly sign any email that it receives - unless Usermin is doing something like including invalid characters in the message (which it doesn't seem to be), a proper signature should be created.

Diabolico's picture
Submitted by Diabolico on Sun, 06/19/2016 - 17:55

Looks like this problem was never solved. Right now i tested again by sending several emails, text only(!) but different in amount of content. So half of them had under 1000 characters and less than 1.4KB and half of them 1.60-2.00KB or more, e.g. from 1000 characters and up.

It looks like the line is somewhere between 1.48KB and 2.00KB as in this range the emails will report broken DKIM - "dkim=neutral (body hash did not verify) header.i=@mydomain.com;". In characters this means any email will less than 1000 characters will have valid DKIM but for any other email i tested with more than 1000 character DKIM fails.

After more than a year it is sad to see that no one ever touch this problem.

Diabolico's picture
Submitted by Diabolico on Sun, 06/19/2016 - 18:22

Title: Dkim failing for HTML emails » Dkim failing for emails bigger than 1.40KB
Diabolico's picture
Submitted by Diabolico on Sun, 06/19/2016 - 18:23

I changed the title as now i have more info and is not related to text or HTML emails but rather the size.

Diabolico's picture
Submitted by Diabolico on Sun, 06/19/2016 - 18:53

Raw email examples from Gmail when DKIM is ok and when it fails:

  1. http://pastebin.com/7ayn8vS3 - DKIM is ok, email size: 1.45KB, text only, body character count: 989

  2. http://pastebin.com/ezhtYn30 - DKIM fails, email size: 1.57KB, text only, body character count: 1107

  3. http://pastebin.com/mGFQ9REn - DKIM pass, email size: 1.57KB, text only, body character count: 1107, manually split the text in 3 parts (new line)

Mail 2 and 3 are same apart of one thing, in 3rd email i manually split the text in 3 parts (e.g. new line). So it looks like the trigger is the character amount in one single block of text. Once it pass 998 it will break DKIM for unknown reason but if you take that same text and manually split so each block doesnt have more than 998 characters DKIM will be ok.

In second email something broke the text:

...controversiis implicatam,
  possem cognoscere....

It easy to see that Usermin or something else after 998 characters broke the line, inserted 3 new characters (gmail raw email show +3 characters in body part) and continue with rest of the text in new line.

So maybe the trigger is having one really really long line of text? Not the total size of the email?

The DKIM signing is done by OpenDKIM and not Usermin, so that's likely where the error lies. I suppose Usermin could work around this by splitting up email messages with really long lines though.

Diabolico's picture
Submitted by Diabolico on Mon, 06/20/2016 - 23:06

Correct, once the character count (including white space) come close to 1000 it will break DKIM. You can see in 2nd email how the text near the end was split in new line plus extra 3 characters were added. I'm pretty sure this is not the problem with DKIM but something else is the culprit. From the logs it is clear that every email is signed with DKIM so no problem here.

I found some info explaining if the email is edited/changed after DKIM was applied it will invalidate DKIM and i think this is what is going on in this case. Must be something with Usermin or combination of more things. Now i dont know how Usermin works but this new info should help you to trace the problem.

I have a strong suspicion that either the sending or receiving mail server is wrapping a line longer than 1000 characters. In fact, the SMTP spec doesn't allow longer lines - see http://www.jebriggs.com/blog/2010/07/smtp-maximum-line-lengths/

It's unclear what Virtualmin could do better here. All I can think of is for Usermin to forcibly wrap text lines longer than 1000 characters in the message body.

Diabolico's picture
Submitted by Diabolico on Wed, 06/22/2016 - 01:30

I made few more testing to see how Gmail deals with long emails. From my Gmail account i sent same email to my VPS and to another Gmail account. Usermin wrapped the email to 80 characters and other Gmail account got the email without modifications. I know that Usermin is by default set to 80 characters and i could change that but in the same time Gmail found the way how to deliver the email with more than 1000 characters without any problem.

Looking again in Usermin under "Writing email" there is option "Width to wrap composed mail messages at" what by default is set to "Don't wrap".

Temporary solution would be to set "Width to wrap composed mail messages at" to "80 columns (standard)" but that means every email leaving my server will be cut out with multiple "new line" and i dont see any reason why i cant do that manually. Looks like this is not well know problem, doesnt appear frequently, people dont see it or they try to find the culprit somewhere else (like it happened here). Initially even you didnt see what could be the problem and if i'm not wrong this is not only bug report with same subject (not sure if other is bug report or forum topic).

While i understand there is no way that one person know everything but to lose so much time for such things makes me angry. I suspect this problem didnt show too many times if at all but when i test something i like to push things as much as i can and then see if and how it will break. Aside of all this maybe is time for Usermin to get some attention and love. Its great software and can be used for many things but it gives a feeling that development stopped at one point and never recovered. Personally i love that Gmail look and feeling and i will always use Usermin over any other email software but its asking for some improvements.

Can i suggest few things:

  1. Leave Usermin with "Don't wrap" (for now) but add info/warning that any text longer than 1000 characters will break DKIM or even entire email.

  2. Change default settings to "80 columns (standard)" but leave the warning with "Don't wrap" (even i think no. 1 is better solution).

  3. Suggestion 2 and 3 should be temporary until the final solution is applied.

  4. If there is no solution then add character count in Usermin what will prevent the email to be sent if one block of text have more than 990 characters.

  5. Same as 4. but instead of preventing the email to be sent to have a warning with info what is the problem and option to send as it is or cancel and return to compose email.

Any opinion from your side?

What I'll do is force wrapping of lines longer than 1000 characters, regardless of the user's setting. Because the SMTP protocol mandates that lines be less than 1000, this seems like a reasonable thing to do.

Thanks for finding this bug - I've seen a couple of reports of DKIM checks failing, but never figured out that it was due to long lines!

Status: Active » Fixed
Diabolico's picture
Submitted by Diabolico on Thu, 06/23/2016 - 05:41

Title: Dkim failing for emails bigger than 1.40KB » Line (block of text) longer than 990 characters will invalidate DKIM

Are you sure it's 990 characters and not 1000 ?

Diabolico's picture
Submitted by Diabolico on Thu, 06/23/2016 - 23:35

Ok this is another test.

Original message was 1001 characters long

1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

Gmail got this email

--1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 --
-- 0 1--

You will notice empty space after 9 and before 0 so basically the text was cut after 998 characters. To not be confused i added "--" to make it easier to see how and where this text was cut in two. Based on this link - https://tools.ietf.org/html/rfc2822

2.1.1. Line Length Limits

There are two limits that this standard places on the number of characters in a line. Each line of characters MUST be no more than 998 characters, and SHOULD be no more than 78 characters, excluding the CRLF.

The 998 character limit is due to limitations in many implementations which send, receive, or store Internet Message Format messages that simply cannot handle more than 998 characters on a line. Receiving implementations would do well to handle an arbitrarily large number of characters in a line for robustness sake. However, there are so many implementations which (in compliance with the transport requirements of [RFC2821]) do not accept messages containing more than 1000 character including the CR and LF per line, it is important for implementations not to create such messages.

Sorry, I was sure its 990 but looks like 998 is more likely to be the number we are looking for. To be on safe side i would go for 998 characters as limit and after that enforce wrapping.

To not make more confusion i will leave to you to edit the title to be more in line with this situation and easier to find for others in case they encounter same problem.

Ok, I will apply a limit of 998 characters in the next release of Usermin.