UTF-8 Problems

Hi guys,

You did tell me before that the non unicode languages support was being removed from Webmin/Virtualmin, and you suggested to use UTF-8 instead of our working non unicode encoding. I of course followed your advice and set up our new Virtualmin Pro to use UTF-8 US as default before migrating all virtual servers from our old CentOS 5 box using non unicode EN(us). I was afraid that we would run into trouble using UTF and I was right! We had to manually change lots of information, especially use info, and some servers has really lots of users, that took some time to finish but that is all done now.

Now we again run into UTF-8 problems! When adding a automatic reply in "Virtualmin/Edit Users/SELECTED-USER/Mail forwarding settings/Send automatic reply" the reply is crap, no matter what I select in "Automatic reply options/Character set for reply message" the user name in "from" field is crap or the whole message is crap when the sender gets the auto reply. User has Swedish characters in his "Real name" field and that name always is crap in the received auto reply! I also tried to change/re-typed the user real name even though it's OK in the form. This is due to the totally worthless UTF-8 encoding that never works anywhere! We never had any problems with this on our old box set up with non unicode EN. NOTE this is done by me(root) in Virtualmin/Webmin as our customers wants us to administrate these things.

I thinking on switching the language in Webmin/Virtualmin to a working encoding, the non unicode English US(en) but I don't know how long you will keep it. I have been asking before if the non unicode languages really will be removed, and when in that case. Before I do the massive work of updating hundreds of users and all other things to non unicode again i want to know how long you will support it.

We do ran in to these UTF problems all the time, and almost everywhere. If you do a web search of unicode and unicode problems or unicode vs. non unicode you get tons if articles about the UTF problems. But if you live in a country with only the basic US/EN characters your in luck, other wise you are about to get trouble. Almost every article recommends people who are thinking of switching to unicode to stick with non unicode.

I use CentOS 7.4.1708, Webmin 1.881, Usermin 1.741, Virtualmin 6.02 Pro, Authentic Theme 19.12 on our server. My workstation in a Windows 10 Pro Swedish 64-bit, MS Office 365 Swedish (mail client is Outlook), and Latest Firefox Swedish (auto selected my language on install).

Best regards, Leffe

Status: 
Closed (fixed)

Comments

I guess the user's real name isn't using UTF-8 characters, so the name in the autoreply isn't being encoded correctly. I don't have a good solution for this, because it's not possible to know which encoding is used for the real name.

However, even though we may not make it obvious in the UI, the non-UTF-8 languages will still be available for the forseeable future.

Ok, this means that I can't use the UTF-8 setup for Virtualmin/Webmin as you suggested! And the encoding in my Firefox is unicode, see the new image.

//Leffe

Here is the image.

As you can see in my previous post my browser is set to unicode, and webmin/virtualmin is set to unicode and the autoreply is set to unicode so this should work, but it don't.

And if I try do ditch the unicode I get warned! non-unicode-warning.jpg

//Leffe

If you check the headers of the autoreply message, which character set is it using?

Hi Jamie,

It is using UTF-8.

From: "Plåt & Byggkonsult - Joakim" <joakimxxxxxx.xxx>
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8
Date: Thu, 12 Apr 2018 14:46:51 +0200 (CEST)

//Leffe

Ok, I think I see the issue - in email headers, there is a special encoding format needed for UTF-8 which is separate from the mail body. We can fix that though.

Are you also seeing problems with the encoding of the email body though?

Hi,

Mail body is fine.

//Leffe

Ok, I see now what we need to do in order to fix this (properly encode the headers). I'll update this ticket when it's done.

Thank you Jamie!

//Leffe

Ok, this has been fixed for the next Usermin/Webmin release.

Status: Active » Fixed (pending)

Hi Jamie,

Will this fix also cover for example when I have a cron job sending logs and messages from virtual servers, these mails when received also gets the "from" name unreadable when the virtual server description name contains Swedish letters. We have several customers complaining about why there company name cant be shown correctly - "it has always been fine before!"

These problems started when we selected UTF8 on our new server instead of ISO-8859-1. We never had problem like this before when using non unicode encoding.

Regards, Leffe

Yes, it will cover all cases where Virtualmin / Webmin sends email.

Hi Jamie,

Has this been fixed?

Starting next week many of our customers wants auto respond messages on their mail accounts during their vacation or changed office times. If not fixed we will have many disappointed/irritated and complaining customers next week, I need to know the status on this is before they are starting to call us about it.

Regards, Leffe

The fix will be included in the next Webmin / Usermin releases, which are happing this week.

Ok thanks!

//Leffe

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.