Ubuntu 18.04 is going EOL in April 2023. If you are like me and you are running your zimbra email server, it is time to upgrade your instance to Ubuntu 20.04. Note that you will also need to upgrade your zimbra installation from zcs-8.8.15_GA_3869 to zcs-8.8.15_GA_4179.
After several unsuccesfull trials I was able to complete the upgrade and have a working server again, and since this was not done on the first atempt I decided to write this post to help others that will try the same.
The upgrade steps are:
- BACKUP! If you are like me and run zimbra on a VM make sure you take a snapshot before you do anything. You might want to go back if things don’t go well.
- Gather server information: Run the following and save the output somewhere. You will need it later.
sudo su su - zimbra zmlocalconfig -s | grep -i password
- Take backup of some important files:
sudo su mkdir -p /opt/{zimbra_backup/.saveconfig,ldap_backup} chown zimbra:zimbra /opt/{zimbra_backup,ldap_backup} -R cp -pvr /opt/zimbra/.install_history /opt/zimbra_backup/.saveconfig/.install_history cp -pvr /opt/zimbra/conf/localconfig.xml /opt/zimbra_backup/.saveconfig/localconfig.xml apt list --installed | grep -i 'zimbra-' | cut -d'/' -f1 > /opt/zimbra_backup/.saveconfig/.installedPackages su - zimbra /opt/zimbra/libexec/zmslapcat -c /opt/ldap_backup/ /opt/zimbra/libexec/zmslapcat -a /opt/ldap_backup/ /opt/zimbra/libexec/zmslapcat /opt/ldap_backup/ - Update the current packages:
apt-get update && apt-get dist-upgrade reboot
- Upgrade ubuntu (providing the default answer to each upgrade question is your best bet) – Note that your DNS resolution might have been done by zimbra, and zimbra is now stopped. Make sure you add your DNS server in /etc/resolv.confand after that:
sudo su do-release-upgrade reboot
- Download the zimbra tgz from https://www.zimbra.com/downloads/zimbra-collaboration-open-source/ and copy it in a folder on your server.
- Untar the zimbra package:
tar -xvzf zcs-8.8.15_GA_4179.UBUNTU20_64.20211118033954.tgz
- Reinstall the zimbra package (Remember to add a DNS server that can resolve your zimbra host to /etc/resolv.conf or the following will fail):
sudo su cd zcs-8.8.15_GA_4179.UBUNTU20_64.20211118033954 ./install.sh -s --skip-activation-check
- Make sure during the install you answer NO to the following question:
The Zimbra Collaboration Server does not appear to be installed, yet there appears to be a ZCS directory structure in /opt/zimbra. Would you like to delete /opt/zimbra before installing? [N] N
- Copy the files you have saved in the begining:
cp -pvr /opt/zimbra_backup/.saveconfig/.install_history /opt/zimbra/.install_history cp -pvr /opt/zimbra_backup/.saveconfig/localconfig.xml /opt/zimbra/conf/localconfig.xml
- Run the zimbra setup:
sudo su /opt/zimbra/libexec/zmsetup.pl
- Before the install is complete you need to put back the correct passwords (use the passwords you have saved in point 2 – Told you this was important ;P ):
******* +Ldap Admin password: Not Verified ******* +Ldap postfix password: Not Verified ******* +Ldap amavis password: Not Verified ******* +Ldap nginx password: Not Verified ******* +Ldap Bes Searcher password: Not Verified ******* +Bind password for postfix ldap user: Not Verified ******* +Bind password for amavis ldap user: Not Verified ******* +Admin Password UNSET
- Once you have a clean menu like the following press a to apply:
Main menu 1) Common Configuration: 2) zimbra-ldap: Enabled 3) zimbra-logger: Enabled 4) zimbra-mta: Enabled 5) zimbra-dnscache: Enabled 6) zimbra-snmp: Enabled 7) zimbra-store: Enabled 8) zimbra-spell: Enabled 9) zimbra-proxy: Enabled 10) Default Class of Service Configuration: s) Save config to file x) Expand menu q) Quit *** CONFIGURATION COMPLETE - press 'a' to apply Select from menu, or press 'a' to apply config (? - help)
- Wait for the process to complete, reboot your server and get ready for the fun part. As zimbra run zmcontrol status and check if all services are running. Chances are you will get:
Host mail.procentric.tv amavis Running antispam Running antivirus Running dnscache Running ldap Running logger Stopped zmlogswatchctl is not running mailbox Stopped zmmailboxdctl is not running. memcached Running mta Running opendkim Running proxy Running service webapp Stopped zmmailboxdctl is not running. snmp Running spell Running stats Running zimbra webapp Stopped zmmailboxdctl is not running. zimbraAdmin webapp Stopped zmmailboxdctl is not running. zimlet webapp Stopped zmmailboxdctl is not running. zmconfigd Running
- To fix this do the following:
sudo su cp -r /opt/zimbra/ssl/zimbra/commercial ~/ssl-backup mv /opt/zimbra/mailboxd/etc/keystore /root/keystore.old su - zimbra /opt/zimbra/bin/zmcertmgr createca -new /opt/zimbra/bin/zmcertmgr createcrt -new -days 3650 /opt/zimbra/bin/zmcertmgr deployca /opt/zimbra/bin/zmcertmgr deploycrt self
- If you got an error on the last line delete /opt/zimbra/ssl/zimbra/jetty.pkcs12 and start over:
rm /opt/zimbra/ssl/zimbra/jetty.pkcs12 /opt/zimbra/bin/zmcertmgr createca -new /opt/zimbra/bin/zmcertmgr createcrt -new -days 3650 /opt/zimbra/bin/zmcertmgr deployca /opt/zimbra/bin/zmcertmgr deploycrt self exit reboot
- After the reboot, as zimbra user check what processes are running:
sudo su su - zimbra zmcontrol status
- You should probably get something like this:
amavis Running antispam Running antivirus Running dnscache Running ldap Running logger Stopped zmlogswatchctl is not running mailbox Running memcached Running mta Running opendkim Running proxy Running service webapp Running snmp Running spell Running stats Running zimbra webapp Running zimbraAdmin webapp Running zimlet webapp Running zmconfigd Running
- as root update and upgrade all packages (This should fix some perl package missmatch):
apt-get update; apt-get -y upgrade
- Reboot and check again the services. They should all be running. If they are not restart from point 14. Also, if you had commercial ssl certificates we need to put them back. For this run as root:
cp -Rf ~/ssl-backup /tmp chown -R zimbra:zimbra /tmp/ssl-backup cd /tmp/ssl-backup /opt/zimbra/bin/zmcertmgr deploycrt comm commercial.crt commercial_ca.crt
- Reboot and check your services they should all be running. login admin interface on port 7071 and check services also.
- Please let me know if you run into problems, and we can maybe figure it out.
Hi, I’m updating ubuntu 18.04 to 20.04, with Zimbra zcs-8.8.15_GA_3869 installed and updated to GA_4179. Everything ok until the creation of the new certificates, where I get this error: /opt/zimbra/bin/zmcertmgr deploycrt self
** Saving config key ‘zimbraSSLCertificate’ via zmprov modifyServer XXXXX.it…failed (rc=1)
** Installing imapd certificate ‘/opt/zimbra/conf/imapd.crt’ and key ‘/opt/zimbra/conf/imapd.key’
** Copying ‘/opt/zimbra/ssl/zimbra/server/server.crt’ to ‘/opt/zimbra/conf/imapd.crt’
** Copying ‘/opt/zimbra/ssl/zimbra/server/server.key’ to ‘/opt/zimbra/conf/imapd.key’
ERROR: imapd keytool(-delete -alias jetty) returned non-zero(1):
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
** Creating file ‘/opt/zimbra/ssl/zimbra/jetty.pkcs12’
** Creating keystore ‘/opt/zimbra/conf/imapd.keystore’
ERROR: com.zimbra.cert.MyPKCS12Import to ‘/opt/zimbra/ssl/zimbra/jetty.pkcs12’ returned non-zero(1):
Exception in thread “main” java.io.IOException: Keystore was tampered with, or password was incorrect
at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
at java.base/java.security.KeyStore.load(KeyStore.java:1473)
at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import.java:104)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:811)
… 3 more
I also tried to delete the file /opt/zimbra/ssl/zimbra/jetty.pkcs12, but the problem persists.
Do you have any suggestions for me?
Thank you
Hi Alessandro.
I completely forgot to approve and answer your comment. Is this still an issue? Were you able to complete the upgrade?
Thank you,
Paul
Hi,
I hat the same problem. For me it helped to set the mailboxd_keystore_password from the zmlocalconfig backup.
zmlocalconfig -e mailboxd_keystore_password=”PASSWORD”
After that it was possible to do step 15.
I upgraded from the latest version (8.8.15 P45). There is also a warning in the wiki regarding OpenJDK. Maybe that was also part of the problem.
https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.15/P45
Regards
I just read your manual for upgrading Ubuntu 18.04 to 20.04 on our Zimbra server. I am currently running zcs-8.8.15_GA_4464 on 18.04? Do your instructions also work on that combination?
Do I have to install the “8.8.15_GA_4177” after upgrading to 20.04? This is the only downloadable version for 20.04 I can find on the Zimbra website…
Hi Prahn,
If you upgrade your Ubuntu version I think you will have no choice but to update zimbra. The libreries used are not the same. I see the latest version for Ubuntu 20.04 is:
https://files.zimbra.com/downloads/8.8.15_GA/zcs-8.8.15_GA_4179.UBUNTU20_64.20211118033954.tgz
I can’t say if the instructions will work. If you are running the zimbra instance on a VM I would take a snapshot of the machine before going ahead with the update. Just in case you get stuck.
Thank you,
Paul
Hi Paul,
Huge thanks for putting these instructions together – it was exactly what I needed and saved me a lot of heartache!
To (hopefully) answer Alessandro’s question above, I got a similar error and needed to delete /opt/zimbra/conf/imapd.keystore as well as /opt/zimbra/ssl/zimbra/jetty.pkcs12 at step 16.
Thanks again
C
Thank you C
Hi Paul!
What do you mean with “no choice but to update zimbra”. Update zimbra to Version 9??
Thank you,
Ingo
Ingo, Zimbra 9 is not available for the community only paid. I meant that you will need to update your zimbra 8.x.x to work with the libraries included with Ubuntu 20.04. If you upgrade your OS the old zimbra installation will stop working.
Hi Paul!
I did the Update yesterday on a copy of my live VM. And everything worked as expected. I also did the update of Zimbra with the repositories of apt, but that was also a very easy step with “apt-get upgrade”.
I am using the paid Network Edition, not the free one. However I also had to delete the file /opt/zimbra/conf/imapd.keystore, that C mentioned above.
Thanks a lot for your manual!
Thank you for the feedback Ingo!
Hi Paul!
I have upgrade ubuntu to 20.04 and updated zimbra, but at the end I have this situation:
Host
amavis Running
antispam Running
antivirus Running
dnscache Running
ldap Running
logger Running
mailbox Stopped
zmmailboxdctl is not running.
memcached Running
mta Running
opendkim Running
proxy Stopped
proxy is not running.
service webapp Stopped
zmmailboxdctl is not running.
snmp Stopped
zmswatch is not running.
spell Running
stats Running
zimbra webapp Stopped
zmmailboxdctl is not running.
zimbraAdmin webapp Stopped
zmmailboxdctl is not running.
zimlet webapp Stopped
zmmailboxdctl is not running.
zmconfigd Running
In particular if I try to start the proxy I have this error:
Starting proxy…nginx: [emerg] a duplicate default server for 0.0.0.0:443 in /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default:34
Do you have any tips on how to resolve this issue?
Hi Andrea,
I would look in /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default.
It looks like you have 2 enteries for default server for 0.0.0.0:443
I would make a backup of that file and remove everything related to one of the enteries.
Please let me know if that helps.
Thank you,
Paul
Hi Paul,
thanks for your fast reply.
In fact the file nginx.conf.web.https.default there are two “server” block with listen port 443.
(this is the file /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default after the upgrade)
server {
listen 443 default_server ssl;
server_name _;
…
}
server
{
listen 443 default_server ssl http2; # <– line 34 reported by the error message
server_name server.name.tld; # add aliases and perhaps public
…
}
Apparently "default_server" is also present in the second "server" block, I've looked into the same files before the upgrade and there "default_server" was not present in the second block:
(this is the file /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default before the upgrade)
server {
listen 443 default_server ssl;
server_name _;
…
}
server
{
listen 443 ssl http2;
server_name server.name.tld;
…
}
I've tried to modify it, but it's always regenerated so it keeps to give the error.
Maybe some configuration in zibra recreate the file?
How may I change this setting?
Andrea,
can you check if there are differences between before and after upgrade for :
/opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template
It is just a guess but that is probably the file that is used to create./opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default.
Can you try to use the file from before the upgrade and see if the server starts?
Please let me know how it goes.
Thank you,
Paul
Hi
I don’t understand how restore password (point 12) that i had saved in point 2
can you help me
thank you
Carmine
The command
zmlocalconfig -s | grep -i password
would give you all the set passwrds.
Than you can use them in step 12