Authentication Bypass in MailboxImportServlet vulnerability (reminder)
https://blog.zimbra.com/2022/08/authentication-bypass-in-mailboximportservlet-vulnerability/

Huge CPU Java

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
weblike
Posts: 27
Joined: Sat Sep 13, 2014 3:08 am

Huge CPU Java

Postby weblike » Fri Sep 23, 2022 1:01 pm

I have a 8.8.15 P33 installation and experiencing a huge CPU usage

Screenshot_11.png
Screenshot_11.png (6 KiB) Viewed 2727 times


Does anyone experiencing this?
Any solution would be much appreciated.


Se7enPRO
Posts: 6
Joined: Tue Sep 27, 2022 11:52 am

Re: Huge CPU Java

Postby Se7enPRO » Tue Sep 27, 2022 12:01 pm

weblike wrote:I have a 8.8.15 P33 installation and experiencing a huge CPU usage

Does anyone experiencing this?
Any solution would be much appreciated.

Hey weblike. I've been experiencing this problem for several months now and can't find a solution on my own. For all available logs stored along the /opt/zimbra/log path, they do not give a direction vector in the search for a solution. Judging by the search for this thread, the problem is systematic, and there are many topics where the java process consumes 100 percent or more of server resources, and no one offers an unambiguous solution, even developers.
If you happen to find a solution, let me know. ;)
weblike
Posts: 27
Joined: Sat Sep 13, 2014 3:08 am

Re: Huge CPU Java

Postby weblike » Wed Oct 19, 2022 7:39 pm

This solution worked for me. I have followed the topic for large deployments https://wiki.zimbra.com/wiki/Performanc ... eployments

After applying recommendations in this article, users does not experience issues anymore.

what I want to optimize further is MySql performance (I see CPU stress from MySQL frequently) ...For this case I want to try this script https://github.com/major/MySQLTuner-perl (I mention that I still study this part). Further more I don't want to make many changes on the server at once...I want to give a time for the first phase and to see how user a experiencing the usability of the web-client.

Hope this helps
Se7enPRO
Posts: 6
Joined: Tue Sep 27, 2022 11:52 am

Re: Huge CPU Java

Postby Se7enPRO » Thu Oct 20, 2022 2:04 pm

weblike wrote:This solution worked for me. I have followed the topic for large deployments https://wiki.zimbra.com/wiki/Performanc ... eployments

After applying recommendations in this article, users does not experience issues anymore.

Thank you for keeping up to date. Please tell me, from all the points of that article, will it be enough to apply the "Recommended Options" settings from the "JVM Options" section? In order to reduce the consumption of java.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2520
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 8.8.15 Network Edition
Contact:

Re: Huge CPU Java

Postby L. Mark Stone » Fri Oct 21, 2022 12:59 pm

Se7enPRO wrote:
weblike wrote:This solution worked for me. I have followed the topic for large deployments https://wiki.zimbra.com/wiki/Performanc ... eployments

After applying recommendations in this article, users does not experience issues anymore.

Thank you for keeping up to date. Please tell me, from all the points of that article, will it be enough to apply the "Recommended Options" settings from the "JVM Options" section? In order to reduce the consumption of java.


Much of the data in that article is for older versions of Zimbra, and a number of the tuning suggestions are now the default.

I think you would be better off looking for the root cause, rather than just trying some changes from that article, many of which are specific to older versions of Zimbra.

So first, let's see what the Garbage Collector is doing. Please post 20 concurrent lines from the output of the following command run as the Zimbra user:

Code: Select all

tail -f ~/log/zmmailboxd.out | grep -v "at \|PROCESSING\|native formatter failure\|Broken pipe"


Next, please run "atop" as the root user and attach a screencap.

Lastly, please tell us how many mailboxes you have, approximately how big the mail store is, and the output from this command, also run as the Zimbra user:

Code: Select all

cat ~/conf/my.cnf | grep innodb_buffer_pool_size


That should be enough to get us started.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/
Se7enPRO
Posts: 6
Joined: Tue Sep 27, 2022 11:52 am

Re: Huge CPU Java

Postby Se7enPRO » Fri Oct 21, 2022 2:02 pm

L. Mark Stone wrote:So first, let's see what the Garbage Collector is doing. Please post 20 concurrent lines from the output of the following command run as the Zimbra user:

Code: Select all

tail -f ~/log/zmmailboxd.out | grep -v "at \|PROCESSING\|native formatter failure\|Broken pipe"


Next, please run "atop" as the root user and attach a screencap.

Here is the result of this command both under Zimbra user and root user
Image

L. Mark Stone wrote:Lastly, please tell us how many mailboxes you have, approximately how big the mail store is, and the output from this command, also run as the Zimbra user:

Code: Select all

cat ~/conf/my.cnf | grep innodb_buffer_pool_size

Number of mailboxes: 560
Mail storage size: 276 GB

Code: Select all

zimbra@zbs:~$ cat ~/conf/my.cnf | grep innodb_buffer_pool_size
innodb_buffer_pool_size        = 1240429363

OS version on the server:

Code: Select all

root@zbs:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

Zimbra Version:

Code: Select all

zimbra@zbs:~$ zmcontrol -v
Release 8.8.4.BETA.1158.UBUNTU16.64 UBUNTU16_64 FOSS edition.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2520
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 8.8.15 Network Edition
Contact:

Re: Huge CPU Java

Postby L. Mark Stone » Fri Oct 21, 2022 2:21 pm

You are running an older version of Zimbra with known security exploits. I would recommend you immediately perform a side-by-side migration to a current 8.8.15 version of Zimbra, ideally on a newer operating system.

FWIW, the tail command of zmmailboxd.out as wriiten only works as the Zimbra user, and what I was looking for was lines like this:

Code: Select all

[506804.442s][info][gc] GC(45684) Pause Young (Prepare Mixed) (G1 Evacuation Pause) 5863M->1200M(8192M) 42.326ms
[506808.598s][info][gc] GC(45685) Pause Young (Mixed) (G1 Evacuation Pause) 3268M->1134M(8192M) 36.974ms
[506820.899s][info][gc] GC(45686) Pause Young (Normal) (G1 Evacuation Pause) 5882M->1223M(8192M) 47.275ms
[506821.063s][info][gc] GC(45687) Pause Young (Concurrent Start) (G1 Humongous Allocation) 1301M->1104M(8192M) 19.245ms
[506821.063s][info][gc] GC(45688) Concurrent Mark Cycle
[506821.517s][info][gc] GC(45688) Pause Remark 1231M->1231M(8192M) 34.825ms
[506821.770s][info][gc] GC(45688) Pause Cleanup 1320M->1320M(8192M) 0.383ms
[506821.777s][info][gc] GC(45688) Concurrent Mark Cycle 713.174ms


I need about 20 of those kinds of lines. Alternatively, you could run:

Code: Select all

tail -f ~/log/zmmailboxd.out | grep  "\[info\]\[gc\]"


But if your CPU is pegged inexplicably, I suspect your server has been exploited.
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/
Se7enPRO
Posts: 6
Joined: Tue Sep 27, 2022 11:52 am

Re: Huge CPU Java

Postby Se7enPRO » Fri Oct 21, 2022 2:37 pm

I have a high CPU consumption of a java process after some time. And when I notice this through "htop", I highlight this process, press F9 and then Enter. After a while, consumption returns to normal. :)
I promise to provide you with the output of that command as soon as consumption rises again.
Regarding migration: Where can I find step-by-step instructions for correctly migrating to a new version with a complete transfer of mailboxes ?
And if there is a suspicion that I might have been hacked, would it be enough to change the passwords for the database ? This refers to the "mysql_root_password" and "zimbra_mysql_password" parameters.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2520
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 8.8.15 Network Edition
Contact:

Re: Huge CPU Java

Postby L. Mark Stone » Fri Oct 21, 2022 2:58 pm

If your server has been breached with the current exploit, the attacker likely already has root shell access. You need to move a current supported and patched version of Zimbra ASAP. Data loss is possible.

There are plenty of guides as to how to do this with the OSE version of Zimbra. For example, you can use IMAPSYNC to copy the emails over, and then export the user's calendars and contacts. and import them into the new server.

You can also use for example:

https://rants.tech/migrating-opensource ... -downtime/

or

https://syslint.com/blog/tutorial/zimbr ... ct-method/
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/
Se7enPRO
Posts: 6
Joined: Tue Sep 27, 2022 11:52 am

Re: Huge CPU Java

Postby Se7enPRO » Fri Oct 21, 2022 3:06 pm

I understand all the responsibility and risks, thanks for the links to the manuals. ;)
Regarding checking the Garbage Collector: as soon as there is a high CPU load, I will report the results in this thread.

Return to “Administrators”

Who is online

Users browsing this forum: Bing [Bot] and 43 guests