Thursday, July 2, 2015

Leap second bug 2015. Linux/Centos, 100% CPU: Java, Oracle, OPMN, Tomcat


Oh dear, looks like there are services having serious issues with the Leap Second added last night. Read more about Leap Second 2015 in Wiki. Fix is simple:

# service ntpd stop; date -s "`date`";service ntpd start;

or

# /etc/init.d/ntpd stop; date -s "`date`"; /etc/init.d/ntpd start;

The problem occurred on an older Java/Oracle running webserver. All CPU's went 100% high. All services that had anything to do with JVM have gone bonkers: Tomcat, OPMN, Oracle, WebCache.

At first I disabled services that where failing and where not so important, but then all the others jumped to 100% CPU. It took some minutes before the situation was clear - all stuck services had one thing in common - JAVA. Once they went down, CPU went to idle. Ones who where prepared for this day did that 3 years ago. Happy restarting all the lazy admins.

No comments: