Thursday, July 2, 2015
Leap second bug 2015. Linux/Centos, 100% CPU: Java, Oracle, OPMN, Tomcat
Oh dear, looks like there are services having serious issues with the Leap Second added last night. Read more about Leap Second 2015 in Wiki. Fix is simple:
# service ntpd stop; date -s "`date`";service ntpd start;
# /etc/init.d/ntpd stop; date -s "`date`"; /etc/init.d/ntpd start;
The problem occurred on an older Java/Oracle running webserver. All CPU's went 100% high. All services that had anything to do with JVM have gone bonkers: Tomcat, OPMN, Oracle, WebCache.
At first I disabled services that where failing and where not so important, but then all the others jumped to 100% CPU. It took some minutes before the situation was clear - all stuck services had one thing in common - JAVA. Once they went down, CPU went to idle. Ones who where prepared for this day did that 3 years ago. Happy restarting all the lazy admins.