Server monitoring recipe with SNMP: Observium + Nagios

Objective

Everyone having at least couple of servers, even a single server, would want to monitor it eventually. Some time ago I used MRTG for all that, but as the needs expanded I could do less and less with it and in the end it even became too complicated to use. MRTG is powerful, yet vulnerable to simple server restarts - you have to remap your pins.

Recipe

Will jump to it right away: the best option currently is Observium + Nagios. Will tell about the first one in a separate paragraph, it might suffer from an early death some day, but currently its a good tool for the job. I ended up using two tools because Nagios has a very good alerting system, but lacks interfaces and as you probably guessed already Observium has interfaces, but lacks alerting system.

Observium

The peckers behind this tool are pretty questionable. Some time ago they had a fundraising campaign to collect some doe and implement an alerting system. After funds were raised - they removed the promised functionality from the Community release and made it part of their paid version. Money is money, but hey, Internet knows everything. Further more I tried to communicate with them on Facebook - all my page messages and comments where removed and all PM's ignored.
Nevermind the folks, their tool is good for one thing - drawing nice charts:

Nagios

Where Observium fails - Nagios can help. Its an open source project, no need to tell more. It lacks interfaces and historical information (excluding payed plugins and extensions), but it has a powerful alerting system. Just setup a couple of users with emails and you are done:

Conclusions

Nagios allows you to receive an email in the middle of the forest when your backup drive hits a warning limit while Observium helps you analyze and plan you infrastructure, workloads and record historical events. I can now see that admin still hasnt added memory to our webserver and I asked for that a week ago.

Gmail attachment cleanup. Cleanup large gmail letters

Even though Gmail gives you quite a lot of storage, for years I have developed a habit to clean up the trash. It just feels right.

Type in "size:20000000" for filtering mails with attachments >20MB. Seach query is in bytes, filter by any size limit you want. "size:5000000" for emails with larger then 5MB attachments.

Once you find the monstrous mails - you can either delete it all or just one of the messages. If you open the conversation - large messages are always expanded. This is just great if you have some development material which is out of date, but you would still like to save the conversation for later. Just select "Delete this message" from message tools.

Cuba Libre

Here is another break from all the typing.

----------------------------------------------------
* 2 oz (50ml) white rum  Place the ice into a highball glass.
* 1 lime                 Pour over the rum. Some more rum.
* Cola                   Cut the lime in quarters. Squeeze
* Ice                    one lime in (optional). Drop all the
                         limes in. Top it up with Cola.
-------------------------------------------------------------

More Java grants on Oracle. ORA-29532 java.io.FilePermission

Abstract

I got a simple file system writer/reader, it starts with Oracle Directory alias and continues generating folders using organization number and some bits of date. Alias part is static, the rest..  ..is suppose to be generated infinitely. Not including mount, ownership and permission details, basically your main folder and subfolders have to be fully available to user running Oracle.

Short spec
Oracle Directory: /attachments/ (alias ATTACHMENTS)
Organization id: 301
Todays date monthly token: 0715
Schema in use: AWS

Error

Lets start with stack trace:

<...>

java.security.AccessControlException: the Permission (java.io.FilePermission /attachments/301/0715/19871_head.txt write) has not been granted to AWS. The PL/SQL to grant this is dbms_java.grant_permission( 'AWS', 'SYS:java.io.FilePermission', '/attachments/301/0715/19871_head.txt', 'write' )

<...>

oracle.jdbc.driver.OracleSQLException: ORA-29532: Java call terminated by uncaught Java exception: java.security.AccessControlException: the Permission (java.io.FilePermission /attachments/301/0715/19871_head.txt write) has not been granted to AWS. The PL/S
QL to grant this is dbms_java.grant_permission( 'AWS', 'SYS:java.io.FilePermission', '/attachments/301/0715/19871_head.txt', 'write' )

<...>

Possible fixes

Thing is you need write permissions in your Oracle dir, but in this case its recursive and never ending. I start with this:

BEGIN
  dbms_java.grant_permission( 'AWS', 'SYS:java.io.FilePermission', '/attachments/*', 'write' );
END;

Small bit that made me spend couple of hours was recursive Java grant, just use dash "-" instead of "*" and grant will be valid for all your subdirectories:

BEGIN
  dbms_java.grant_permission( 'AWS', 'SYS:java.io.FilePermission', '/attachments/-', 'write' );
END;

Just in case you need more then write - use full fleet of file permission types:

BEGIN
  dbms_java.grant_permission( 'AWS', 'SYS:java.io.FilePermission', '/attachments/-', 'read,write,delete' );
END;

Leap second bug 2015. Linux/Centos, 100% CPU: Java, Oracle, OPMN, Tomcat

Oh dear, looks like there are services having serious issues with the Leap Second added last night. Read more about Leap Second 2015 in Wiki. Fix is simple:

# service ntpd stop; date -s "`date`";service ntpd start;

or

# /etc/init.d/ntpd stop; date -s "`date`"; /etc/init.d/ntpd start;

The problem occurred on an older Java/Oracle running webserver. All CPU's went 100% high. All services that had anything to do with JVM have gone bonkers: Tomcat, OPMN, Oracle, WebCache.

At first I disabled services that where failing and where not so important, but then all the others jumped to 100% CPU. It took some minutes before the situation was clear - all stuck services had one thing in common - JAVA. Once they went down, CPU went to idle. Ones who where prepared for this day did that 3 years ago. Happy restarting all the lazy admins.

Java source compilation in Oracle

Its a pretty lame subject, I had very limited access to customers box and was not able to use my toys on Toad. Amazingly it took a while to get things in one place and there are a couple of problematic points along the way.

Abstract

A couple of Java source packages residing in Oracle. Need them transferred and compiled.

Compiling Java source in Oracle

ALTER JAVA SOURCE OWNER.JavaSource COMPILE;
ALTER JAVA CLASS OWNER.JavaClass RESOLVE;

After I tried to compile the sources, compile returned success, resolve though returned nothing. This means we have errors or dependencies. My Java source is called "ReceiveMail", name stands for itself..

Debugging Java source compile

Actually its very easy, you will easily find additional filters for your query. All_errors/user_errors/dba_errors tables dont have timestamp, but its not needed, they store recent errors (or warnings in case of Java source):

SELECT * FROM user_errors WHERE name = 'RECEIVEMAIL';

Compiling Java source in Oracle example

There is one more (un)expected twist, normally Oracle creates objects in UPPERCASE, just thats not the case for Java source. Standard is respected. This also led to a stupid one hour delay because I was not able to locate the source which was already compiled and valid.

ALTER JAVA SOURCE RECEIVEMAIL COMPILE;
ALTER JAVA CLASS "ReceiveMail" RESOLVE;

Dont forget the correct "ClassName" in correct case. The resolve command can actually also compile the source I did not performed a full scale analysis on when and how, but two other packages that I had got automatically compiled while I was playing with resolve command.

Use this select to see the state of you Java objects:

SELECT object_name, object_type, status FROM user_objects WHERE object_type like '%JAVA%';

You will see if there are any more filters needed, I only had 8 packages, so it was enough. After a successful compilation objects status changed to valid.

Extra grants

If you got this far and your sources are working, then thats it. I was missing one more extra grant for my schema which is called 'INVOICE', you must have appropriate permissions to run this and choose your own schema instead of 'INVOICE':

BEGIN
  dbms_java.grant_permission('INVOICE', 'SYS:java.util.PropertyPermission', '*', 'read,write' );
  dbms_java.grant_permission( 'INVOICE', 'SYS:java.net.SocketPermission', 'pop.server.com:*', 'accept, connect, resolve' );
END;

PDF page count on Linux commandline

# pdfinfo yourpdffilename.pdf | grep Pages | awk -F: '{print $2}' | tr -d '[:blank:]'

pdfinfo on Centos is provided by package poppler-utils