Larry Ellison on self

“When you write a program for Android, you use the Oracle Java tools for everything, and at the very end you push a button and say, ‘Convert this to Android format.'” - Larry Ellison, 2013, CBS interview.

Oracle 10g on Centos 6.7 - x86 issues: ntcontab.o, snmccolm.o, ORA-27125

Abstract

Its getting harder and harder to install good old 10g on newer Centos versions. Last attempt on 6.5 was semi-problematic, with some extra packages missing, 6.7 deployment was even more challenging.

System specifications

Centos version (/etc/issue): CentOS release 6.7 (Final)
Oracle version: Version 10.2.0.1.0 Production (10201_database_linux_x86_64.cpio)

Problems and solutions

#1: Error invoking target 'ntcontab.o' of makefile

This error occurred around 65% in installation progress, aborting is not an option - more errors will follow and in the end whole process fails. Did some testing and it appears that one process is building the file, next one is instantly deleting it:

# cp /misc/oracle/product/10.2.0/db_1/lib32/ntcontab.o /misc/oracle/product/10.2.0/db_1/lib/

But in the end it looks like it was one of the following (where not needed in 6.5):

# yum install libaio-devel.i686 -y
# yum install zlib-devel.i686 -y
# yum install glibc-devel -y
# yum install glibc-devel.i686 -y
# yum install libaio-devel -y
# yum install ksh -y
# yum install glibc-headers

#2: /misc/oracle/database/product/10.2.0/db_1/sysman/lib/snmccolm.o: could not read symbols: File in wrong format

It might be that it may be solved with some more x86 packages thrown into the pile, I was not able to find the exact culprit. Many of the sources online simply telling to ignore this and fix it with 10.2.0.4 patch. Thats what we'll do: Ignore.

#3: ORA-27125: unable to create shared memory segment

Reason is unknown, it was thrown by DBCA with continuous installation process.
The solution is very simple, first check the oracle user group information:

[oracle@storage] $ id oracle 
uid = 500 (oracle) gid = 502 (oinstall) groups = 502 (oinstall), 501 (dba) 
[oracle@storage] $ more /proc/sys/vm/hugetlb_shm_group 
0 

Execute the following command as root, the dba group is added to the system kernel:

[oracle@storage] $ echo 501 > /proc/sys/vm/hugetlb_shm_group

Continue with DBCA, step will fail, but then retry DBCA, the problem disappeared and database was created.

#4: bonus problem, not Centos 6.7 related: You do not have enough free disk space to create the database

My bad was that I had 12TB storage mounted, looks like I missed the 10g storage requirement: 400MB, but less then 2TB. Found no other solution then to unmount /dev/sdb1, shrink it with gparted to get ~500GB space, ext4 it and mount the new device /dev/sdb2 to another mount point /oracle.

Conclusion

Move to 11i or 12c, its about time.

Server monitoring recipe with SNMP: Observium + Nagios

Objective

Everyone having at least couple of servers, even a single server, would want to monitor it eventually. Some time ago I used MRTG for all that, but as the needs expanded I could do less and less with it and in the end it even became too complicated to use. MRTG is powerful, yet vulnerable to simple server restarts - you have to remap your pins.

Recipe

Will jump to it right away: the best option currently is Observium + Nagios. Will tell about the first one in a separate paragraph, it might suffer from an early death some day, but currently its a good tool for the job. I ended up using two tools because Nagios has a very good alerting system, but lacks interfaces and as you probably guessed already Observium has interfaces, but lacks alerting system.

Observium

The peckers behind this tool are pretty questionable. Some time ago they had a fundraising campaign to collect some doe and implement an alerting system. After funds were raised - they removed the promised functionality from the Community release and made it part of their paid version. Money is money, but hey, Internet knows everything. Further more I tried to communicate with them on Facebook - all my page messages and comments where removed and all PM's ignored.
Nevermind the folks, their tool is good for one thing - drawing nice charts:

Nagios

Where Observium fails - Nagios can help. Its an open source project, no need to tell more. It lacks interfaces and historical information (excluding payed plugins and extensions), but it has a powerful alerting system. Just setup a couple of users with emails and you are done:

Conclusions

Nagios allows you to receive an email in the middle of the forest when your backup drive hits a warning limit while Observium helps you analyze and plan you infrastructure, workloads and record historical events. I can now see that admin still hasnt added memory to our webserver and I asked for that a week ago.