6 commandments to effectively monitor your packaged application
The common denominator for applications in the SCM space is the fact that they are packaged, n-tiered, comprised of the application deployed in clustered and load-balanced containers, interface heavily with the external applications, and a database to persist all the transactional data.
Given the nature of these applications everything is hunky-dory if the application is running well and there isn’t any problem. But once a screen starts rendering slowly or one of the asynchronous interfaces starts accumulating messages or an application container crashes randomly everything goes haywire.
Is it the Big IP that is not load balancing well, is there a memory leak at the application server level, is it a database problem, are there deadlocks in the application server JVMs, is it network related, is there a resource contention (CPU and memory) or could it be something inherent to the packaged application. A big O Hullabaloo indeed!!!
I often see that due to un-prepped production environments any not-so-apparent problem can take a while to diagnose. Emergency tickets are raised, production support teams are involved and it goes into an eternal abyss of inter-team correspondence.
I thought of writing about these elemental logs which, if enabled, will save you a ton of time by catalyzing the troubleshooting process. Here is the list of logs in no particular order:
GC Logs: Will help you determine the way heap is consumed as the objects are instantiated and garbage collected. It will also help you determine which phase of Garbage Collection is taking the most amounts of time and the overheads thereof. It will also help you determine Memory Leak patterns. Observing the GC logs you can notice if duration of GC is a bottleneck to the throughput/response time.
Heap Dumps: Will help you compile a list of data structures (particularly the notorious collections) that could be prospects for memory leaks. It will also help you determine if the heap is fragmented (great help if you are running JDK/JRE 1.4.2 and below).
Java Core Dumps: You diagnose deadlocks if any using the java core dumps. In many of the OMS implementations it is very helpful if the order lines are not being fetched in a sorted order. You can also get a summary of different types of data structures you are using in the code and the space
DB Reports: Will help you determine long running queries, bad query plans adopted by the DB, the wait time involved in reading from/writing to the physical disks (especially if your storage layer is virtualized), database level locks all of which affects the performance significantly.
Application Logs: In the packaged s/w world it is always important to have application logs to help you determine any issues that are inherent to the packaged app.
Resource Monitoring Logs: You must have an understanding of the CPU and Memory utilization. If the CPU utilization spikes without any increase in the load, it could be because you are running out of memory. You can then focus on the utilization at the process level which will help you determine which JVM suddenly spiked its CPU utilization. You must then take a look at the GC log and Heap dump for that JVM for more information.
If you follow these 6 commandments, regardless of your religious origin, you will be able to localize the defect very quickly which is 90% of the work in any troubleshooting process especially in production environments and you are on the hot seat.
