By The APM is an abbreviation of Application Performance Monitoring. It monitors application code. For example, when you click on a button in an application the APM runs in the background and tracks your activity.
The APM can help us catch bugs easily. For instance, a development team that I work with had an unpredictable application behavior. They tried many ways to find what caused this bug. After two years of searching, they ask me to install SCOM APM in a production environment. It took only several days to catch the bug and figure out what caused it.
I had many successful SCOM APM cases like this one. Therefore, I was surprised to hear from my counterparts that the SCOM APM is not a favorite SCOM feature. I think that the reason is a lack of knowledge. You can destroy the monitored application if you do not know how the SCOM APM works. Therefore, a typical SCOM administrator that has a system administrator background would prefer not to implement the SCOM APM while a SCOM administrator with a developer background would probably know how to tune the APM so that it return meaningful data with low-cost effect on the application.
Having said this, this post is all about how the SCOM APM works so you could tune the SCOM APM wisely to experience the power of the SCOM APM.
It is important to understand all the different APM tools using similar resources to monitor an application code. Currently, I will focus on a DotNet application and SCOM APM tool, but under the hoods, this tool works very similarly to other APM tools.
SCOM APM uses the .Net profiling API. It is a part of .Net Framework and basically it allows you to observe the .NET runtime in action. The .NET profiling API is COM-based. Using the profiling API involves creating an in-process COM server that implements a single interface. Each interface method represents a single event from one of the following categories:
- CLR startup and shutdown
- Application domain creation/shutdown
- Assembly loads/unloads
- Module loads/unloads
- Class loads/unloads
- COM interop VTable creation/destruction
- Just-in-time (JIT) compilation and code-pitching events
- Thread creation/destruction/suspends
- Remoting activity
- Exception handling
- Managed/unmanaged code transitions
- Garbage collection and managed heap objects
- Method entry/exit
In other words, the SCOM APM uses a COM server object DLL that implements the required profiling interface. The next task is to force the CLR into loading the COM server and invoking the methods. This trick is accomplished by configuring environment variables. The SCOM APM configures these variables in the registry of the preferred application.
There are two main variables in the registry: COR_PROFILER and Cor_Enable_Profiling. These are used by CLR for loading the COM Server object. The loading process begins with CLR check of the Cor_Enable_Profiling (it can have one of the two values 1 – enable or 0 – disable loading the COM server object). If the value is 1 the CLR will attempt to load the COM server as specified in the COR_PROFILER environment variable. In the SCOM APM case, COM server object is the StubProfiler.dll, as you can see with OleViewCOMObject tool:
Once the CLR loads the StubProfiler.dll it will implement the required profiling interface and from this moment and on, every event will notify StubProfiler. The StubProflier will pass the data to the SCOM Agent where all the magic happens.
Let’s go deeper by using Sysinternals Process Explorer tool; it is an excellent tool that can help us investigate the In-Process mechanism, in this screenshot you see the SCOM Agent managed DLLs inside the w3wp.exe:
After that we understand the In-Process feature, we can go forward to investigate these DLLs managed code. For this we can use ILSpay tool, it is an excellent tool for reflection of .NET code:
I won’t go through all of what it does, although, it is important to mention that the main class is InstrumentationClass. It is responsible for the monitor functions. These functions are marked as may interest the user either for performance or exception analyze, as well as, to get all monitored function arguments and collecting event data that will be sent to SCOM APM AppDiagonstics. To mark functions the SCOM APM uses a configuration file it is a PMonitor.config file. It contains entry points, application, and all the data producers definitions. Note! A wrong configuration can cause an unpredictable behavior of the monitored application because each producer must have a proper implementation in InstrumentationClass class, as you can see here: 
After understanding this you ready to tweak the SCOM APM configuration.
In the next post, we go forward on all possible configurations in SCOM APM template, and how you can decrease the effect on the application on the one hand and the other side returns a meaningful data.
AnalyticOps Insights 