Thursday, January 31, 2008

Best Practices for Active Response Time Monitoring

At OpenWorld, I was asked about the proper way to set up synthetic transactions for monitoring applications. It was a good question, and I wanted to document my answer in some sort of whitepaper or technote. So far I still haven't gotten around to writing the formal document, so I am going just post it on this blog. Perhaps I could evolve this into the actual document.

As I discussed in the post “Response Time Monitoring - Real User vs. Synthetic”, there is a place for both real user and synthetic monitoring of applications. There are several challenges in using synthetic transactions, however, and these challenges are not unique to Oracle Enterprise Manager. You would have to consider them no matter which tool you use.

First, unless carefully designed, the tests may not be representative of actual end user activities, reducing the usefulness of the measurements. Therefore, you must be very careful in defining those tests. It would be a good idea to sit down with real users to observe how they use the applications. If the application has not been launched, work with the developers, or if there is one, the UI interaction designer to define the flow. In addition, work with your business sponsors to understand where the application will be used and the distribution of user population. You would want to place your synthetic test drivers at locations where it is important to measure user experience.

Second, some synthetic transactions are very hard to create and may introduce noise into business data. While it is usually relatively easy to create query-based synthetic transactions, it is much harder to create transactions that create or update data. For example, if synthetic transactions are to test for successful checkouts on an e-commerce website, the tests must be constructed carefully so that the test orders are not mis-categorized as actual orders.

To mitigate these potential problems, you should set up dedicated test account(s) to make it easier to tell whether something running on the application came from real users or the synthetic tests. For operations that involve changing data, determine ways to exclude those data from your reports. If it is possible, look for ways to purge the test data out of the system. It is not always possible or easy to do this, as some business processes do not allow changes to data after a certain point. If you are working with a custom application, consider building a “test mode” into the application to make it easier to roll back changes.

Third, security and authorization policies might impact the tests as well. You need to make sure that whatever test application user account has the proper access privilege to access the application elements to be tested. If authorization policies change, you need to verify to make sure that the tests are not affected. The same kind of consideration applies to passwords as well. If you are required to change passwords due to password aging policies, you need to make sure that those changes are reflected in your test setup.

Fourth, synthetic tests may introduce load into your application, so be judicious when setting up test frequency to avoid overloading your application. This means that you may not want to just use all your functional or load test scripts for production monitoring. These scripts were created for different purposes – testing functionality, and stress testing the application, and they may be an overkill for what you need to do to just test the application enough so that you can tell whether key operations are working.

Lastly, make sure your monitoring scripts log out of the application at the end of execution. This is especially important for applications that maintain some sort of session state on the mid-tier. If you do not log out, the resources would not be freed up in a timely manner, and this may impact the scalability of the application. By the same token, be sure to allocate resource to account for test connections on top of connections made by regular users.


Tuesday, January 8, 2008

Response Time Monitoring - Real User vs. Synthetic

Response time monitoring is a very important aspect of application management. In a way, it is nothing new. People have been monitoring response time ever since the days of green screen mainframe terminal applications. The only variable is the technologies involved, both on the application and the tools for monitoring them.

When I speak with customers, I sometimes get these questions about the pros and cons of monitoring response times of real user vs. synthetic transactions, also referred as passive vs. active monitoring, and whether one approach should be used over the other. The truth is both approaches are relevant and they complement each other. Here's why.

Real user monitoring is obviously important because it measures the actual experience of actual end users. Despite of its appeal, however, it is not a one size fits all solution to measure the performance of an application. First, there can be a lot of noise in the actual end user response time data, which may make it difficult to determine the relative performance of an application over time. Usage pattern can vary a lot at different times, and the performance of transaction requests can vary a lot depending on the data that are processed. Consequently, we may end up with a lot of apple vs. oranges when trying to compare response time measurements. Second, real user monitoring only works only when there are real users on the system. For example, if there is no end user on the system at 2 a.m., data collected from real user monitoring won't tell if the application is working or not.

On the other hand, synthetic transactions can be used even when real users are not around. Because the tests are well defined and their executions are controlled, it is also easier to do long term trending analysis to see if application performance has improved or degraded over time. Synthetic transactions do have their own shortcomings, however. First, unless carefully designed, the tests may not be representative of actual end user activities, reducing the usefulness of the measurements. Second, some synthetic transactions are very hard to create and may introduce noise into business data. While it is usually relatively easy to create query-based synthetic transactions, it is much harder to create transactions that create or update data. For example, if synthetic transactions are to test for successful checkouts on an e-commerce website, the tests must be constructed carefully so that the test orders are not mis-categorized as actual orders.

As you can see, there are tradeoffs to both approaches, which are complementary. It is not real user vs. synthetic, it should be real user and synthetic.