Tuesday, January 8, 2008

Response Time Monitoring - Real User vs. Synthetic

Response time monitoring is a very important aspect of application management. In a way, it is nothing new. People have been monitoring response time ever since the days of green screen mainframe terminal applications. The only variable is the technologies involved, both on the application and the tools for monitoring them.

When I speak with customers, I sometimes get these questions about the pros and cons of monitoring response times of real user vs. synthetic transactions, also referred as passive vs. active monitoring, and whether one approach should be used over the other. The truth is both approaches are relevant and they complement each other. Here's why.

Real user monitoring is obviously important because it measures the actual experience of actual end users. Despite of its appeal, however, it is not a one size fits all solution to measure the performance of an application. First, there can be a lot of noise in the actual end user response time data, which may make it difficult to determine the relative performance of an application over time. Usage pattern can vary a lot at different times, and the performance of transaction requests can vary a lot depending on the data that are processed. Consequently, we may end up with a lot of apple vs. oranges when trying to compare response time measurements. Second, real user monitoring only works only when there are real users on the system. For example, if there is no end user on the system at 2 a.m., data collected from real user monitoring won't tell if the application is working or not.

On the other hand, synthetic transactions can be used even when real users are not around. Because the tests are well defined and their executions are controlled, it is also easier to do long term trending analysis to see if application performance has improved or degraded over time. Synthetic transactions do have their own shortcomings, however. First, unless carefully designed, the tests may not be representative of actual end user activities, reducing the usefulness of the measurements. Second, some synthetic transactions are very hard to create and may introduce noise into business data. While it is usually relatively easy to create query-based synthetic transactions, it is much harder to create transactions that create or update data. For example, if synthetic transactions are to test for successful checkouts on an e-commerce website, the tests must be constructed carefully so that the test orders are not mis-categorized as actual orders.

As you can see, there are tradeoffs to both approaches, which are complementary. It is not real user vs. synthetic, it should be real user and synthetic.

1 comment:

Prasad Chitta said...

An old blog post of me on the same subject...