Do you find that despite the small fortune you have spent on products to manage your servers, networks and applications, users still complain about the performance, usability and availability of key business applications?
If so, it might be time to expand your performance-management horizons to include the end-user experience. Traditional monitoring and management tools focus on factors that impact but are blind to the actual end-user experience. While they are excellent at measuring application availability, they provide no insight into real application performance from the perspective of the end-user. And what is that perspective? As far as end-users are concerned, an application is “not performing” whenever it does not work as expected — when it’s slow, when it’s continually spitting out error messages or when the user interface is so counter-intuitive the only option is to create a workaround.
A focus on the end-user is the key to driving adoption and efficient and effective use of your critical business applications. This process starts with deciding on your approach. There are three basic types of end-user experience management products: scripted synthetic agents, passive network appliances and passive client agents.
Scripted synthetic monitoring works by installing an agent on a PC to monitor a client-side application. The agent executes a script that emulates the actions of a user. The script can do anything that a real user can do, and you can have a script do things that you would not expect a real user to do. A script can run through 100 key transactions at 5am every day and verify that the system is working before the real users show up.
Despite the flexibility of synthetic agents, there are some downsides. First, someone has to develop and maintain all of those scripts. The time and effort required to maintain (and test) these scripts means that this approach is best suited to verifying the most important transactions in the most important applications. Also, the synthetic transactions create a maintenance task to reverse the effects of the transaction. But, most critically, while the synthetic transaction measures the performance delivered to the scripting engine, it doesn’t measure the actual end-user experience. So, at best, it is a proxy for the end-user experience.
Network appliances that measure end-user experience attach to the same switch that supports the appliances used to balance the load for web servers. Since they attach to a mirror or spanned port on the switch these appliances are able to see all of the web traffic that flows in and out of each switch. Custom hardware and some very sophisticated software crack into the web and network protocols, and allow you to see any transaction that is defined to the appliance. Most of these appliances focus upon the web layer of an application system, and therefore focus upon HTTP/HTTPS. All of these products allow you to see how many transactions of interest are occurring. You can see the response times of those transactions, any errors that show up in the HTTP protocol stream, and the path that your users are taking through your web-based applications.
The great advantage of this approach is the non-invasive nature of the network appliance, coupled with complete coverage for those web-based applications and transactions you define to the appliance. The fact that no software is required on the client-side, means that this approach is ideal for business-to-commerce web applications.
Jonas Hirshfield, director of infrastructure development at web education tool provider Blackboard, points out the value of such an approach: “As a hosting provider, we needed insight into how our product was delivered to our customers — beyond how fast disk drives were spinning, and CPU and memory utilisation. We use Coradiant to understand how users interact with the software and how the environment responds to them. TrueSight gives us the ability to quickly show a quantitative report that indicates performance issues that are most often outside of the Blackboard network. That is really beneficial to the team.”
A factor that can limit the value of the appliance approach is the effort required to configure these devices to recognise the applications and transactions of interest amid all the web traffic flowing through the switch. And since there is no client-side insight, the user experience is measured only from the perspective of the web server and the load balancer, not the actual end users themselves.
Many companies balance the pros and cons by using a variety of tools, which is the strategy deployed by Meijer, a grocery and general merchandise retailer.
“The end-user experience has always been the Holy Grail,” says Tim Osbeck, director of ITS e-commerce at Meijer.
“We use a combination of Compuware Vantage scripted robots and a network probe to give us quantifiable data for the best of both worlds. It’s data that everyone believes, so we can have a common dialogue. It’s the one truth on how applications perform here at Meijer.
Passive client agents are installed on the end-user’s desktop and capture a record of their experience. Typically, the agent works in tandem with a profile or a template for each application to be monitored (whether it’s a “fat” Win32 application or Internet Explorer). The advantage of a client agent is that it is the only approach that enables monitoring of the actual end-user experience from the perspective of users themselves. The constraint of this approach is that, for global coverage, agents have to be installed on every user’s desktop.
Therefore, this approach works best when the users are employees, partners or customers. This approach is just not a fit for large-scale business-to-consumer applications, which are better served by the network appliance approach. The depth of information these products deliver typically means that they will be focused on an organisation’s most significant applications — not on all the activity occurring on the end-user’s desktop.
The combination of the user-base coverage and application depth provided by this approach is tough to beat. More sophisticated client agents can expand the perspective of the user experience beyond transaction response times and capture usage metrics, user errors and system errors. With this information, companies can manage the performance achieved by end-users, as well as the experience they get.
This is also the only technique capable of capturing real user interaction with the application before, during and after the transaction.
British Telecom is one company that leverages a passive client agent for end-user experience monitoring.
“We had been looking to deploy a response time monitoring tool when we evaluated Knoa EPM, which monitors the complete user experience with a passive agent approach,” says Stuart Smith, director of CRM application performance at BT.
“The insight into actual end-user experience gives us benefits across the board, helping target response time issues, solve end-user problems faster, identify usage and adoption issues, and even make priority decisions on ongoing application investments.”
According to a recent Forrester Research study, 67% of the surveyed organisations are more focused on the end-user experience than they were two years ago. If you’re not looking at the potential of this application it might be time to expand your performance management horizons to include the end-user experience.