In e-business, as in any business, one key to success is to know who your customers are, what they're buying and how they behave.
Bricks-and-mortar businesses can look at the till receipts at the end of the day, and conduct customer surveys, which will tell them the second and a little of the first; the third goes largely unknown.
At the front end of an e-business, data is easily collectible. All Web servers have the ability to log streams of clicks or entered text to a data repository in real time.
At the back end, a number of business intelligence tools exist to slice and dice the data and tell you where you are succeeding and where you are not doing so well. This has been a relatively long-standing discipline with some bricks-and-mortar businesses.
It is equally applicable, if not more so, to the e-business. Here the greatest advantage, says Craig Catley of Cognos agent CDP, is that we can follow the customer through the store, identifying every click at every choice-point - particularly the points where the customer chooses not to buy. A bricks-and-mortar establishment only knows customers after they have purchased.
If the company can induce the e-commerce customer to provide their details on the Web site, a deeper knowledge of a customer is naturally possible than for a customer who walks in the door.
Data from the e-business can profitably be combined with the data obtained from other streams.
"Just because an organisation's traditional and e-business operations are distinct doesn't mean they have to be looked at separately," says Simon Lawrence, business intelligence projects manager for CDP.
A company may, of course, want to analyse the two streams of data separately, for comparison purposes. But in general, the electronic data form only part of overall information on transactions - albeit an increasingly important part.
Business intelligence tools from Cognos, he says, let an organisation see and understand the performance of all processes at the same time. This can include customer analysis from CRM (customer relationship management) data, supplier analysis from SCM (supply chain management) data, traditional ERP (enterprise resource planning) data and/or data from your e-commerce site.
All Web servers have the ability to log client interactions to files or databases in real time. This log can be forwarded to a clickstream datamart. However, the potential volume of data from the clickstream dwarfs even the largest corporate data warehouses. The aggregation capability of business intelligence tools helps keep this volume under control as well as making sense of it.
Another good start on reducing the volume is to filter out the clickstreams you're probably not interested in, like the users who stay longer than one minute.
Analysing Web data
Analysing data from a Web server is not an easy job, Lawrence says.
"The nature of the data makes it very difficult."
The main source is the system log; a text file that can vary in format from server to server. However, there are some standards evolving that could ameliorate this problem, though.
The operator of an e-business will probably be interested in sessions. A Web server could be dealing with hundreds or thousands of these concurrently. You get a lot of non-contiguous records that may or may not relate to the same session.
You can help by designing your Web site appropriately; putting tags on the pages so you know where the user is at what time. Time synchronisation is important. A single session may take the user through more than one server, and they may not have the same time.
One well-known technique of tracking users is a persistent cookie that sits on the user's PC between sessions; but some people don't like those and delete them, or refuse to accept them at all.
You end up classifying the sessions into people you know about and anonymous entities which refuse to let you know anything about them.
Many sites try to collect some kind of survey data; persuade visitors to log -in with their correct names (a large percentage of people don't). It comes down to building a trusted relationship with your customers, or resorting to using anonymous cookie system numbers.
Once you get it into (the form of conjectural sessions) then it becomes no different from using ordinary bricks-and-mortar transaction records.
E-business data critical
Anil Reddy of IBM says E-business data is really "just another kind of data" - transactional data like that which has been extracted and analysed using BIS tools for a long time in some organisations. But in a sphere so competitive and sensitive to customer whim as e-commerce, it can be critical. As more and more customers embrace e-commerce, it will be e-business data and its competent analysis that is critical to the company's competitive advantage.
Analysis of incoming transaction data can give the company using e-business a wealth of information on patterns of business; but the right tools must be there to extract and cleanse the data before putting it into some repository such as a data warehouse to perform decision-support analysis on it.
"In the past, some people tried to do an expensive data warehouse set-up, met those challenges of cleanliness and reliability of the data and simply ran out of money," Reddy says.
"But there are now lots of tools available to do that, so it's easier."
ETI, from an IBM partner of the same name, is one such tool.
There are a lot more intelligence tools attached to database management systems as well, says Reddy. DB2, for example, has an Intelligent Miner for Text, which can detect the frequency of a particular word or phrase in a database of text that might, for example, be email queries to a helpdesk.
A significant problem is the sheer volume of the data created by the interaction of users with an e-commerce site.
Reddy agrees with the Cognos representatives. As more and more end-users embrace e-commerce this problem will be exacerbated.
Once a company has set up its Web site and introduced "customer-facing software", to transact business collection and interpretation of transaction data is the valuable third step, says general manager of SAS New Zealand Rory Stoddart.
SAS has a finger in the pie at all three levels, but is particularly well known for its analysis software and products in related areas like data warehousing.
The company is about to release WebHound, a product designed to follow transactions on the Web, and determine which part of the site customers visit most often and why they abandon efforts to purchase.
An associated analysis technique, e-Discovery, combines the clickstream data with other customer data with the aim of discovering more about the customer.
This, says AS, will keep the right customers coming back, analyse the effectiveness of different channels and identify opportunities to cross-sell and up-sell.
The science of interpreting clicks and other data into a customer profile has advanced markedly in such organisations as US banks, he says.
"They know all about you once they've collected enough clicks, and every time you go in they pitch something at you; it's quite scary.
"You could never collect all this information over the counter; it would be too expensive and inefficient."
An accurate customer profile allows a business to pay more attention to its "high-value, low cost" customers.
In contrast to the US, Kiwi businesses are lagging in their application of analysis to customer transactions; "they're still at the [invention of the] wheel stage", he says.
Web page data is usually not enough, says Stoddart. The business has to bring together data from a number of sources - over the counter interactions and the call centre, for example - and translate it all from individual activities with a customer number attached, to a "subject record", which stores information about that particular customer, with his or her number as the primary key.
A lot of information is collected by CRM systems, and thanks to emerging standards, SAS routines can read the data repositories used by market leaders like Siebel and Vantive, he says.
One of the dangers of analysis is that a company collects information through from the Web, and then is "frightened to do anything with it", because it might mean too drastic a change in the way the business operates, he says.
He cites a project SAS did for a bank, trying to lift its response to offers of a new product.
"We tested our model, then applied it to the whole data population, and produced a mailing list of people we thought should be sent the offer.
"SAS' mailing list increased the bank's response rate from 4% to 10% ... you would have thought it a no-brainer to adopt the new way of doing things, but they chose to keep on doing what they had been doing.
"The only [reason] of substance they could come up with was that they would have to change what they were currently doing, which might have meant some people would have to take on new roles or even be surplus to requirements."