Sportingbet decides against Hadoop to analyse 60Tb of data
- 19 January, 2013 08:18
Online gaming site Sportingbet has decided against using popular big data analytics software Hadoop in favour of technology provided by British-born SME Logscape, in order to analyse up to 60 TB of data within one year.
Computerworld UK spoke to Andrew Coates, enterprise integration architect at Sportingbet, who explained that the centralised approach to data analysis using Hadoop wasn't suited for monitoring Oracle Coherence, a data grid system, and needed a tool that could monitor problems at the end-point.
Coates said that Logscape allows Sportingbet's IT team to run queries on vast amounts of data, without the need for expensive storage and costly, process-heavy infrastructure - which wouldn't be the case had it opted for Hadoop.
"The way it works is that you have a Logscape server running on dedicated hardware. Then you have a little agent running on every single machine you want to monitor, which means when you run a query, it runs the query on every single machine that you have told it to run it on," he said.
"Each agent on each machine knows what the log files are, it's indexed them, and then you run a query on it and it returns the necessary data back to the server. The server then aggregates across all the results."
He added: "Other tools tend to move your logs off and onto a centralised repository. The problem with that is that you need a lot of storage and then you need a big processing machine to crunch that data. Logscape is quite clever, it leaves the data on each of the servers."
Coates explained that his IT team could run a query that could tell if there had been any errors on any of the systems integrated with Logscape simply by limiting the parameters to a specific time period and typing the word 'error' on Sportingbet's production system. He said: "It will run on every single machine, hitting every single file, to see if that word has appeared in any of the log files."
It also enables Sportingbet to quickly analyse what has changed when a new code release takes place.
Coates said: "We can actually say this release put an additional load somewhere else where we weren't expecting. We are able to work out why and qualify it to see whether it is a legitimate load or whether someone has done something wrong and a new patch release is needed to improve the way we are using our systems."
He went on to explain why Hadoop wasn't suitable for Sportingbet's environment.
"So, it's not Hadoop, or that kind of product. Personally I'm not a big fan of Hadoop. Although it does have its applications, this is more your standard DevOps monitoring of a system. It also monitors the host, so will tell you the CPU, the memory use of the disk, the network IO of the host," said Coates.
"I just don't think Hadoop is the right direction [for us]. You need a big dedicated infrastructure to run it and it's the wrong processing pattern. Coherence, for example, you have the data and you pass the algorithm to the data - because the algorithm is tiny and that data is big - leave the data where is and leave the algorithm to it."
He added: "Logscape follows a similar pattern. It leaves the data on each storage node and you run the processing algorithm on the data. That's much better, especially as datasets continue to grow and grow."
Although Logscape's monitoring was initially for Sportingbet's implementation of Oracle Coherence, it is now also looking to use it for the company's IAS logs, which will allow it to see what its users are doing on the front-end website.
One of the main benefits for Sportingbet is that it is now able to better analyse its bandwidth costs and also optimise the functionality on its website.
"It's allowed us to work out and rationalise where our bandwidth costs are going. With Logscape we are trying to bring bandwidth costs down and identify where we are spending the most money. Traditional websites work by going to the website and you pay for the bandwidth of downloading that content," said Coates.
"These days everything is polling for updates or pushing information out, so identifying which business features and functionalities are costing us is really useful. We can now say that if we have someone sat on our in-play site, it is going to cost us this many pence a month."
He added: "It also allows us to optimise the site and allows us to come up with roadmaps in terms of what is going to save us the most money first. It allows us to say: If we bring in this new functionality, it will bring in this much revenue, but it is also going to cost us X amount. It's very useful to have that kind of transparency."
Sportingbet will now integrate Logscape monitoring into every new application built, and is retrofitting it into key legacy applications. Coates said that he also considered tools from RTView and Splunk, but these were more costly and as a result lost out. He added that Logscape has 'repaid itself many times over'.
However, despite the numerous benefits, Coates did warn that the change in culture required was Sportingbet's biggest challenged with Logscape. This is because the company's developers were typically using standardised dashboards and heavily structured analytics - whereas, Logscape requires a proactive approach to running queries.
"It was a challenge getting people to adopt it. It was something that we didn't really have before, because we previously had pretty static monitoring. Our dashboards were pre-defined and the Ops team knew how to use them and what they meant. But with Logscape you can dive into the data," said Coates.
"What Logscape allows you to do is if something goes wrong, or it is not performing as you expect, you can then open up that dashboard, cross-correlate between two or more searches, and then you can open up a search and customise it. Zoom into specific bits of data etc. There is a learning curve with any product, but it takes a while for people to realise they can do all these things."
He finished by saying: "I have been in IT for 15 years and there are two things developers don't like doing: documentation and monitoring. Getting them to develop their applications alongside the monitoring, or monitoring alongside the applications, is a definite mind-switch. But what it does is it means they suddenly start to think about monitoring, which means they develop applications that are more monitorable."
Computerworld UK recently reported how Sportingbet is in the process of scrapping its polling architecture in favour of a push-based architecture, which will also help to reduce bandwidth and provide its customers with a better experience when gambling on live events.
Microsoft misjudges customer loyalty with kill-XP plea
Education ministry gets new CIO
Facebook coughs up $19bn to buy WhatsApp, draw younger users
Telecom to change name to Spark
Nov'IT says flashing a new ROM onto your Android phone can make it more secure