Companies that are fed up with sluggish processing speeds and the tedious chore of maintaining traditional databases are increasingly turning to data warehousing appliances to analyse large data sets.
The appliances, dedicated devices that include varying combinations of high-performance hardware with database, storage and other types of software, are usually preconfigured for specific tasks like strategic analysis.
Book publisher O’Reilly Media recently implemented an open-source data warehouse appliance jointly developed by Sun Microsystems and Greenplum. The appliance, unveiled late last month, includes a Solaris 10-based Sun Fire X4500 data server and Greenplum’s massively parallel distribution of the open-source PostgreSQL database.
Roger Magoulas, director of market research at O’Reilly Media, says the company plans to use the appliance to perform analysis on 1TB and 2TB data sets that the company expects will double in size every year. The data sets are mostly made up of unstructured text and “didn’t lend themselves to adequate performance with regular databases”.
With the appliance, “you can spend more time on the business reason you have the data, rather than optimising or tuning the database, because you have unlimited disk space and parallel database performance”, Magoulas says.
O’Reilly now uses the open-source MySQL database running on a cluster of machines, he says.
Doug Mackie, vice president of enterprise technology at The NPD Group, says his company has been using a data warehouse appliance from Datallegro since April as a query engine for its customers and to process point-of-sale data. The market research firm has also formed a so-called sandbox test bed to evaluate other areas where the appliance could be used, Mackie says.
With the appliance, NPD was able to reduce the time needed to process some queries from hours and days down to minutes and seconds, Mackie says. Previously, the company completed the queries on an Oracle database.
“It is a single-minded device and it just runs the queries,” Mackie says. “We think we will see efficiency and speed and that DBA activity requirements will be greatly reduced from an appliance versus continuing with the relational databases.”
However, he notes that while the appliances have “great potential”, the technology is relatively new and the vendors are inexperienced.
Analyst firm IDC estimates that the appliance market will reach US$500 million (NZ$780 million) in five years from US$50–$75 million today.
Mark Beyer, an analyst at Gartner, says companies are buying the appliances mainly because they are preconfigured and therefore easier to put into production than traditional databases. However, he cautions that an appliance tuned for a specific use “will fail in some cases, and in other cases it will have performance problems” if it’s used for a different application.