Spiders, robots and Internet users down-loading the entire Acts of Parliament have been causing big headaches for the Auckland-based Web site The Knowledge Basket.
The Knowledge Basket — a research and archiving site which provides information from sources such as GP Print (formerly Government Print), NZPA and Radio New Zealand — receives constant visits from spiders causing congestion and performance degradation. Spiders are software used by Internet search engines to index millions of Web pages thereby creating databases for search engine users hunting down information. Some individual users are also proving a move, with none against.
“Generally the robots and spiders from the big US directories are well behaved and I don’t have any problem with the legitimate New Zealand spiders. However there are a lot of people out there who suddenly find they can build a spider and decided to go experimenting - the number visiting us is going up all the time. The Web community is going to have to come to some sort of consensus about this. It’s especially hard for sites in New Zealand because we’re billed for every page that’s taken.”
The Knowledge Basket is also gearing up to provide access to the rest of GP Print legislation (regulations and bills) and Hansard parliamentary reports.