Scalability and Performance

The management of structured and unstructured content requires a platform that can meet the most rigorous performance requirements and be easily resized commensurate to business needs. IDOL scales to support the largest enterprise-wide and portal deployments in the world, with presence in virtually every vertical market. Since IDOL's scalability is based on its modular, distributed architecture, it can handle massive amounts of data on commodity dual-CPU servers. For instance, only a few-hundred entry-level enterprise machines are required to support ChoicePoint's 10 billion record footprint. By comparison a competitor uses 150,000 machines to handle the same amount of data.

A single IDOL engine can:

This enhanced scalability results in hardware cost-savings as well as the ability to address larger volumes of content. Though IDOL scales extremely well on commodity servers, its flexible architecture can take full advantage of massive parallelism, SMP processing capabilities, 64-bit environments (such as Intel Itanium 64-bit architecture), software platforms (such as Solaris 10, Linux 64, Win64, etc), distributed server farms, and all common forms of external disk arrays (i.e. NAS, SAN etc) to further improve performance. This flexibility extends to being able to leverage one or a combination of these different environments.

How It Works

Content from various repositories is aggregated by connectors and then indexed into the IDOL Server or for dissemination across multiple IDOL Servers, through the Distributed Index Handler (DIH). The DIH can efficiently split and index copious quantities of data into multiple IDOL Server instances, optimizing performance by batching data, replicating all index commands and invoking dynamic load distribution. The DIH can perform data-dependent operations, such as distributing the content by date, which allows for more efficient querying. Performance is augmented by the Distributed Action Handler (DAH), a distribution server that allows the user to distribute action commands, such as querying, to IDOL Servers. Multiple copies of IDOL Servers to which the DAH propagates actions further ensure uninterrupted service in the event of server failure. For flexibility, both the DAH and the DIH can be configured to run in mirroring mode (IDOL Servers are exact copies of each other) and non-mirroring mode (each IDOL Server is configured differently and contains different data). In addition, the Distributed Service Handler (DiSH) component allows effective auditing, monitoring and alerting of all other Autonomy components.

Linear Scalability

Performance and capacity can be doubled by simply replicating the existing machine. This allows scaling predictions to be made without worrying about bottlenecks.

Load Balancing

Data is automatically replicated across multiple servers and user requests are load-balanced across these replicas, guaranteeing performance, reducing latency and improving user-experience.

Mirroring / Failover

Automatically generated replicas are used to provide a pool of servers, the primary resource is automatically selected and the system switches to secondary systems if it fails so that service continues uninterrupted.

Distribution

For organizations that are geographically distributed, local replicas are automatically created and utilized where possible. Remote copies are only used when a local system fails, thereby building fault tolerance, the benefits of local performance and a reduction of resource overhead into a single, seamless service.

Adaptive Probabilistic Concept Caching

Frequently used concepts are maintained in memory and query results are returned as quickly and efficiently as possible.

Multi-dimensional Index & Query Throttling

By using a multi-dimensional index to provide valuable information to the distribution components, IDOL precludes bottlenecks and unbalanced peak loads during the indexing and query process.

Autonomy provides prioritized throttling based on:

Intelligent Repository Storage

Autonomy allows for the scheduled or discrete transfer of assets from higher stages (powerful, expensive hardware) to lower stages (cost-effective hardware) as the importance of assets declines over time. This potentially maximizes the value of information by making it available through more adequate hardware for a period of time, then again maximizing the existing investment in hardware by archiving information to machines better suited to that task.

Context and Concept Sensitive Indexing

Autonomy prioritizes the indexing queue using all contextual information, including user profiles and their usage of content. IDOL is aware of where the most important content is stored based on intelligent indexing, and performance is optimized at querying time.

Dynamic Storage Structures and Resource Sensitive Mode

Autonomy's distribution components are self-moderating and automatically re-distributes components based on peak hours and server. As an example, the DIH would not add new content to a full server but only send updates.

Advanced Data Transmission

In the information management context, data such as queries and results are travelling around the enterprise infrastructure. This information can impact the transport layers in which it rides. Recognizing this fact, and in an effort to save bandwidth, Autonomy has developed several mechanisms to reduce and minimize the impact on the network, one of which being a proprietary compression algorithm that reduces even further IDOL's low hardware requirements.

Distributed Link

Autonomy optimizes the streaming of rich media assets over high latency links, removing requirements for high performing speed and bandwidth.

Distributed Hot Deployment Mode

Hot deployment modes within the IDOL management units and server infrastructure allow software to be deployed in live environments and new requirements for areas such as functionality, resilience and scalability to be met without any disruption or downtime.

Distributed Abridged Mode

Autonomy ensures that the size of the installation does not impact performance or index cost. Autonomy's Distributed Abridge Modes apply to systems in which result sets of several million documents are routinely returned as part of a query. These modes allow for the overall size of the data that is returned across the network or transferred between components of the distributed system to be compressed and reduced to a minimum expression, achieving reduced transport and latency.

Further Reference:

Autonomy Performance and Scalability White Paper

Forthcoming Events

Archived Events

There do not seem to be any upcoming events related to this page at the moment, please visit our seminar and webinar pages for more information.

There do not seem to be any case studies related to this page available yet, however please visit our publications site at http://publications.autonomy.com/ for more information.