ENTERPRISE SEARCH PLANNING AND EVALUATION FRAMEWORK, VERSION 3
How to Plan and Select Search, Navigation, and Discovery Solutions for Web Sites, Applications, Intranets, and as Enterprise Platforms
By Susan E. Aldrich, January 3, 2008

We define enterprise search, navigation, and discovery as the technologies companies deploy to guide a seeker to the information that will resolve her question, as well as support information owners’ efforts to deliver adequate quality of seeker experience. Enterprise search is used not only on corporate Web sites, but in intranets, customer and partner portals, and all business applications that use a combination of structured and unstructured information.

Great search and navigation has the potential to deliver huge benefits. We have spoken with companies that eliminated 60 percent of customer support calls, increased cart size by 270 percent, and saved $5 million by increasing employee productivity. Whether or not your results reach these heights (or exceed them) depends on how poor your seeker experience is today, the quality of your content, and how effective your organization is at managing the seeker experience once you’ve got the right technologies in place.

Search is not solely a technology solution, because information and seekers’ needs are constantly evolving. You must assign responsibility for the quality of the seeker experience, monitor the quality, and take actions to improve it. You must also assign responsibility for the quality of your information collections, monitor the quality, and take actions to improve information quality.

To simplify the selection of products that can contribute to enterprise search implementation, we’ve developed our framework for evaluating enterprise search products and architectures. This framework describes evaluation criteria in the areas of seeker interfaces, seeker experience management, marketing management, information collection management, architecture, and product and company viability. We also provide our recommendations on search-related metrics and responsibilities.

Since 2003, we have been using our earlier evaluation frameworks to assess vendors’ offerings for search and navigation. To date, we’ve assessed offerings from ATG, Autonomy, Celebros, Endeca, FAST, Fredhopper, Google, InQuira, IBM, Knova, Mercado, Microsoft, Northern Light, Oracle, SLI Systems, and Thunderstone, as well as solutions from EasyAsk (acquired by Progress Software), iPhrase (acquired by IBM), Verity (acquired by Autonomy), and WebSideStory (acquired by Omniture).

This report presents an updated framework that replaces the earlier frameworks and addresses key new requirements:

• Cross-lifecycle customer experience
• Search as a business intelligence tool
• Search and navigation embedded in business applications
• Customer-contributed content
• Increasing prevalence and value of audio content
• Geographic search

This framework also provides our assessment of goals and metrics for key stakeholders in successful search and navigation. As with our earlier frameworks, we plan to evaluate leading solutions against our criteria.

Enterprise Search Planning and Evaluation Matrix

© 2008 Patricia Seybold Group

Illustration 1. This diagram presents our evaluation matrix for enterprise search solutions.

WHAT IS ENTERPRISE SEARCH?

A Definition

First of all, what do we mean by search? We don’t separate search from navigation, because, to the seeker, they are inseparable tools. And, from an information manager’s perspective, both techniques must be employed consistently to deliver high quality results. If you separate search and browse algorithms, rules, and technologies, seekers may not be able to get back to that document they recently viewed (because they’ve forgotten the exact steps they took to find it), or searching for a category will yield very different results from navigating to the category. From a technology standpoint, navigation should––or at least could––be driven by search.

What do we mean by enterprise search? Simply, it is search technology deployed within a corporation: portals; intranets; public Web sites; ecommerce sites; knowledgebases; and customer service, CRM, and other applications. Enterprise search solutions also provide tools for people who manage findability for information collections and the quality of seeker experience for customers, employees, partners, and suppliers. The solutions offer approaches to organizing search results, tuning the relevance of search results, and customizing the presentation.

There are two types of search tasks. In one, you know what you are looking for and want it retrieved for you: I want a ream of bright white copier paper. In the other type of search task, you don’t really know what you are looking for: I need to find a new approach to packaging this product. We see the second example as more discovery than search, and it has a few requirements in both technology and search management.

Enterprise search does not exist in a vacuum. It relies on information stored in transactional databases, data warehouses, Web pages, documents, and images. It uses organizing principles such as categories, taxonomies, metadata, and attributes. And it relies on wise decisions from information owners who recognize that a customer searching for “how to” information is not going to be happy with marketing collateral. All three elements—search management, information, and search technology—work together and have some capacity to make up for each other’s shortcomings and disabilities. For example, search technologies can identify attributes that you haven’t explicitly codified; hard work from the marketing department can make up for a search engine’s poor vocabulary; and rich product information (by containing all the right words) can mitigate the effect of poor search technology.

Search Solution Landscape

There are a lot of solutions to choose from, and a number of approaches companies take to achieve the right seeker experience. You can attempt to buy or build your tools, try for a single or very few vendors, or opt for the best of breed in key areas. Here are the choices you’ll be offered:

WHAT’S NEW IN VERSION 3 OF THE FRAMEWORK

A software industry rule of thumb is never to touch software before Version 3 (a standard set by Windows, we think). Our evaluation framework is quite a bit lower risk than most any software you’ve encountered, so it’s been safely used by hundreds of companies in its first two versions.

In the latest revision, some existing requirements are made more stringent. As products mature, customer expectations advance, and customer sophistication increases, it makes sense to raise the bar a bit.

We’ve also expanded the range of search applications that are specifically called out in the framework, adding customer support; customer and partner portals; research, scientific, and technical departments; and legal department search applications. If you need search technology to address these areas, we’ve identified capabilities that are particularly important for your solution to have. For the most part, the capabilities are mentioned elsewhere in our matrix, but we felt it was important to be explicit about the critical requirements for these search applications. We want to help you make the right decision.

We’ve also updated our requirements to address search as a business intelligence tool, search and navigation embedded in business applications, customer-contributed content, the increasing prevalence and value of audio content, and the increasing value of geographic search.