Five Steps in Selecting a Search Engine
Increase the Success of your Search Selection and Implementation
By Susan E. Aldrich, October 23, 2008

NETTING IT OUT

There are five steps to successfully selecting a search engine:

1. Inspect scenarios for technical requirements

2. Assess content

3. Estimate limits

4. Plan for special cases

5. Perform comprehensive technical evaluation and planning

Following these five steps, or even using just some of these steps, will greatly increase the success of your search selection, and of the implementation project that follows.

INTRODUCTION

The five steps I have identified for selecting a search engine will help you avoid undertaking a technology project that can’t be successful. Some of the steps can be strenuous for both you and your search vendors. Depending on obstacles in your organization, you might not manage to complete all the steps. But your selection process will be greatly enhanced by considering all, and completing some, of the steps.

I’m assuming you have laid the groundwork for technology selection: you have a clear vision and the support of key stakeholders. (If you don’t, check out my advice in The First Big Mistake in Search and Knowledge Projects.) Your vision defines search success metrics to be achieved, and the metrics establish the value of achieving the vision. The five steps in successful search engine selection are as follows:

1. Inspect scenarios for technical requirements

2. Assess content

3. Estimate limits

4. Plan for special cases

5. Perform comprehensive technical evaluation and planning

(Back to top)

STEP 1. INSPECT SCENARIOS FOR TECHNICAL REQUIREMENTS

The right way to identify requirements is to inspect your audiences and their key scenarios using real life situations and questions and answers. A customer looking for guidance on using a product might be best answered by a video another user posted to YouTube. The answer another customer needs might be explained in 30 seconds buried deep in a two-hour webinar. Your distributors need to embed your search results into their applications. Does your requirements list support those scenarios?

In determining the technical requirements for your search engine, it’s critical that you consider a broad set of audiences and use cases, both current and future. Think big during requirements, so you’ll select a product that will fit your needs for some years. (Think small during implementation, so your projects can attain quick success.)

Consider all the types of audiences your information retrieval service will serve, and list their key scenarios or use cases in business terms. We’ll use this list again in step 5. Make sure you don’t only include seekers of all types, but also your search managers, people who manage or contribute content, business managers, and IT architects and developers. All of these people have specific needs. People responsible for managing the search experience need reports, tools, and technology to support their efforts.

(For a detailed list of stakeholders, scenarios, and requirements, check out Enterprise Search Planning and Evaluation Framework, Version 3.)

STEP 2. ASSESS CONTENT

I believe the most common cause of poor search experience, and therefore the most common reason for replacing search technology, is the organization’s failure to consider how content impacts findability. You must assess content to determine:

• Does it cover all topics adequately?

• Is it tagged for findability?

Quick tests you can perform: download a free trial search engine or use your current search engine to test some of the scenarios you identified in Step 1. To what degree are poor results due to poor content? To state the obvious, if the content doesn’t exist, the search engine can’t retrieve it. To state the not so obvious, if the content has poor or no titles and headings or is badly written, a search engine may find it but rank it very low in the results – and the user will never find it. If it doesn’t have a date on it, the search engine may rank six-year-old content well ahead of current information.

If your content is in poor shape, you must acquire tools that will automatically analyze and tag it for findability and report on coverage of what seekers are asking for. Quite a few search offerings include tools that perform these tasks, and there are also standalone tools and services. Whatever sourcing you chose, plan to assign resources to manage the initial and ongoing efforts of ensuring your search engine can present the right content. If you can’t fix poor content, there is little point is proceeding with better search technology.

(Back to top)

STEP 3. PLAN FOR SPECIAL CASES

Special cases in search are the information, audiences, or scenarios that aren’t readily handled by your technology or processes. The most common examples today are complex requests from researchers and scientists, experience and expertise search, and audio search. Failing to plan for special cases is, sadly, the de facto standard in dealing with these thorny problems. But not handling them creates pockets of justifiable resentment in the organization and erodes the benefit you should be receiving from your search technology investment.

Devise a strategy for dealing with each of them, including a high-level resource plan and timeline for implementation. If the strategy is not acceptable – too costly, or too lengthy—then you are not taking an approach to search that is going to meet your organization’s requirements. Don’t just assume the situation is not practical. Rethink your approach and look for a solution that leaves the door open for everyone.

STEP 4. ESTIMATE LIMITS

Overestimating the organization’s patience, budget, and commitment is a sure path to trouble. As you begin the selection process, make your best estimate of the following:

• Maximum time to payback investment

• Maximum time to achieve noticeable results

• Amount of resource each business unit will commit during the implementation project, and ongoing

• Degree of success that must be achieved (e.g., Search Success Percent or Conversion Percent)

• Effort required to get content in adequate shape

• Effort required to maintain content in adequate shape

This information will help you evaluate the solutions, and prioritize your requirements. And, if the effort required clearly transcends the investments you think are committed, then it’s time for a heart-to-heart talk with your sponsor. There may be some pressure to back off on your estimates of initial and ongoing effort. Before you bow to that pressure, remember that failure to invest properly in content and special cases is probably the root cause of your users’ dissatisfaction with the current search technology.

(Back to top)

STEP 5. TECHNICAL EVALUATION AND PLANNING

A stringent proof of concept (POC) test is the final hurdle to successful selection. It is critical that the POC test be comprehensive and difficult. Devise tests using searches that don’t work today, that touch on your special cases, and that address all the key scenarios you identified in Step 1. Use test automation tools to test dozens or even hundreds of queries. Have knowledgeable people identify the two best answers for each query. The single most important criteria: how many of the right answers appear in top 5 results.

If your requirements include content improvement, such as metadata extraction and classification, you should test these capabilities in your POC as well.

Part of your evaluation should also be the implementation plan. It will no doubt start with a pilot, typically delivering big results in a short time, perhaps 30 days. Too many implementations never get farther than the pilot. You need commitment and plans that carry through your entire set of requirements (including those special cases).

CONCLUSION

Following these five steps, or even using just some of these steps, will greatly increase the success of your search selection and of the implementation project that follows. One final word of advice: Keep your vision for your search project at the front of your thoughts and make sure that every decision will move you closer to, or at least no farther from, the ultimate goals of the investment.

This report continues...