NETTING IT OUT
How do the five leading e-commerce search engines stack up? Between December
2005 and March 2008, our search and findability expert, Susan Aldrich, reviewed
five leading ecommerce search solutions—from Celebros, Endeca, Fredhopper, Mercado, and SLI Systems—using her in-depth Enterprise Search Planning and Evaluation Framework. In this Bull’s-Eye comparative ranking report
we evaluate and rate the five products according to how well they addressed the
230 criteria in our framework. What surprised us most is how well all five solutions
address all of Sue’s criteria. There were only about 65 criteria where
the solutions differed.
The report detailing our findings and presenting our enterprise search evaluation framework, now in Version 3, is available at no charge at our Web site.
The lone category where criteria were missed by all products was Marketing Management.
None of the products supports defining product bundles via the merchandiser interface.
In the area of integrated marketing, none of the products provide integrated
reporting of end-to-end search activity, conversion, and ROI. Only one vendor,
SLI Systems, provides tools to manage paid search campaigns based on site search
activity.
We apply our Customer Scenario® methodology to determine the requirements
on which we base our evaluation criteria. We see two distinct groups of people
whose success depends on search capabilities. The first consists of seekers,
those people who are your customers, partners, employees, and other stakeholders.
The second group consists of information owners who have a variety of goals,
from quality of customer experience to cost cutting to relationship deepening.
We address the goals and metrics of these stakeholders in our analysis of the
planning and evaluation criteria for enterprise search solutions.
The detailed evaluations of each product are available for free to subscribers and
can be purchased and downloaded by non-subscribers.
The comparison spreadsheet containing our evaluations of these five products against all 230 criteria is also available to subscribers or available for purchase
and download. We provide this comparison in editable
form so that you can add your own evaluations and/or assess other products (or
your own home-grown solution) against our criteria.
The solutions themselves vary in age from 4Q2005 to 1Q2008. Not surprisingly,
the oldest solution performed most poorly. Surprisingly, the newest did not rank
the highest. Fredhopper Access Server V 6 and Endeca IAP V.1 tied for first place.
Mercado 4 and SLI Systems Learning Search January 2008 tied for second place.
Celebros Salesperson V4 came in fifth. But, to be fair, the Celebros version
we reviewed is not the latest version, which would have ranked better. We hope
to update our oldest in-depth evaluations before the end of the year and will
re-rank these products as appropriate. Also, bear in mind that SLI Systems is
an SaaS offering, while the other four products are available in both in-house
installable and hosted versions, so there is an apples and oranges aspect for
some of the criteria.
An important caveat to remember in reviewing this report is that two of the most
important criteria are beyond our capacity to measure, since there are no acceptable
benchmarks: the effectiveness of the retrieval and ranking functions and the
scalability of the solution. So you’ll need to run your own trials using
your own e-commerce content to test and to tune search effectiveness and scalability.
OVERVIEW OF THE EVALUATION PROCESS
During the past 18 months, I evaluated five ecommerce search solutions using
my Enterprise Search Evaluation Framework. This report analyzes a detailed,
side-by-side comparison of 230 framework criteria for each of five products.
The five products are:
• Celebros Salesperson V 4.0
• Endeca IAP V 5.1
• Fredhopper Access Server V 6
• Mercado 4
• SLI
Systems January 2008
It took me too long to wrap up this project. There were a few delays beyond
my control, a few delays I reluctantly agreed to, and a few that were my
own fault. Today I am ruefully facing the music: because of the elapsed time,
two of the products on this list have a new version number and the comparison
is therefore somewhat dated. But I think it’s time to publish and be
damned. Literally, perhaps, in this case, as people involved in developing
the products I review here will no doubt have a few choice words for me,
including suggestions for my future residence. It is my goal to update the
product reviews and publish a new comparison by year end, mollifying the
currently enraged but enflaming the currently mollified. I hope I’m
not too optimistic: achieving my target requires the commitment of the vendors
involved, and their current good intentions must survive the pressures of
year-end sales cycles.
Keep in mind as you read through this report that the age of the products reviewed
spans two years, a very long time in the rapidly moving search space. Table
A contrasts the product versions I reviewed against the current versions.
Step 1: Reviews of Each Product
Using my evaluation framework, over the past 18 months I produced five detailed
product reviews that assesses how the solution addresses each of the framework
criteria. The individual, detailed reports analyzing each product are available
on our Web site.
Step 2: Side-by-Side Comparison
My esteemed colleague, Fanny Wong, pulls the tables from each of the individual
reports to create an Excel chart with a side-by-side comparison of the 230 criteria for all five products. This report is also now available on our site.
Step 3: Scoring and Ranking
I used a simple system of allotting one point per criterion, where possible.
Not all criteria are designed such that they can be met or measured. For
example, “What is the pricing model for the product” is an example
of a framework element that is not scored. The products with more points
have the higher ranking.
The Enterprise Search Planning and Evaluation Framework
The products were reviewed using Version 2 of our Enterprise
Search Planning and Evaluation Framework. It has six categories covering
some 230 evaluation criteria. (Download the PDF to see Illustration 1.) The first four of these
categories address capabilities that interact to deliver the seeker’s
search and navigation experience. The six evaluation categories are:
•
Seeker Interface. Seeker experience addresses what types of searches or
queries can be performed and how the query is translated and executed.
It also addresses how the results are organized and presented, including
capabilities that ensure that the user is neither overwhelmed with choices
nor presented with no results or guidance.
•
Seeker Experience Management. The key search management activities center
on tuning search results to improve the Quality of Customer ExperienceSM
(QCE) and to enhance profitability by boosting revenues and trimming costs.
•
Marketing Management. Site search and navigation are critical contributors
to marketing campaigns and product merchandising activities. Key goals
supported by site search are connecting buyers with products, tracking
effectiveness of campaigns from the first Yahoo! search on through to the
campaign end point (e.g., a purchase, a lead generated, a page view), and
strengthening brand awareness and reach.
•
Information Collection Management. Information collection stewards and
owners have their own goals of efficiently delivering consistent, complete,
and accurate information. They want to ensure that the collection has all
the information its users will need and no extraneous or obsolete information.
Their top findability objective is ensuring that the people using their
collection find the answers they need on their first request. This goal
drives development and evolution of metadata and taxonomies, as well as
synonyms, concepts, intents, and adjustments to search ranking. They are
responsible to assist in tagging and classifying their information, and
to assist in the evolution of metadata standards and taxonomies.
•
Architecture. The search engine architecture determines how the search
solution will fit into the existing environment, how it will be managed,
how it scales, and how it will incorporate all the relevant data sources
a company currently has. Architecture also addresses the extensibility,
scalability, and manageability of the technology.
•
Product and Company Viability. Product viability criteria consider the
business aspects of enterprise search solutions and their suppliers. These
criteria are much easier to evaluate than functionality criteria, but they
can be deal breakers. A company’s history and current financial statistics
are key markers for its future viability.
We apply our Customer Scenario® methodology to determine the requirements
on which we base our evaluation criteria. We see two distinct groups of
people whose success depends on your search capabilities. The first consists
of seekers, those people who are your customers, partners, employees, and
other stakeholders. The second group consists of information owners who
have a variety of goals, from quality of customer experience to cost cutting
to relationship deepening. We address the goals and metrics of these stakeholders
in our analysis of the planning and evaluation criteria for enterprise
search solutions. The report detailing our findings and presenting our enterprise search evaluation framework, now in Version 3, is available
at no charge on our Web site.
Performance by Product
© 2008 Patricia Seybold Group Inc.
Illustration 2. Each solution’s performance in the five major
technical categories is presented in this chart. The total points possible
in each
technical category are provided next to the category name.