Recommendation Evaluation Framework, Version 1
Evaluating Recommendation Engines, Platforms or Services
By Susan E. Aldrich, Sr. VP and Sr. Consultant, January 7, 2010

NETTING IT OUT

Recommendations are hitting their stride. A decade ago, merchants and publishers saw the recommendations on Amazon’s site and wanted it. A decade ago they had to spend a fortune to get it. Today, recommendation engines can be had for some thousands of dollars a month and can be implemented in a few weeks. Recommendation technologies—services, products, and platforms—aren’t just used to enhance the shopping experience. They can be used to present relevant and personalized results to customers at any stage of the customer lifecycle—to present the most relevant customer support information, for example. My guess is that recommendations will be ubiquitous within the next three to four years. I’m always optimistic on these guesses, but since recommendations are widely available as a service, rollout can be very swift.

SaaS offerings greatly simplify the technology aspect of implementation and virtually eliminate up-front costs, making it relatively easy for customers to sign up with a vendor. Nevertheless, choosing a vendor wisely is always better than choosing often. To aid in that choice, I offer a set of requirements and evaluation criteria set forth in this evaluation framework. And I will be evaluating a number of the leading products using the framework during 2010, leading to a detailed comparison.

WHAT’S INTERESTING ABOUT RECOMMENDATIONS

Benefits of Using Recommendation Technology

Web site owners worldwide have yearned for recommendations ever since Amazon started telling us that “people who bought this also bought that,” and today tell us “52 percent of people who looked at this bought that; 26 percent bought this other thing.” A decade ago, marketers had to spend a great deal to implement recommendations on their sites. Today, recommendation engines can be had for some hundreds of dollars a month and can be implemented in a few days. God, don’t you just love SaaS?

For the user, recommendations on a site can mean pretty darn good search results much of the time; emails with interesting offers; banner ads that catch your eye; useful guidance when you are selecting content, whether it’s a digital camera, a news article, a problem resolution, or a research document; clicks that take you from Google right to the content you need.

For the content owner—whether merchant selling products, publisher presenting articles, or marketer presenting offers—recommendation technology means delivering the most attractive item in those few seconds before you lose your audience’s attention.

What Is Recommendation Technology?

A recommendation engine can double or quadruple your click through rate as compared with handcrafted recommendations selected by your experts—the merchandisers or researchers or support specialists. Automated recommendations typically have significant impact on revenue, time on site, employee productivity, and customer satisfaction.

Recommendation technologies, at a high level, all operate like this:

1. Gather information about content items, typically via a data feed or crawl.

2. Gather information about all users’ interactions with all content.

3. Observe the current user’s activity.

4. Apply any of a variety of algorithms to select content.

5. Display the content.

6. Track and analyze the effectiveness of the recommendations.

7. Charge the client for the uplift.

Within this general outline, there are big variations, and the variations at each step are, in fact, important. It would seem that variations in step 4, the algorithms, would be the most important. Indeed, in the long term (several years), algorithms may prove to be the biggest enabler or limiter. In the first few years you deploy recommendations, however, methods of gathering information and displaying recommendations seem to have at least as big an impact.

Where Are Recommendations Used

Because recommendations had their most visible debut in ecommerce, people tend to associate recommendations with shopping. In the ecommerce arena, recommendations are used most often on product pages, shopping cart pages, category pages, and order confirmation pages. Recommendations are also used to tailor the content of those emails that entice you to stop work for a moment and shop. As a marketing team becomes more familiar and more confident with using recommendations, they expand recommendations to cover more of the interactions across the customer lifecycle.

But I think product recommendations are the tip of the iceberg for recommendations. Recommendations belong everywhere that content must be winnowed for a user, or everywhere that personalization improves a user’s productivity or experience. For example:

• A news site that knows I love football and don’t care about rugby, and always shows me the most interesting world news

• A personalized view of the corporate intranet that highlights my department, my division, and my projects

• My view of corporate research knowledge bases, weighted to what I’m working on

• Web sites that deliver coaching, e.g., for runners, dieters, investors, tailored to my style and goals

• My personalized support portal to the corporate help desk

• My company’s portal to a key supplier, e.g., Cisco, personalized by role or person

• My dashboard with my KPIs and corporate reports, with the hottest items on the first page

• A personalized investment site, e.g., Fidelity, tailored to the kind of investor I am

• A personalized commerce site tailored to my relationship, e.g., the car I drive, the sports I play

• Billing inquiry that always shows the disputed bills first

• Order history inquiry that always shows the most referenced order first

My guess is that recommendations will be ubiquitous within the next three to four years. I’m always optimistic on these guesses, but since recommendations are widely available as a service, rollout can be very swift.

The technology effort and the upfront costs for deploying recommendations are relatively small, which makes it relatively easy for companies to sign up for a recommendation service. The greatest effort for recommendation customers seems to be in learning how to use them, where to use them, and how to use them effectively—skills which will be portable to any vendor’s solution. Nevertheless, choosing a vendor wisely is always better than choosing often. To aid in that choice, I have compiled a set of requirements and evaluation criteria set forth in this evaluation framework. And I will be evaluating a number of the leading products using the framework during 2010, leading to a detailed comparison.

REQUIREMENTS

The requirements for recommendation services are derived from customer requirements. “Customers” for recommendation services cover several roles, including the end consumers of recommendations, the business people managing recommendations, and technical staff. Their requirements generate the evaluation criteria which are listed in Table A.

Recommendation Consumer Requirements

People consuming recommendations—the visitors, shoppers, readers, researchers to whom content is being recommended –need relevant and enticing recommendations of course; that’s the whole point. But they also need privacy and may wish they had some control over what personal information is used and how it is used. As recommendations are increasingly used to personalize experiences, consumers may also want a mechanism to indicate the persona they represent. “Don’t recommend for me, recommend for my [boss, niece, colleague].”

Recommendation Maker Requirements

The marketers, merchandisers, editors, business analysts, and other business people who are using recommendations as a tool to improve user experience have a broad range of requirements, starting from the initial deployment.

The business people who are responsible for recommendations need help making great ones, including:

• Guidance on how to deploy recommendations effectively

• Advice on how to increase recommendation effectiveness

• Training and tools to track and analyze recommendation effectiveness

Business people also need tools to deploy, test, analyze, and optimize recommendations:

• GUI or wizards for specifying rules for how recommendations are selected and presented

• Consistent interfaces designed for their business processes, not for the structure of the recommendation product

• Granularity in controlling the selection process, e.g., using customer history to select sports content but crowd wisdom in selecting fashion

• Testing and reporting that will compare different recommendation deployments

• Integration with other marketing tools, if recommendations are used for marketing

Business people need to manage the recommendation service and therefore need capabilities that include:

• Reporting on recommendation results, e.g., click through, conversion, revenue

• Reporting on recommendation engine service level, e.g., response time and availability of service

• Security of their content and user information; privacy of consumer information

• Reliable, high performance

• Controlled, role-based access to functionality

This report continues...

(Back to top)