Build 384: Annotating Search Results with Large Datasets


Kynetx Logo

Recently Azigo and the Better Business Bureau launched the BBB app that helps people locate BBB accredited business:

The BBB's Accredited Business Locator is powered by Azigo's free, downloadable browser plug-in and displays the "BBB Accredited Business" seal whenever consumers search for products or services using popular search engines such as Google, Yahoo! and Bing. This seal indicates that a business is reputable and offers one-click access to the BBB report for that business. This provides maximum consumer protection and minimizes the risk of online fraud.

This app is powered by Kynetx and the techniques necessary to make it work have been factored into the Kynetx Rule Language (KRL) so that they're available to everyone. The BBB app required two things that KRL didn't support well:

  1. The BBB data set of accredited businesses was very large running to millions of records
  2. The BBB was interested in annotating local search results

The problem with large data sets is that previous versions of KRL had an action called annotate_search_results that relied on preloaded data. The Kynetx Network Service (KNS) ensured that the data was available based on information from the KRL program. With millions of records we could hardly just load up the entire data set and there was no way to segment it since you don't know what the user's going to search on (and thus what results are relevant) until they do it. Enter remote

We've added a new option to annotate_search_results called remote (optional parameters to actions are specified using a with clause). The remote option specifies a URL that is the location of a remote data set. When it's used the selector function is ignored (and can be left out) of the annotate_search_results call. Instead the remote data URL is called after the search results are available and passed the relevant data from the results. The remote service uses the data about the search results to query the database and returns a JSONP response that will be evaluated to annotate the results.

Given that this runs after the search results page has loaded, you might think it would be slow, but our experience with BBB has convinced us that it works just fine.

We've also added a annotate_local_search_results action that works just like annotate_search_results, but is specifically tailored to annotate the local results in each search engine.

You can find more information about search annotation in KRL in the documentation. You can also download and try out the BBB app to see how it works for yourself. We've built a sample app so that you can see a sample app that uses these two new methods with remote data.

One other thing: build 384 included the let_it_snow() action because it's winter and snowy Web sites can be fun.