My G+ Profile
« May 2011 | Main | July 2011 »
June 27, 2011
Anonymous eCommerce: Building a Real 4th Party Offer Application with Kynetx
This is a long post. Don't worry, there are plenty of place to stop reading. You can stop at the end of each major section and have a complete picture for a given level of detail. Developers trying to see how to build a 4th party ecommerce application in KRL should read to the end to understand the complete picture.
The caption of Peter Steiner's legendary 1993 New Yorker cartoon reads: "On the Internet, nobody knows you're a dog." If you've paid attention to the Internet Identity Workshop logo, you know we use that concept for the conference, although our dog has a mask. But as a recent NY Times article, Upending Anonymity, These Days the Web Unmasks Everyone, points out, that's not really true anymore.
This erosion of anonymity is a product of pervasive social media services, cheap cellphone cameras, free photo and video Web hosts, and perhaps most important of all, a change in people's views about what ought to be public and what ought to be private. Experts say that Web sites like Facebook, which require real identities and encourage the sharing of photographs and videos, have hastened this change.
"Humans want nothing more than to connect, and the companies that are connecting us electronically want to know who's saying what, where," said Susan Crawford, a professor at the Benjamin N. Cardozo School of Law. "As a result, we're more known than ever before."
From Upending Anonymity, These Days the Web Unmasks Everyone - NYTimes.com
Referenced Wed Jun 22 2011 18:58:33 GMT-0600 (MDT)
This loss of anonymity is sometimes voluntary--say, when I post something on Twitter--but often is a side effect of our online activities. People call this "exhaust data." The point of this post is to illustrate an architecture for ecommerce that minimizes the tracable exhaust data from the activities associated with shopping for a product and trying to find the best deal.
The Scenario
One of the benefits of online shopping is the ability to quickly check out the same product from multiple vendors and find the best deal. There are lots of sites that try to do this for you, but they don't really take your individual situation into account and rarely give you comparisons between final prices (including discounts, taxes, and shipping).
In this demo, we're going to build a 4th party shopping assistant using some Kynetx rulesets. As I talked about in Building Fourth Party Apps with Kynetx last year, it is possible to imagine a Kynetx ruleset acting as a representative of the shopper (the 4th party) and other rulesets acting on behalf of the merchant (the 3rd parties). These rulesets can interact to provide the shopper with the information she needs while taking into account her personal situation without that personal data being shared with the merchant or their representatives in a way that is tracable back to the shopper.
In this demo, we'll build a simple offer app that shows the shopper what offers other merchants will make to them for the product they're currently viewing online. Think of this as a simple personal "request for quotes" (RFQ) system. We'll have two shoppers Sally, who lives in Utah and is an Amazon Prime member, and Bill, who lives in California and is in the military. To keep things simple, both shoppers have the same two favorite online merchants: Amazon and Home Depot. Further, they'll both start by looking at the same product, a Panasonic Microwave.
The scenario starts when the 4th party shopping app, we'll just call it the offer app, notices that the shopper is visiting a relevant product page and places a "want" button on the page. Here's what happens:
- Shopper visits Scott's Microwave Store
- Shopper clicks on the "want" button next to a product that has caught her idea
- Shopper gets personalized offers for the product from some of her favorite merchants
Here's what the offers look like:
Behind the Scenes
As far as it goes, that's nice, but the point here is to really build this, so let's get a little more detailed. The following schematic shows what going on behind the scenes.
We'll assume that our shopper, Sally, already has a personal data service that contains important information like her zipcode, what discounts she's entitled to, and her list of preferred merchants. We'll also assume that Sally has previously authorized access to this data for the offer app which is representing her in the transaction.
- The customer, Sally, visits Scott's Microwave Store.
- The offer app that Sally has installed to help with online shopping and offers, sees the request and places a "want" button on the page so that Sally can register her interest in this product if she desires.
- Sally clicks the "want" button.
- The app gathers information about the product from the page, encoded using the Schema.org Product specification
- The offer app accesses the personal data store and puts information about the product and Sally (zipcode and discounts) into a scratch space using a one-time identifier.
- The app raises an RFQ event passing along the one-time identifier so that the merchant rulesets have the required information.
- Merchant rulesets for Sally's preferred merchants, Home Depot and Amazon, see the event and execute rules to determine what offer they are willing to make for that particular product.
- As part of their effort to respond, the merchant rulesets get information (like zipcode and discounts) from the scratch space.
- Merchant rulesets contact their own offer systems, passing along information about the product and the relevant personal data
- Merchant rulesets write offer information into the personal data store and raise an event to signal they are finished with their offer
- The offer app sees the events from the merchant rulesets and uses offer information from the scratch space to present comparison offers for the product to Sally
There are several features of this interaction that bear comment:
- The one-time identifier protects the shoppers information from being aggregated and correlated by merchants.
- Merchants are responsible for their own rulesets and the logic that determines how they will respond to the RFQ.
- In this scenario, once Sally actually decides to purchase the product, she interacts directly with the merchant, the offer app only mediates the RFQ process.
- The system is using Schema.org encoded data from the page to gather information about the product Sally wants, but that's not required.
Using this system, Sally received personalized offers that take into account what she wants and who she is. During the process, her personally identifying information is shared in a way that can't be directly linked to her. As built, the system only presents offers, but other components--rulesets--could be added to manage a wishlist system that gathers relevant products and their comparison offers from the RFQ for extended shopping sessions.
The Gory Details
Now, let's dig into the details and see how each of the rulesets and complementary systems that make up the offer system work. Before we get into the individual rulesets, however, the following diagram shows the event hierarchy for the interactions.
There are only two Web events in the hierarchy, reflective of the simplicity of the overall user interaction: the pageview causes the "want" button to be placed when a place_want_button event is raised. When Sally clicks the "want" button, a product_found event is raised which sets off the rest of the interaction.
The product_found event is seen by the process_product rule in the offer ruleset. This rule eventually raises the rfq event. Merchant rulesets are watching for that event. The rfq event causes the discount rules to fire and eventually, the process_offer_solicitation rules wrap everything up. In this simple example, there's only one discounting rule in each merchant ruleset, but there could be as many as necessary.
When the process_offer_solicitation rules are done they raise finished_offer events that cause the process_offers and finalize rules to fire in the offer ruleset. The process_offers rule is looking for both offers to be returned, but more generally could look for the first two or three in a given period of time.
The pattern in the event hierarchy shows the overall structure of the system. The rules for the offer ruleset are at the top and bottom. The merchant rulesets operate, in parallel, in response to the rfq event and raise the finished_offer event. Their internal structure and operation is up to the merchant who owns them.
Now, let's look at the individual rulesets to see the detail of what's happening. We'll start with the scratch space module, move onto one of the merchant rulesets, and finish with the offer ruleset.
The Scratch Space Module
The various rulesets in the offer app need a way to share data with each other. They could attach it as attributes to the events, but that's cumbersome. Currently the Kinetic Rule Engine has no built-in means for apps to share data, so for purposes of this demo, I built a simple tag-space store as a scratch space that the apps could use. The scratch space is used to store information about the shopper and the offers that are returned.
The idea of a tag-space is that objects (in our case JSON encoded) are stored in a specific namespace and associated with one more tags. The objects can be retrieved by querying with tags. The module provides a function for querying the tag-space, getd that takes a namespace ID and a set of tags and returns an array (possibly empty) of objects that have been tagged with all of the submitted tags. The module also provides an action for storing new JSON objects called setd. Setd takes a namespace ID, a set of tags, and a JSON object to store.
We'll use the tag-space by picking a namespace for each shopper (in real life, we'd do this for each interaction to preserve anonymity) and storing relevant personal information such as preferred merchants, zipcode, and a list of discounts the shopper is entitled to in the tag-space. Merchant rulesets retrieve shopper information from the tag-space and store offers into it for the offer ruleset to use.
The Merchant Ruleset
For this demo, the Amazon and Home Depot rulesets are very similar, so I'll just go over the Amazon ruleset. In a real system, of course, that need not be the case. They would simply need to watch for the rfq event and raise a finished_offer event when they are finished, making appropriate entries in the scratch space along the way.
The Amazon ruleset contains a rule called prime_discount that is selected on an rfq event that determines the whether the shopper is a member of Amazon Prime (and thus qualified for free 2nd-day shipping) or not.
rule prime_membership {
select when explicit rfq
pre {
customer_id = event:param("customer_id");
discounts = tagspace:getd(customer_id, "discounts").head();
}
if(discounts.filter(function(x){x eq "AmazonPrime"}).length()>0)
then noop();
fired {
raise explicit event discount_offer with membership="Prime"
} else {
raise explicit event discount_offer with membership="Standard"
}
}
The prime_membership rule consults the scratch space for the customer with the tag discounts, tests the list of discounts to see if "AmazonPrime" is there, and then raises an event discount_offer with the membership event attribute set to either "Prime" or "Standard" depending on the result of the test.
I anticipate that rulesets will not do all the work, but will make use of other online systems that the merchant has in place. For purposes of this demo, I dummied up a system that takes information about the customer and product and returns an offer in JSON format. We declared the API as a datasource named offers in the global section of the ruleset:
global {
datasource offers <- "http://.../amazon.cgi";
}
The real work of creating the offer is done by the process_offer_solicitation rule which is selected when there has been an rfq event and a discount_offer event.
rule process_offer_solicitation {
select when explicit rfq
and explicit discount_offer
pre {
customer_id = event:param("customer_id");
zipcode = tagspace:getd(customer_id, "zipcode").head();
modelno = event:param("modelno");
oid = event:param("offer_id");
is_prime = event:param("membership") eq "Prime";
std_offer = datasource:offers({"modelno":modelno,
"zip": zipcode});
offer = {"price": std_offer.pick("$.price"),
"shipping": is_prime => 0.00
| std_offer.pick("$.shipping"),
"shipping_type": is_prime => "2nd Day"
| std_offer.pick("$.shipping_type"),
"tax" : std_offer.pick("$.tax"),
"notes" : is_prime => "assumes Amazon Prime member"
| "",
"url" : std_offer.pick("$.url")
};
tags = "offer|#{oid}|Amazon";
}
tagspace:setd(customer_id,
tags,
{"offer": offer,
"zip": zipcode,
"modelno":modelno,
"merchant": "Amazon",
"icon": "http://.../want/amazon_icon.png"
});
always {
raise explicit event finished_offer for a16x108
with merchant = "amazon"
}
}
After retrieving customer information from the scratch space, the rule gets a standard offer from the Amazon offer API (in the datasource:offers call) and then calculates a final offer using information from the discounting rule. The offer, along with other relevant information is stored in the scratch space using the setd action. Finally, the rule raises the finished_offer event to signal it is done.
The merchant rulesets shown here are simple, but the pattern is clear: the ruleset responds to an rfq event and raises a finished_offer event when it's done. In between, the ruleset can use as many rules and data sources as necessary to compute the merchants offer.
The Offer Ruleset
Normally, I'd skip explaining how the "want" button is placed on the page since that aspect is fairly rudimentary, but because we're processing Schema.org microdata about the product as part of that rule, it beats consideration. The offer ruleset makes use of a slightly modified version of Philip Jagenstedt's jQuery module for processing microdata.
rule place_want_button {
select when pageview "/want/"
pre {
want_button = <<
<span id="want_button"><img src="want%20button.png"/></span>
>>;
}
every {
after("#buy_button", want_button);
emit <<
$K("#want_button").click(function(){
var jsonText = $K.microdata.json(
"[itemtype='http://schema.org/Product']");
var prodprops = jsonText.items[0].properties;
var offer = prodprops.offers[0].properties;
var seller = offer.seller[0].properties;
app = KOBJ.get_application("a16x108");
app.raise_event("product_found",
{"prodname":prodprops.name[0],
"modelno":prodprops.model[0],
"produrl":prodprops.url[0],
"price":offer.price[0],
"shipping":offer.shipping[0],
"seller":JSON.stringify(seller.name[0], undefined, 2)
});
});
>>
}
}
Most of the work is done by JavaScript emitted by the rule. The after action places the button and the JavaScript places a listener on the browser "click" event that uses a callback function to process the microdata on the page and raise the product_found event to KRE.
The process_product rule is selected by the product_found event. The process_product rule is responsible for placing the notification on the page that the shopper will see with the correct structure for later rules to write into (the <div/> named offers). The notification also contains information about the product being quoted including the calculated final price for the product on the page the shopper is visiting.
rule process_product {
select when web product_found
pre {
price = event:param("price");
shipping = event:param("shipping");
final_price = price+shipping;
oid = "oid" + math:random(9999);
a = <<
<div class="prod">
<a href='#{event:param("produrl")}'>
#{event:param("prodname")}</a>
<br/>Model No: #{event:param("modelno")}
for $#{final_price} (including $#{shipping} shipping).<br/>
Sold by #{event:param("seller")}<br/>
<hr/>
<div id="offers">
</div>
</div>
>>;
}
notify("Compare Offers for #{event:param('who')}", a)
with sticky = true and width="300px";
fired {
set ent:oid oid;
raise explicit event rfq for merchants with
modelno = event:param("modelno") and
customer_id = customer_id and
offer_id = oid;
}
}
The rule postlude stores the offer ID (generated anew for each interaction) and then raises the rfq event with the model number, the customer ID, and the offer ID. As we saw, the merchant rules will use the customer ID as the namespace in the scratch pad and the offer ID to store data relevant to this offer.
The process_offers rule is designed to run when all the offers have been calculated by the merchant rulesets. The select statement in this demo is true when offers are complete from Home Depot and Amazon. In a production ruleset with many merchants, we might be content with the first two or three offers. The rule loops over the offers, calculates the final price for each merchant, and displays it in the notification placed on the page by the process_product rule above using the append action.
rule process_offers {
select when explicit finished_offer merchant "amazon"
and explicit finished_offer merchant "homedepot"
foreach tagspace:getd(customer_id,ent:oid) setting (of)
pre {
tax = of.pick("$..tax")>0 => "$"+of.pick("$..tax")+" tax"
| "no tax";
final_price = of.pick("$..price") +
of.pick("$..tax") + of.pick("$..shipping");
shipping =
of.pick("$..shipping") > 0 =>
"$" + of.pick("$..shipping") +
" for " + of.pick("$..shipping_type") +
" shipping"
| "no shipping fee";
prod_url = of.pick("$..url");
notes =
of.pick("$..notes") neq "" =>
"Notes: #{of.pick('$..notes')}"
| "";
merchant_id = of.pick("$.merchant").replace(re/ /g,"");
offer = <<
<div id='#{merchant_id}' class='offer'>
<a href="#{prod_url}" broder="0">
<img height="40px" border="0" align="left"
src='#{of.pick("$.icon")}'/>
</a>
#{of.pick("$.merchant")} offers this product
for $#{final_price} (including #{tax} and #{shipping})
#{notes}
<a href="#{prod_url}" border="0">
<img src="green_arrow.png" border="0" valign="top" height="13px"/>
</a>
<br clear="both"/>
</div>
>>;
}
append("#offers", offer);
always {
mark ent:prices with {"price": final_price,
"merchant": merchant_id}
}
}
The finalize rule postlude stores a map with the price and merchant ID in an entity variable for use by the finalize rule which highlights the offer with the lowest price. The finalize rule is selected by the same eventex that was used to select the process_offers rule.
rule finalize {
select when explicit finished_offer merchant "amazon"
and explicit finished_offer merchant "homedepot"
pre {
lowest =
ent:prices.as("array").sort(
function(a,b){
a.pick("$.price")> b.pick("$.price")
}
).head();
lm_id = "#" + lowest.pick("$.merchant");
}
every {
emit <<
$K(lm_id).attr("style","background-color: palegoldenrod");
>>;
}
always {
clear ent:prices
}
}
The rule sorts the prices entity variable that was created by the process_offers rule to find the lowest price and uses the merchant ID to change the background color for that line in the display. Finally, the rule postlude clears the prices variable.
Conclusion
The system described here is a working demo of a system for creating offers in a way that protects customer identity from being divulged. The system uses a set of merchant rulesets, built and maintained by the merchants themselves, that create offers given data about the product and the customer along with an offer management ruleset that oversees the process for the customer--effectively acting as a 4th party.
The merchant rulesets are not single purpose, but rather can be written so as to be used by any number of systems that need offers on products for customers. The offer management ruleset used them to present an offer immediately to the customer. Another 4th party ruleset might use them to look for offers for things that the customer has placed on her wishlist.
By protecting customer's personally identifying information and presenting them with relevant offers, we create a system whereby shoppers feel safe in looking to merchants for information. At the same time, merchants can place highly relevant offers in front of people who have expressed an intent to buy.
10:35 AM | Comments () | Recommend This | Print This
June 17, 2011
When the 20th Century Meets the 21st: Health Information Exchanges
Yesterday, Utah announced a statewide health information network called the Clinical Health Information Exchange. The system is run by UHIN, a quasi-public entity that has long run a health information network in the state that has been used to link payers and payees. Extending their reach into clinical information is a natural and will be a great thing if it comes to pass. From their "About Us" page:
The goal of the cHIE is to improve the quality of care you receive by increasing efficiency and maintaining patient safety. This is accomplished by enabling healthcare professionals to be better informed, and by reducing time and expense associated with missing information and ordering of duplicate tests.
The cHIE is a Utah effort to improve the quality of the healthcare you receive by making it easier for your participating clinicians to view critical medical information about you, no matter where you receive health care in Utah. It simply provides the means to locate and view information available from the healthcare entities that participate in this community effort.
The information that is shared may include laboratory and radiology results, transcription reports, medication and immunization histories, allergies, and other medical reports.
From myCHIE :: What is cHIE?
Referenced Fri Jun 17 2011 08:57:46 GMT-0600 (MDT)
My first thought was "cool, I want to sign up!" Heh.
Patients have to complete a consent form (PDF) and take it to each provider. The signing of the form has to be witnessed by an "authorized agent." Basically, you've got one more form to fill out. (I think it has to be taken to each provider. That's unclear from the site. If not, then once I give consent to one, I'd be giving consent to all.)
If you're a provider it's even worse: three forms that have to be faxed or mailed in. And your electronic health record and software vendors all have to do the same.
Hard to imagine how an online health information network can be founded on faxed paper forms, but I guess that's just normal in the health field. But it's really much worse than just paper forms. Since there's no way for me to "join" as an individual--just give my consent to providers to put my information in the system, I presumable don't have access to it, can't review it, and can't use it anywhere else.
What a lost opportunity. Imagine if it were set up as a personal data store, where I can join as an individual and really control my health information. Rather than some blanket consent, I could authorize and deauthorize provider access to specific information much more easily.
To be fair, there are legal restrictions on what can be done. And some patients wouldn't be comfortable with an online signup and account. Even so, the structure of this system reinforces the vision of the "all knowing physician" and further weakens the ability of patients to participate in their own care.
9:22 AM | Comments () | Recommend This | Print This
June 9, 2011
Modules in the Cloud
After I wrote my last blog post on "Hover Me - A Social Media Dashboard" I got to thinking about the power of modules and realized that there's a special power that is available from modules that operate in the cloud.
First, a little discussion of what Ed's Empire Avenue Module does. Empire Avenue's current API doesn't allow you to look up information by Twitter handle, only by Empire Avenue ID. Ed discovered that you could use the site to do it, so he used Kynetx to essentially do a little screen scaping to get the data and return it. KRL, the Kinetic Rule Language, has built-in tools that make that pretty easy. Ed essentially built an API on top of a Web site.
The problem, of course, is that any time you're scraping the Web site, you're subject to the vagaries of the HTML in the page. If the structure changes substantially, then you're in trouble. Ed has two weapons in his arsenal to protect him from that: (a) a jQuery-like query language for selecting parts of the page that allows him to get the data he needs without over-specifying and (b) the cloud.
The cloud-based nature of modules in KRL is important for what Ed's doing. If Empire Avenue changes the structure of the page beyond what the selector Ed wrote can handle, he can update the module and everything that depends on it will start working again. Now, of course, this is what we all love about the cloud, but I hadn't thought about how it applied to development--and especially modules--until now. There is great power in creating modules that operate in the cloud.
1:16 PM | Comments () | Recommend This | Print This
June 1, 2011
Hover Me - A Social Media Dashboard
Last December, a developer on the Kynetx platform, Ed Orcutt, released an application that used the Qwerly API to show people the other social networks that Twitter users were on. Called HoverMe, the application pops up a hover card when you put your cursor over a name or picture on Twitter or Facebook like so:
The app surprised us all because it got a little press love and took off with over 30,000 installs in just 3 days. Since it's original release, Ed has enhanced HoverMe to add PeerIndex scores and Empire Avenue value.
Last week I was sitting in the audience at Defrag and Delyn Simons mentioned HoverMe, Kynetx and Qwerly in her talk saying that HoverMe was a good example of someone else building out a user interface on top of an API, noting that Qwerly had closed their user-facing Web page not too long ago.
As part of building HoverMe, Ed has released three modules that will make it easier for other Kynetx developers to use Qwerly, PeerIndex, and Empire Avenue in their apps:
- Qwerly Module - The KRL Qwerly API Module provides a programmatic interface to the Qwerly profile service. (Ed's documentation)
- PeerIndex Module - Module for PeerIndex API
- Empire Avenue Module - Retrieve Empire Avenue Ticker Symbol, Share Price and ProfileURL based on Twitter Username.
The last one of these is interesting. Empire Avenue doesn't have an API that allows you to look up ticker symbols with a Twitter handle. Ed figured out how to use their Website to do it and built the module around that. In esseance, Ed built an API for them inside Kynetx. What's more, he was nice enough to release a module so other Kynetx developers could do it too. He told me that Kynetx made it easy for him to do this with built in support for HTTP, regular expressions, and jQuery style selectors.
I'm excited to see apps like HoverMe suceed, but even more excited to see developers like Ed coming onto the platform and being willing to share their work.
Meanwhile, go install HoverMe and see how it works.