Phil Windley's Technometria

« October 2008 | Main | December 2008 »

November 29, 2008

Tail Optimized Mutual Recursion in Clojure

Clojure, a LISP-dialect dynamic language that targets the JVM has been generating some interest in the programming language community lately. By targeting the JVM, Clojure gets speedy performance in a cross platform way.

One of the problems with targeting the JVM with a dynamic language that relies heavily on recursion is that the JVM doesn't support tail recursion (also called tail call optimization). The idea is pretty simple: for some common patterns of recursion the function call can be removed and the recursive function be optimized to a loop. The result is not only programs that run faster, but in some cases where the recursion goes deep programs that run period.

So you might imagine that this is a problem for a language like Clojure. For simple recursion, the recur special form takes care of the problem, but wouldn't work for mutual recursion.

Via Lambda the Ultimate, I learned of Rich Hinkey's implementation of trampolining in Clojure to make tail optimized mutual recursion possible. He gives an example.

"Here's how it works. Normally, if you have mutual recursion (i.e. which can't be replaced with recur), you can blow the stack:"

(declare bar)

(defn foo [n]
  (if (pos? n)
      (bar (dec n))
          :done-foo))

(defn bar [n]
    (if (pos? n)
        (foo (dec n))
	    :done-bar))

(foo 1000000)
-> java.lang.StackOverflowError

"To convert to a trampoline, simply return closures over your tail calls, rather than direct calls. This is as simple as prepending #"

(declare bar)

(defn foo [n]
  (if (pos? n)
    #(bar (dec n))
    :done-foo))

(defn bar [n]
  (if (pos? n)
    #(foo (dec n))
    :done-bar))

"Then make the top-level call via trampoline:"

(trampoline #(foo 1000000))
-> :done-foo 

On LtU, someone asked:

That there is "buzz" around a non-TCO LISP dialect in 2008 is utterly incomprehensible to me. I'm all for a modern/practical LISP, but come on...

Is there something specific about the JVM that makes it impossible, or is it just a naive implementation?

To which James Iry gave a great response:

Native instruction sets often let you do whatever you want with the stack. C doesn't in the ANSI standard, but you can do it with a bit of assembly. .NET IL has an explicit instruction for tail calls. The JVM, on the other hand, is very strict about how you use its stack and has no tail call instruction.

There are full TCO implementations in a couple of Scheme's for the JVM. Kawa uses trampolining and SISC uses a heap based custom stack. Either solution has performance implications as well as implications regarding Java interoperability. Java interop is a major design goal for Clojure.

Hickey's choice to make the programmer indicate when the trade-off is desirable is a pragmatic workaround. There's been some talk in the Scala mailing list about offering trampolining in the standard library - I wouldn't be surprised to see it in 2.8.0.

Well said. In the meantime, if you're familiar with SICP, the you might find Chris Rathman's first chapter of SICP in Clojure a useful way to see how the language differs from Scheme.

12:08 PM | Comments () | Recommend This | Print This

Online Stores Crowded on Black Friday

The Business Technology blog at the Wall Street Journal reported last night that several ecommerce merchants experienced slowdowns and in some cases service disruptions due to higher than expected demand.

It shouldn't come as a surprise that people are shopping online Friday. This year, retailers have been promoting online sales more heavily than in the past. In the case of Sears.com, the promotions seem to have worked too well: The site was unavailable for many visitors between 10 am and 12:40 pm EST Friday, according to Keynote. (It was also down when we tried to access to access the site at 3:30 EST.) A spokesman for Sears said that traffic was "higher than anticipated" and that the company was taking steps to ensure the site would be available on Monday---another popular online-shopping day.

From Business Technology : Retail Sites Crash as Shopping Season Opens
Referenced Sat Nov 29 2008 10:56:56 GMT-0700 (MST)

One of the realities of the online retail world is that most online retailers lock down their sites sometime in late October and don't make code changes and sometimes even template changes until after Christmas. There's just no point taking a chance during the time you collect a disproportionate chunk of your revenue. For ecommerce tool vendors, that means that the fourth quarter can be slim.

I have a feeling that most retailers aren't taking advantage of cloud resources to scale with demand. I'd love to see more data on that. With this years failings and the increasing popularity of cloud computing, maybe next year will be different.

Related articles by Zemanta

• Over 70% of People Surveyed Don't Enjoy Shopping Online
• Online shopping jumps 61% in two years: StatsCan

11:03 AM | Comments () | Recommend This | Print This

November 28, 2008

Kynetx Puppet Update

Pat Eyler sent me a note asking if we were still using puppet. In the spirit of trying to leverage my response to him for the widest possible audience, I thought I'd blog the answer.

Kynetx has been using Puppet (and a little Capistrano) to automate our infrastructure since this summer. While we haven't gotten everything as automated as we'd like, we're quite a ways down the road and it's making life as a small startup bearable.

My ultimate goal is to register a node in the system that delivers Kynetx Network Services (KNS) to the machine database (iClassify) and have it come up, configure itself, and be ready to put in production in less than 15 minutes. While there are still a number of manual steps in that process we are remarkably close and getting closer all the time.

I haven't found learning puppet to be difficult. There's plenty of documentation, including recipes that if not usable as is, are great resources for figuring out best practices. Also, there's a very helpful group on IRC (#puppet on irc.freenode.net) that I've turned from time to time.

The best thing about Puppet is that I can count on nodes under puppet management to be consistently configured and the puppet recipes represent an unambiguous record of what should be happening.

A recent example, we needed to ensure that data files from an OEM partner were on all of our initialization servers. I used Puppet in two ways:

1. Puppet is used to distribute the data files from the ops server to each of the initialization servers.
2. Puppet is used to manage the cron job that grabs updated data from the partner and put it in the right directory for distribution.

If I add another initialization node puppet will make sure it has the right data files just because I declare it to be an initialization node. Nothing else need occur.

I recently released Kynetx's operational procedures. For a small company with no fulltime engineering staff other than me, I think we do a pretty good job operationally. Much of that we owe to Puppet.

Needless to say, I remain, more than ever, a Puppet fan.

12:45 PM | Comments () | Recommend This | Print This

Bring You Own Context

Om Malik writes about the recent terrorist attacks in India and the torrent of information on Twitter. He says:

Despite the tremendous volume of information --- and its immediacy --- coming from Mumbai via Twitter, getting context about the situation has been a struggle. While a few people have been tweeting firsthand accounts, much of the information has been re-tweets or just rambling, reaction-based tweets. Maybe I was overcome with emotion, but the sheer volume of tweets and lack of clarity only fed my frustration with Twitter. (I'm sure it's the same kind of frustration people feel with blogs at times as well.)

From With Twitter, a Desperate Need for Context - GigaOM
Referenced Fri Nov 28 2008 12:14:11 GMT-0700 (MST)

Trying to figure out what's happening from a stream of rather random comments and commentary on Twitter is indeed frustrating. It's like being in the middle of a crowd that you can't see over and you know something's happening on the edge, but you can't tell what and you're trying to figure it out from what people around you are saying. In many cases, they can't see either-it's mostly hearsay.

That said, Twitter did a great job last week of telling me that something was happening and the general idea of what it was. Details, of course, are another matter.

Web technologies, in general, aren't real good at giving us context. Users manage, build, and maintain their own context in most situations. Threaded email is maybe the best example I can think of where something on the 'Net helps manage context.

Even a simple shopping experience is mostly context-free. You search for the product in Google, read reviews, on a few sites, look at prices and ultimately make a purchase. You're responsible for managing all the context of that transaction by yourself.

One of the themes I picked up on at Defrag is that there are any number of early stage companies who've realized this and are creating tools that help Web users with context. We're a long way from solving the problem, but it's a fun area to think about because there are tremendous productivity gains to be had from building tools that help users do something they've been doing ad hoc for years.

Related articles by Zemanta

• Crowdsourcing the Terror Response to Mumbai
• Tweeting the terror: How social media reacted to Mumbai
• Mumbai Attacks: Twitter Adds to the Noise but Is Still Valuable
• Mumbai attacks: Twitter and Flickr used to break news

12:20 PM | Comments () | Recommend This | Print This

Rocket Geeks

For anyone who grew up in the 60's, watched the space program move from satellites to men on the moon, and built model rockets while dreaming of going into space, this Wired article on homebrew rockts and the rocket geek who build them is a must read.

My kids do model rockets and have varying levels of interest, but it isn't anything like what my friends and I had. We lived for the next mission and news of every flight. Launching model rockets was, for us, a way of participating in that grand endeavor, even on a small, small scale. I'm happy to see that serious amateurs are building amazing rockets and it's not all left to NASA's pathetic shuttle program.

11:57 AM | Comments () | Recommend This | Print This

November 26, 2008

The Storm Blows

I had a friend who wanted an iPhone, but decided he'd wait and see how the Blackberry Storm stacked up. His son had an iPhone, so he took it into a Verizon office and played with them side by side. He tells me that in 5 minutes it was clear to him that he'd hate the Storm. He told me this as he was showing off his new iPhone.

David Pogue reviewed the Storm for the NY Times and reached the same conclusion.

I haven't found a soul who tried this machine who wasn't appalled, baffled or both.

From No Keyboard? And You Call This a BlackBerry?
Referenced Wed Nov 26 2008 19:41:08 GMT-0700 (MST)

The keyboard doesn't work, navigation is awful, and there's no wi-fi. Hard to believe people waited in line for this.

Related articles by Zemanta

• BlackBerry Storm: the aftermath
• Huge line and riots this morning at one NYC Verizon store?
• Pogue Trashes the BlackBerry Storm

7:32 PM | Comments () | Recommend This | Print This

EFail, not EMail

Jeff Atwood has a longish post on the problems with email. Of course, the biggest problem with email is there's way too much of it. I used to try to respond to each (non-spam) email I got but now I can't keep up. Unfortunately, I can't let each email commit me to spend time.

Jeff references Tantek Çelik's excellent post on the subject and gives three pieces of advice:

1. Channel that private email effort into a public outlet. Discussion boards, blog entries, comments, wikis, you name it. If it can be indexed by a web search engine, you're in the right place -- and many more people can potentially find, answer, and benefit from that information.
2. If you must send email, make it as short as possible. Think of it as Strunk and White on speed. Can you reduce your email into a single paragraph? How about two sentences? How about just the title field with no body, even?
3. Remember the theory of communication escalation. Email is just one communication tool in our toolkit; that doesn't mean it is always the right one for whatever situation is at hand. Take advantage of phone calls, instant messaging, text messages, and so forth, as appropriate. Scale your choice of communication method to the type of conversation you're having, and don't be afraid to escalate it (or demote it!) as the ebb and flow of the conversation shifts.

There are all kinds of ideas on how to effectively manage email and that's all well and good, but ultimately someone else (actually lots of someone else's) control how much is there. Any kind of management is going to fail as email volume grows.

7:25 PM | Comments () | Recommend This | Print This

November 24, 2008

Good Succeeds by Creating Useful Products

An old friend from Excite@Home, Jeff Huber was quoted in the NY Times on Google's product marketing strategy and the issue of data privacy:

"We do have a philosophy that our products should speak for themselves. We tend not to make a lot of noise," said Jeff Huber, senior vice president for engineering at Google.

As always with Google, the price point is appealing: zero, if you don't count the amount of personal data that I am trading for all that utility. With Google, it is always simple, and any engineer will tell you that simple is hard. There had been a lot of talk within Google about creating video chat as a PC-only application, a much easier endeavor for the company, but it would not have been simple for the consumer.

If Google owns me, it's probably because I am in favor of what works.

"I'm glad to hear it," said Eric E. Schmidt, the chief executive of Google, who was in New York last week. "We want a little bit of Google in many parts of your life."

Mission accomplished, at least on my desktop, but I asked Mr. Schmidt if I shouldn't be worried that I am putting all of my digital eggs in one multicolored, goofy-lettered basket.

"That depends on what you think of our company and our values," he said. "Do you believe we have good values?"

Mr. Schmidt seems nice enough, but I sometimes wonder if I will come to regret the easier, softer road I have chosen. A record of my surfing lives on its servers for 18 months --- not by name, but still. Google continues to insist that my IP address is not me, but a motivated government with a subpoena in hand could find me, lots of me, on Google's servers.

Most data privacy experts would call me a fool to index my life into any one company so deeply, and diversification in all matters is just common sense.

Mr. Huber countered that I am free to come and go as I wish.

"The nice thing is that we don't force you to use only our stuff," he said. "It is not tied tightly together, and the content is all easily exportable. If you feel like we are letting you down, or you don't like our products or we are failing to innovate, you can pick up and go where you want."

From The Media Equation - Google Seduces With Utility - NYTimes.com
Referenced Mon Nov 24 2008 07:04:42 GMT-0700 (MST)

Interesting when your SVP of engineering is speaking about your marketing strategy to the Times.

Related articles by Zemanta

• Google - the world's first firewalled monopoly
• Defending Google, the video

7:06 AM | Comments () | Recommend This | Print This

November 21, 2008

The Halting Problem, in Verse: Scooping the Loop Snooper

Geoffrey K. Pullum, Professor of General Linguistics in the School of Philosophy, Psychology, and Language Sciences University of Edinburgh has written a marvelous verse, in the style of Dr. Suess, that explains the proof that the Halting Problem is undecidable in a clear and humorous way. The verse is called Scooping the Loop Snooper (PDF). If only all of Computational Theory were this easy to understand!

7:27 PM | Comments () | Recommend This | Print This

Geopolitics and Cloud Computing

When I first read "Pentagon's New Map" and heard Tom Barnett talk about how he analyzed geopolitics, I realized that here was an theory a geek could love. Tom uses concepts like and system administration to talk about how the world does and ought to work. I got to interview Tom about his second book, Blueprint for Action and I'm anxious to get my hands on the new book, Great Powers.

So, I wasn't surprised when a post from Tom called A nice primer on cloud computing and its relationship to SOA showed up in Snackr. The concepts behind cloud computing are critical to the global economy so we ought to be scared when someone telling us about the world doesn't know them rather than be surprised when someone does.

Tom brings up the notion of "free-trade zones" for data centers. Interesting idea. Reminds me of Cryptonomicon.

2:54 PM | Comments () | Recommend This | Print This

The Conservative View on Guantanamo

Yesterday a federal judge--ironically the same one who'd ruled earlier that Guantanamo prisoners weren't entitled to civil review--ruled that five men held there for seven years be released. There was insufficient evidence that they were involved in any crime. In fact, that's maybe too charitable. If you read the details, it seems like the Government had nothing more than a hunch and an uncorroborated accusation.

Think for a minute about what this means: five human beings were held in prison for seven years without much recourse. Think about what that means to them and their families. I hate that that happened and feel very bad about it.

The real shame is that it didn't have to happen. We have 200 years of legal precedent in this country that has given us a justice system that while far from perfect is designed to prevent this kind of abuse and we refused to use it.

I titled this post "The Conservative View" because I think that conservatism would take two positions that would have argued against what has happened on Guantanamo:

1. Conservatives should fight against government intervention in human lives where ever possible. A conservative position would argue for, not against basic human rights.
2. A conservative position would argue for following existing, well understood rulesets and not using ad hoc procedures.

In the case of Guantanamo, we've done neither. We've allowed government to intrude into the lives of humans without following established due process procedures to ensure that that intrusion is warranted.

Note that this doesn't mean we should be soft on terror, security, crime, or anything else. It does mean that a position consistent with basic conservative philosophy would argue for human rights and due process--not against it.

Related articles by Zemanta

• Judge Rules U.S. Violated Human Rights
• Guantanamo detainee wins ruling

8:39 AM | Comments () | Recommend This | Print This

November 20, 2008

Kynetx Operational Procedures

In the spirit of transparency, Kynetx has published our operational procedures and practices online. As we gain more control over our infrastructure automation, we'll also have real-time operational information and statistics.

I'd appreciate feedback on what's there. Keep in mind, this isn't out "dream" but rather we're trying to present a true picture of where we're at. We obviously have goals to make this better and have quite a bit of experience in mature operations so we know where we need to be and where we currently fall short. I don't believe that there's anything there that would compromise operational security, but i you think there is, please let me know.

Obviously we're taking some risks in being open and honest about the true nature of our operational capability as a young start-up. But I believe that in the long run transparency will win out and our customers will appreciate the openness, recognize that we're a start-up with lots of competing priorities, and not punish us for places where we may fall short compared to more operationally mature companies.

10:56 AM | Comments () | Recommend This | Print This

November 19, 2008

Skype on My Mac Book Pro is the Best Conference Phone I Have

Today I had to do a job interview with a candidate in Chile. He had Skype and wanted to use it. I was leery because I had four people on my end who needed to be in on the call, so USB headphones weren't going to cut it. We decided to press forward and try the call with the MBP's internal speakers and microphone.

It worked beautifully! We could hear him fine and he could hear us--even with some people sitting 8-10 feet from the laptop. So much so that this evening when I was getting ready to get on another call, I decided to Skype out to it rather than use my Polycom. Skype plus my Macbook Pro is the best conference phone I own.

5:34 PM | Comments () | Recommend This | Print This

OS X Leopard Technical Details

Jordan Hubbard, Apple's Director of Engineering of Unix Technologies, spoke at LISA '08 last week. Most people are commenting on the date he gave for the release of Snow Leopard (10.6), the newest version of OS X. I have to admit, I'm ready for some stability improvements, but I was much more intrigued by the details of his talk (PDF).

He spent the bulk of his talk on technical features in Leopard (10.5) that many aren't aware of. He starts with a number of security improvements in Leopard: file quarantine, sandbox, package and code signing, application firewall, parental controls, non-executable (NX) data, address space layout, and randomization. I was completely unaware of most of these improvements.

Jordan also talks about the Unix improvements in Leopard. Leopard is fully Unix compliant. But more than that includes a number of additions like DTrace, Launchd (complete), ASL (replacement for syslog), a read-only version of ZFS (for future compatibility) with a read/write version available. He also talked about Apple's evolving open source strategy.

Last, he talks about improvements coming in OS X that will help developers take better advantage of the multicore chips and sophisticated GPUs that already ship with most Macs. Future kernels will provide better facilities, along with APIs, for managing multi-threaded programs. He says:

Forget everything you thought you knew about multi-threaded programming (and, as it turns out, most developers didn't know much anyway). The kernel is the only one who really knows the right mix of cores and power states to use at any given time - this can't be a pure app-driven decision

I don't know if there's audio or video of the talk available, but it would be very good to hear firsthand.

BTW, anyone know what "LWFLAF" stands for? Jordan uses it as some kind of metric in discussion the various versions of OS X, but I couldn't figure out what it meant.

Related articles by Zemanta

• ZFS file system coming to Snow Leopard server edition
• Snow Leopard screenshots surface

6:53 AM | Comments () | Recommend This | Print This

November 18, 2008

CloudFront Seems Like a No-Brainer

Doug Kaye, who actually wrote a book on hosting, has been beta testing Amazon's CloudFront service--a high performance front end for Amazon S3. Doug's favorably impressed.

My calculations show that Kynetx would be able to put 80% of our bandwidth load on CloudFront (most static JS libraries) for $1.19 per day and if Doug's experience is typical get better performance to boot! Seems like a no-brainer to me given that we're paying several hundred dollars per month for a 750 Kbps circuit that I'd rather not upgrade for a while. Offloading 80% of that traffic would give us a lot of head room!

Related articles by Zemanta

• Amazon Web Services Launches CloudFront
• Amazon CloudFront: Outlook for CDN Is Cloudy (and That's Good)

5:38 PM | Comments () | Recommend This | Print This

November 17, 2008

Do You Support P2P File Sharing Inside the Enterprise

The topic of P2P file sharing inside the enterprise recently came up in a conversation I was having. I count myself as pretty enlightened on these kinds of things, but beyond getting large Linux distros quick or sharing disk images, virtual machines, and virtual appliances, I had a tough time thinking of legitimate reasons an enterprise might support it or even allow it.

The other side of course is that there are bandwidth issues, both network and people. If you're just supporting people watching the latest episode of The Office in the office, then you've not accomplished much. I did a couple of searches on Google trying to find discussion about P2P in the enterprise and mostly found stories about how to block it.

There are, I'm sure, security and legal issues surrounding some of the more notorious uses of P2P file sharing. Has anyone cataloged them and worked out mitigations?

So, what do you think? Is there a legitimate reason for a CIO to support P2P file sharing? What do they tell the CEO or board about it?

4:55 PM | Comments () | Recommend This | Print This

Leavitt as America's CTO

There's been considerable discussion about Obama's intention to appoint a Chief Technology Officer for the United States. Count me as a supporter of that move. It's almost cliche to say that Technology plays a vital role in the US economy and our place in the world.

Of course, when I say CTO, I mean CTO and not CIO. And I think that the job would be vastly different than what CTOs do in a high-tech business. America's CTO would be focused almost 100% on policy issues. After all that's what government does.

To be qualified, you need to (1) pass the ethics tests, (2) understand how technology relates to government and the public in a wide range of areas, and (3) be a policy wonk. Items (1) and (3) disqualify most of the people who've been suggested.

Given these requirements I think Mike Leavitt, the current Secretary of HHS would make a great CTO. He understands technology very well and relates it well to policy. He'd be an able spokesman. I can imagine no one better. Of course, he's a Republican--in fact his name has been put forth as a candidate for the next Chair of the RNC. Still, if I got my wish, that's who would be the CTO.

Related articles by Zemanta

• The White House CTO - Web 2.0 need not apply
• Nominations, please
• Technology in the Obama Era
• National CTO a Good Idea But Please God Not Schmidt

9:45 AM | Comments () | Recommend This | Print This

Obama to Ditch Blackberry and Email

A story in Sunday's NY Times about Obama surrendering his Blackberry caught my attention. Until Jan 20, 2009 whatever he writes on it is private. After that, it's all public. Who among us could live with that kind of requirement? Not many.

It's a sad irony that we've constructed a public world--and believe me, this extends far further than the President--where public figures must eschew the kinds of tools we all lean on every day. I know of what I speak. I used to work in Governor Mike Leavitt's office and "channel" was a constant thought in the back of everyone's mind. Pick the wrong channel for certain types of messages and you might find yourself answering questions you'd rather not.

You might be thinking, "if you're honest, you've got nothing to fear!" Oh how I wish that we true. The fact is that there will be people out to get Obama who care nothing about the truth. That's true of anyone in public life.

Here's the acid test: anything that would take more than two sentences to explain is something you'd rather the other side not get a hold of an twist. If you need 5 minutes to counter their 30 second sound bite, you lost.

So what will Obama do? The obvious answer is give up the Blackberry. But Obama has surprised before in like situations; maybe he'll hold onto it. If you're curious about what this might feel like, go back through the last week's email and consider whether you'd post them all online. Then reconsider that thinking how someone opposed to you and your agenda might twist them. It's a sobering exercise.

9:27 AM | Comments () | Recommend This | Print This

November 15, 2008

4000th Blog Post

This post on the death of advertising was my 4000th blog post on Technometria. Kind of snuck up on me. The mountain picture has nothing to do with blogging or anniversaries or milestones. I just liked it.

6:11 PM | Comments () | Recommend This | Print This

November 14, 2008

Social Web TV and TechCrunchIT at IIW

John McCrea, David Recordon, and Joseph Smarr recorded a special edition of their video podcast Social Web TV at IIW this week. Their guests were Max Engel of MySpace, Eran Hammer of Yahoo, Dick Hardt of Sxipper, Paul Trevithick of Parity, and Doc Searls of the Berkman Center for Internet and Society.

Steve Gillmor recorded a TechCrunchIT show with David Recordon of Six Apart, Kevin Marks of Google, and with the help of Echovar's Cliff Gerrish. where they talk about the open standards debate.

8:30 AM | Comments () | Recommend This | Print This

After the Death of Advertising, Shopper and Merchants Can Start Talking

Dave Winer wrote yesterday about the death of online advertising. He says:

I've been saying it for as long as people have been building businesses on advertising on the web, it's not a longterm thing. Now we're at the end of the road.

Assuming the economy comes back from the recession-depression thing that it's in now, when it does, we will have completely moved on from advertising.

The web will still be used for commercial purposes, people will still buy things from Amazon and Amazon-like sites, but they will find information for products as they do now, by searching for it, and finding out what other people think, not by clicking on ads and buying things on the pages they link to.

No one needs advertising, and there are much better ways to sell products.

From Online advertising is now dead (Scripting News)
Referenced Fri Nov 14 2008 08:02:18 GMT-0700 (MST)

I don't know that I'd go as far as saying it will not come back at all, but merchants will find new, innovative ways to reach customers without advertising when their budgets get tight.

Of course, we might call these new things "advertising." I get that frequently when I explain what Kynetx does. People call any message from a merchant an advertisement even though, in many cases, it's not widely broadcast.

There were plenty of discussions relevant to Dave's post at IIW this week. The VRM crowd was there in force with grounded discussions about how the implicit contract between merchant and shopper can be rewritten in ways that are better for both.

Most exciting to me, the discussions about r-buttons were getting down to the level where you could see real protocols and standards developing behind the talk.

I was troubled by some VRM discussions that still see to border on being hostile to merchants and even commercial efforts in general. I think that only hurts the opportunity to have to redefine what commerce is and how it takes place. I also think that some of the discussion gets lost in relationships in general, not just those that exist between the merchant and shopper. Nothing wrong with figuring out infrastructure for relationships, but it's nice to keep things focused on the task at hand.

The next steps we take can be as important as the final end state. Dave's insight that the economic downturn opens up opportunity is a good one. At Kynetx we're working on new ways for shoppers and merchants to relate. We're creating new channels for relevant messages--both from shopper to merchant and from merchant to shopper. Not advertising--just plain old communication.

8:19 AM | Comments () | Recommend This | Print This

November 13, 2008

A Great Internet Identity Workshop!

About a month ago, Kaliya and I had a serious conversation about possibly having to cancel the Internet Identity Workshop this time. Registrations were not coming in as fast as usual and no one had committed to any of the major sponsorships. I was concerned I'd end up personally eating the cost of the conference if we moved forward. Shortly after that, Ping Identity and Microsoft both stepped up and gave us confidence to move forward. That's a good thing because this turned out to be the best IIW I can remember.

There seem to be peaks and valleys for when people need to com together to cooperate and this must have been one of the peaks. There was a lot of energy from start to finish and I had many people tell me how much they'd accomplished. I'm glad that the workshop was able provide the venue for that progress.

Thanks to the sponsors and especially to the attendees for coming ready to work. Kaliya did an amazing job, as usual. The next IIW will be May 18-20 in the Computer History Museum in Mountain View, CA. I hope to see you there!

5:57 PM | Comments () | Recommend This | Print This

November 11, 2008

Supporting Authentication Discovery in a Standard Way

I'm sitting in a session at Internet Identity Workshop that is discussing what standardized support browsers could provide to all authentication systems. Right now all browsers support one: Username/Password over HTTP Authentication. Authentication's come a long way since 1993.

Dick Hardt of Sxipper made the observation that users view what's "inside the chrome" as the application. The browser chrome is largely ignored. That seems right to me.

Authentication systems like basic form-based, openid, and information cards are all existing without explicit browser support. Forms have password fields, but that's just so that the browser blanks out the characters. Beyond that you're on your own.

This kind of discussion is a good example of how far the Internet identity discussion has come. When you get to the point of talking about getting these protocols "understood" in HTTP in the same way as BasicAuth, you're getting past the plumbing issues that have been part of the ID discussion for the past 3 years.

Ultimately this is about taking the discovery process that started wit YADIS and XRDS to the next level and letting it work across even more protocols.

Once the discovery protocol is decided upon, standard plugins could be written for Firefox, IE, Chrome, and Safari that would implement the discovery process for identity enable the browser for whatever identity system(s) the relying party supports. Four open source, community supported plugins could replace the myriad proprietary plugins available today. That would lead to greater penetration and also give browser manufacturers something to code against when the time comes that they want to build the discovery code into their product.

3:54 PM | Comments () | Recommend This | Print This

Tweetdeck Rocks

Yesterday I found out about Tweetdeck, an Adobe Air application for managing twitter. Tweetdeck is much more than a way for watching your tweet stream and posting tweets. Tweetdeck is a dashboard for Twitter. You can create separate panels, for example, to follow searches. Yesterday I was using it to follow three different searches related to Internet Identity Workshop and seeing tweets from all kinds of people who I don't normally follow. Of course, I found more people to follow!

Related articles via Zemanta

• Looking for Mr. Goodtweet: How to Pick Up Followers on Twitter
• TweetDeck Offers Features Twitter Lacks [Featured Download]

10:00 AM | Comments () | Recommend This | Print This

November 10, 2008

Where Oil Comes From

One of the reasons I love reading Jon Udell's blog is that he shares the results of his curiousity. Not only is Jon curious in general, but he's especially curious about data and how your can mundge it to produce information.

The latest example is Jon's look at where Oil comes from--not from where you think probably. The answer, if you live in the US is Canada and Africa. 33% of US oil comes from North America (with Canada being the largest "foreign" supplier) and 20% comes from Africa.

How did Jon find this out? By importing the data into DabbleDB, manipulating it, linking locations to maps, and then publishing it. It took him about 45 minutes. These kind of tools are freely available, but not widely used. Amazing information at our fingertips.

Related articles via Zemanta

• W. David Stephenson - Citizen-Government Relationships
• Jon Udell Explains FeedSync and How It Might Benefit You

2:39 PM | Comments () | Recommend This | Print This

November 8, 2008

Breaking Some Ironclad Rules about Startups

Joel Spolsky and Jeff Atwood built Stackoverflow.com, a site for programmers to ask other programmers questions. If you haven't been over there, you ought to check it out. They have some very nice concepts for building a community site and it's nicely done. Joel and Jeff have been discussing programming, and the building of Stack Overflow on their weekly show on IT Conversations.

In this article in Inc magazine, Joel talks about seven iron clad rules he has for starting a technology venture:

1. Vet programmer carefully
2. Put everyone in one office
3. Plan
4. use bug tracking
5. Test software before releasing it
6. Measure the success of your venture by its profit

Ironically, Jeff and Joel broke all seven of these and nothing bad happened. In fact something very good resulted. Joel explains why he thinks they cheated the devil in some detail in the article. He concludes with this, however:

The truth is, the three guys who coded Stack Overflow are great programmers. They're smart, and they get things done. And in the end, that's what really matters. Entrepreneurship boils down to the simple fact that a team of really smart people who can get things done are going to get smart, useful things done. Need proof? No problem: Check out stackoverflow.com.

From How Hard Could It Be?: The Unproven Path - technology ventures - Joel Spolsky
Referenced Sat Nov 08 2008 16:03:05 GMT-0700 (MST)

Related articles by Zemanta

• StackOverflow: A Teeming Beehive of Programming Questions
• Launching stackoverflow.com

3:51 PM | Comments () | Recommend This | Print This

WPA Crack

WPA, or WiFi Protected Access, is one of the primary means of protecting Wi-Fi hubs. Ars Technica reports that Erik Tews, a PhD candidate from Germany is prepared to present a paper at PacSec this week that explains how he was able to crack it.

The exploit doesn't actually crack WPA keys, but does allow an attacker to sniff a packet, make minor modifications to the checksum and then use the access point to check the results. This man-in-the-middle attack could allow attackers to make ARP poisoning or Even DNS poisoning attacks.

Related articles by Zemanta

• Wap Hacked!
• Once thought safe, WPA Wi-Fi encryption is cracked
• Researchers crack WPA encryption

3:08 PM | Comments () | Recommend This | Print This

November 6, 2008

Has Anything Gone Right for the GOP the Last Few Years?

Utah Governor Jon Huntsman (who was re-elected with 78% of the vote) was quoted in a Washington Post article on rebuilding the Republican Party asking "Was there anything that went right for [Republicans] over the last several years?"

Utah's Republican governor, Jon M. Huntsman Jr., predicted a "broad discussion of the future of the party" with virtually every big issue on the table. "Was there anything that went right for us over the last several years?" he asked, saying that the party's international agenda has been "flawed" and U.S. prestige abroad "squandered, in terms of where you see our level of respect overseas."

"Domestically we have been totally tone-deaf in terms of recognizing the environment and where most Americans are in terms of having a healthy environment," said Huntsman, a popular governor who easily won reelection. "We have been missing in action in terms of any semblance of fiscal responsibility, [and] we have put forward nothing meaningful in terms of health-care reform that has any traction."

From Republicans Confront Formidable Task Ahead - washingtonpost.com
Referenced Thu Nov 06 2008 07:11:18 GMT-0700 (MST)

This aligns nicely with the feelings I expressed in my post on conservatism. Interestingly, Republican Governors haven't been totally missing in action on these kinds of issues. They are on the front lines when it comes to tacking health care, balancing environment and development, and other issues that matter to voters. The National party would do well to turn to them.

7:22 AM | Comments () | Recommend This | Print This

November 5, 2008

Detecting URLs with Regexps

Jeff Atwood talks about the problem of detecting URLs in text. The problem, as Jeff points out, is that lots of interesting characters are legal in URLs, including parens. So, writing a regular expression to distinguish between these two URLs is hard (but not impossible):

My website (http://www.example.com) 
http://en.wikipedia.org/wiki/PC_Tools_(Central_Point_Software)

Jeff's solution is pretty comprehensive and cuts the Gordian Knot of enclosing the URL in parens by removing them programatically--a good solution since we're not worried about nesting.

8:05 PM | Comments () | Recommend This | Print This

November 4, 2008

Why Can My Car Move When I'm Not in It?

Jonathan Yarmis from AMR Research just told the story of someone who visited his house to pick up some kids and forgot to put the car in park. She went in the house as her car took off down the driveway. It ran over a power transformer and then crossed the street and came to rest 150 yards into the trees on the other side of the road. The woman became aware that her car was on the move when Mercedes called her to tell her that it's airbags had just deployed (the power went off first, but that clue was less specific). My question: why can a car move when no one is sitting in the seat? Why not tie the airbag sensors to the transmission? There could be an override for cases where the sensor failed.

11:43 AM | Comments () | Recommend This | Print This

Worlds: Tabs for Programming

Via Lambda the Ultimate a pointer to a paper by Alessandro Warth and Alan Kay (PDF) called "Worlds: Controlling the Scope of Side Effects". I didn't get as much out of reading the abstract as I did from the opening paragraphs of the intro:

Suppose that, while browsing the web, you get to a page that has multiple links and it is not clear which one (if any) will lead to the information you're looking for. Maybe the desired information is just one or two clicks away, in which case it makes sense to click on a link, and if you don't find you're looking for, click the back button and try the next link. If the information is more than a few clicks away, it might be better to open the link in a new tab in which you can explore it to arbitrary depths. That way, if you eventually decide that wasn't the way to go, you can close the tab, and easily try a different path. Another option is to open each link in its own tab, and explore all of them "concurrently".

Something like the tabs of a web browser would be even more useful in a programming language, where undoing actions is a lot trickier than clicking a back button. As an example, consider the task of programming a robot to open a locked safe, as shown in Figure 1. There are two keys, A and B (each in its own room), but only one of them unlocks the safe. Using a conventional programming language, we might tell the robot to grab key A from room A, then go to the safe and try to unlock it. At this point, if we find that key A does not open the safe, we probably want to have the robot clean up after himself before trying the next alternative (nobody likes a messy robot). So we must tell the robot to take key A back to room A, and then return to its initial position.

In a programming language that supports "tabs", these clean-up actions would not be required: we could simply open a new tab, and inside it try to open the safe with key A. If A turns out to be the wrong key, we can simply close this new tab to return to the initial conditions.

This paper explores the idea of "tabs for programming languages", which we call worlds.

I like this idea. I like that they used Javascript to build their prototype even more. Javascript's extensibility and flexibility make it an interesting place to explore new programming constructs.

11:10 AM | Comments () | Recommend This | Print This

Connecting in Afganistan

Michael Yon has a good post about mobile phone use and misuse in Afghanistan. If you're not reading Yon, he's an independent reporter who's been embedded in Iraq and now Afghanistan. I've been reading him for a while. Very good to get a first hand, detailed report. And I love the pictures he puts up on his blog.

9:31 AM | Comments () | Recommend This | Print This

November 3, 2008

Remember Twitter Vote Report

Don't forget Twitter Vote Report when you go to the polls tomorrow! From the "about" page:

Twitter Vote Report is a non-partisan, all-volunteer network of software developers, designers, and other collaborators have teamed up with the award-winning blog techPresident to launch this effort. The only resources contributed to this project are the participants' time and expertise!

Millions of Americans will be voting this Election Day. Many of these voters will have terrific experiences and we'd love to hear about those. But many voters will experience voting problems that we have been hearing about for years: long lines, broken machines, and registered voters who can't vote because their names aren't showing up on the registration rolls.

Using Twitter Vote Report, voters will be able to share their experiences and resources with one another (e.g. "#wait:120" meaning that the wait time is 120 minutes). These messages will then be aggregated and mapped so that we can "see" voting problems around the country in real-time.

From Twitter Vote Report | About
Referenced Mon Nov 03 2008 17:31:40 GMT-0700 (MST)

5:34 PM | Comments () | Recommend This | Print This

Stowe Boyd on Lifestreaming on the Edge

The notion that we're drowning in information is false. The world is full of information and we've been dealing with it from the pre-agricultural era. We're reaching back to systems and techniques that we've never lost. There's no such thing as information overload. Attention isn't a resource that needs to be parceled out. This is like the earlier failed metaphor of "knowledge management." You can't manage knowledge like bricks.

The movement of message control away from large organizations (the center) to the edge has destabilized the status quo. Non-market collaborative efforts (like open source) have had a similar effect. Networks are all edge and no center. This is what Doc Searls calls the giant zero. This is the demassifying media.

Social media has led to an egalitarian situation where everyone can get involved. Anyone can get involved. But that doesn't mean it's democratic because your influence depends on your talents and other factors.

We live in a world that instant messaging built. RSS is analogous to instant messaging. These tools set the stage for a different way to perceive and interact with the world. Blog comments have moved off of blogs to disaggregated community sites (like Disqus). This gives commenters more control and makes them full fledged participants. Whether Digg or Techmeme, value is being harvested from communities, but the value is very different.

He shows his desktop: Snackr, Friendfeed flow UI, Flickr, Twitterfox, and so on. These are all flow apps. There are dozens of streams now and there will be lots more in the future. These differ on the basis of the social interactions they enable. There will be 5 or 6 themes, but lots of implementations.

This leads to a model called "lifestreaming." People are continually broadcasting their life to groups of friends and even strangers. People know where you are and ask you questions about things in your life because of life streaming.

Attentionitis is the notion that we're being driven crazy by multi-tasking and continuous partial attention. While it's true that these things take time, but they also add value. He says "we're all made greater by the sum of our connections, but so are our connections." He references research that shows we're being cognitively changed by Web experiences. We're moving ourselves into a slightly different state of cognition. It's like juggling. You can't explain it--you have to do it and practice.

Stowe believes that where we've been with social tools and where we're at now is leading to a new culture that is more egalitarian, more progressive, and more liberal. I guess we can only hope he's wrong about the last one. :-) He mentions his son's involvement in games where he leads groups of people. I've had the same experience with my son. He concludes "if we don't destroy it first, what kind of world will our kids make?"

5:06 PM | Comments () | Recommend This | Print This

Getting Past Telco 1.0

Doc Searls has a good post on Getting Past Telco 1.0 at Linux Journal. He uses T-mobile's ridiculous "roaming charges" as an example of the kind of thing old-style telcos do that makes their customers hate them. He concludes:

We're always going to have big companies. There are many things only big companies can do. But when those things involve the Net, those companies need outside help from free-range developers. They can't do it alone. They can't mandate it from the inside. Won't work.

Dan Frye once told me that it took IBM several years to realize that they couldn't tell their Linux kernel hackers what to do, and that in fact it was those hackers who were actually telling IBM what to do. We need similar realizations in the Internet space. We need hackers to develop new applications that make the most of a Net that's wide open and free. We need to show the telcos and cablecos of the world that the Net is a vast frontier that it is their privilege to open, that free-range developers are going to be their primary source of solutions, and that customers are more than cattle to be herded and milked.

From Getting Past Telco 1.0 | Linux Journal
Referenced Mon Nov 03 2008 16:21:45 GMT-0700 (MST)

4:22 PM | Comments () | Recommend This | Print This

Connected Platforms for Connected Applications

Ian Davis, the CTO of Talis is speaking about connected platforms. He starts with a lesson in economics.

In the pre-industrial era, most goods were produced by individuals or small groups working together. consequently, most trading was local. Trading to a large market required a very large company capable of vertically integrating all its functions. The East India company is an example: ships, docks, towns, armies, and even currencies were all done by that one company.

Transactional costs for a given market size were usually much larger than the associated value in the pre-industrial era. Transaction costs fell with the industrial revolution as infrastructures like railroads, telegraphs, and canals were built. Large markets opened up and allowed greater specialization. Adam Smith observed that the division of labor is a function of the zie of the market.

The industrial revolution caused the cost of transactions to drop below the value associated with a given market size and allowed the development of the horizontally integrated firm. Large horizontally integrated firms like Li and Fung employ relatively few employees for their revenue. Li and Fung has $7.5m in revenue per employee (and 8000 employees). The average reveue per employee is in the hundreds of thousands of dollars. Google in $1.2m.

The Internet's huge connectivity and ubiquity has put more people in economic contact with one another than ever before. The Web's been around 15 years. Think about the companies that couldn't have existed 15 years ago. This isn't just the big ones like Google and Yahoo! but thousands of small businesses.

Could computing holds the promise of allowing businesses to scale costs with revenue. That is, you don't need large capital investments to prepare for future capacity needs.

Similarly, social networks are enabling businesses to reach millions with almost zero friction. The fixed costs of provisioning users from a social network are small. The semantic web will radically change the economics of sharing and combining information.

The combination of these trends means that increasingly large markets are available to businesses with much less cost. The implication is that companies will become smaller and more specialized, focusing on creative talent, not heavy engineering. I'm not sure I agree with these trends doing away with the need for engineering. Architecture and engineering will become increasingly important, it seems.

The bottom line: companies can become loosely coupled, small, efficient, very creative, and highly networked.

2:25 PM | Comments () | Recommend This | Print This

Flow Applications

Chris Shipley is moderating a panel with Dick Hardt (Sxipper) and Sam Huleatt (Workstreamer) on applications that support flow. Chris turns the time over to Dick who demos Sxipper (the new version) that provides information about things your browsing. This reminds me of something I saw at ETech years ago called Dashboard (Nat Friedman). Ultimately this is an idea about getting context about information.

Can flow applications make the problem worse by increasing the amount of information we have to process? Possibly. UI helps. Flow applications need to be intelligent about what information is presented. Presence and location information help. Flow apps tend to be wrappers for "everything else."

Who sets the metric for filtering? Flow apps can make use of social relationship in information passing. The social layer is a great filter. Who do you follow on Twitter? They're your filters. I used to do this with blogs, but there's not so much happening there now. Twitter is a better source now. Followers/following is a new model for information sharing.

Filters can be dangerous. They lead to echo chambers if not properly managed. How can you find new ideas? The word "filter" is maybe a misnomer. We might want more of something. It's easier to recommend things that are relevant than to "find" things that aren't.

11:31 AM | Comments () | Recommend This | Print This

It's Always a Good Time to Start a Web Business

Howard Lindzon is up next speaking on by it's always a good time to start a web business. Howard runs a hedge fund. The headlines we're seeing today aren't good. Leaders need to look beyond the headlines. The goal: zig when other's zag (with hat tip to Warren Buffet's "the time to be fearful..." The best time to start a "premium" business or one base don ad revenue was the last four years.

Now we need to focus on being "too small to fail." Get your idea and product ready and the first customer under your belt as soon as possible. Over the next 6 months, VCs are going to wake up and set up parts of their funds to address these kinds of businesses.

It's never a good time to start any business. Good businesses can be started at anytime. 80-90% of small businesses fail--all the time.

Businesses fail for lots of reasons: taking too much money, having the wrong partners, having an idea that isn't well tested or well thought through. Focus on getting the right product and the right customers. Fish where the fish are. Everyone wants to be the shark (Google), but better to find what you do and do it really well. Don't try to be everything all the time.

The discussion of leverage comes up. Social leverage is good. Financial leverage is bad. You can't have too many friends. Eric asks Howard to talk about StockTwits, a site for twittering about stocks.

Be an expert at something.

10:59 AM | Comments () | Recommend This | Print This

Paul Kedrowsky: Around the Horn (Defrag08)

Paul Kedrowsky starts the panel by referring to a James Surowiecki article in The New Yorker called Everyone's Watching. If I understand the reference, the key point is that the current financial meltdown wasn't caused by too little information, but too much--and with no one paying attention.

There's some discussion about whether we need to take a step back and pay more attention. On Twitter, @kevinmarks says "this makes me think of Neal Stephenson's ideas from Anathem of deliberately introducing isolation and delay to gather deep insight." Paul doesn't believe that paying more attention is an option.

Is the problem with gatekeepers? Have the old gatekeepers (like editors--weird to hear the word used in a positive sense) given way and there's nothing to take their place? Questions of trust abound. Sound bites convey a lot of information to the initiated (or true believers) but leave others out. Understanding sounds bites requires previously established context.

(There's some funky things going on with the sound system. Different panel members are coming out of different speakers and it keeps throwing me, thinking someone is talking from the back or side of the room. Several of the panelists are simply not understandable due to poor audio, at least by me.)

Stow Boyd makes the point from the audience that hyperconnected financial markets are examples of things that are inherently complex and unpredictable--like the weather. The solution isn't better understanding or magic oracles (gatekeepers?) but simplifying things so that they're understandable. Paul disagrees that there were a few key measures (like debt to GDP) that we just ignored.

Panelist makes the comment that every one in the room is sitting on a mountain of data that they could use to make better sense of their world. Being data driven is a culture.

Paul makes the point that extracting useful signal from noise leads to constantly telling people how things are different and you risk getting into a Cassandra syndrome, where you are constantly perceived as a "doom and gloomer."

Eric Norlin makes the comment from the audience that our metaphors are breaking down. Email has a metaphor that anyone can understand because the understand letters. Humans are not good at dealing with asynchronicity.

Paul says that the financial crisis is a great problem for this group because we aren't talking about the theme of the conference in some abstract sense, but we're living in an example.

Final comment from Brad in the audience: we're not in a difficult time compared to most people in the world and especially most people through time. The people in this room have an incredible opportunity to innovate and solve problems. We need to look at this from the positive side.

10:16 AM | Comments () | Recommend This | Print This

November 1, 2008

Being Lazy the Right Way

Mike Farmer takes on code generation in his blog and argues that frameworks are a better way to be lazy:

Newbies can use them as a crutch instead of learning the language like they should. When things break or don't work the right way, they are stuck. Then come countless hours stepping through code that looks like it was written in greek trying to figure out what the problem is. Not my idea of a good time

From Frameworks Over Code Generators
Referenced Sat Nov 01 2008 21:38:41 GMT-0600 (MDT)

9:39 PM | Comments () | Recommend This | Print This