My comment is still in moderation, so in the spirit of the real-time Web, here it is:

Tom, great post. The problem is spot on. The solution makes one key assumption — that you can’t approach the problem with pre-computation.

That’s true for a service like Hunch which sees a tiny amount of data about me, especially when I first show up. Given how little you know about me, you have two choices: either you pre-compute info in an impossibly large space, which is impractical, just as you describe, or you do the type of real-time processing which is much more effective. So far so good.

But that’s not necessarily the best way to approach the problem from the standpoint of someone who had a lot of data about me, e.g., Google or Facebook or even Amazon. The set of Internet-connected humans is small from a computational standpoint and the meta-data trail we leave is growing at a much slower rate than compute/storage. Pre-computing starting with 100 dimensions doesn’t work. Pre-computing starting with a few billion humans works really well, if you have a lot of data on the humans.

This is one of the fundamental advantages Amazon, FB, GOOG and others have compared to point services such as Hunch. It’s not a fair fight. So you have to innovate like crazy to compensate. Rock on!

People who do data analysis and machine learning have learned one thing through experience (can someone claim it as their Law?): to solve a complex problem with little data you need fancy technology but you may be able to solve the same problem with much simpler technologies if you have a lot of data.

So, the big guys have a fundamental advantage. However, there are ways to even out the playing field. It starts with users owning their data and giving it to a trusted third party that can do the same type of pre-computation the big guys can do. Then you put the right access control mechanisms allowing users to share this info with third party services like Hunch.

Facebook already does something like this through F8. Facebook apps get access to a lot of valuable information not available through other means. But Facebook doesn’t share any really interesting pre-computed analytics, at least not right now. They are the smartest of the big guy bunch so far. With the exception of the the string of privacy faux pas, I’ve been consistently impressed with their strategy.

Today, we’re adding a fifth principle that clarifies a specific use case that we feel is particularly damaging to the user experience: promotions that include hidden renewals without specific opt-in will not be permitted. Because it’s our belief opt-out offers are misleading and do not have the best interests of the users in mind, we will be updating our Terms of Use this week to better clarify this for users and developers.

via MySpace Says Zero Tolerance For App Scams, Changes Terms Of Use

There is a simple principle at work here. Visibility is key. There are many shady things quietly going on on the Net today.  Once someone shines a big, bright light it becomes harder to hide. Everything starts with visibility.

By itself, this may be absolutely OK depending on how much of this data is collected and how it’s used by third parties. However, most of the entities (targeting companies, ad networks, etc.) operate in a rather opaque manner for most consumers. First, you often don’t even know who those third parties are because they have no visible presence on the websites you visit yet your browser makes HTTP requests to them and they typically set multiple cookies on your machine. Many of them don’t even have a web site for a consumer to visit, for example, to figure out which business sits behind the URL and read their privacy policy.

A good (random) example is ad.yieldmanager.com, requests to which are typically hidden in the HTTP redirect chain. You need a tool like Live HTTP headers for Firefox to see them. If you make a browser request to ad.yieldmanager.com or yieldmanager.com you get nothing. The WhoIs record shows that Yahoo owns the domain.  If you go to www.yieldmanager.com you’ll be redirected to the HTTPS version of the page https://www.yieldmanager.com which Firefox will refuse to display because it has an invalid security certificate. You’ll have to go through several dialog screens in Firefox to make a security exception and see the site. It will show a login screen for RightMedia (an advertising exchange bought by Yahoo) but no information for consumers. If you hit the home page of RightMedia, you’ll see a tiny link to “privacy” on the bottom which takes you to the privacy policy of the corporate site. At the end of the third paragraph, there is a link to the privacy policy on ad.yieldmanager.com. If you click on that, you’ll finally get to find out what RightMedia collects from you and what they do with it. Well, sort of.

The privacy policy says “Non-personally identifiable information is automatically sent to the Yield Manager technology by a user’s web browser. This information includes the date and time of the ad request, the user’s Internet Protocol Address, browser type, and the web page that the user is visiting.” How can they be sure that the web page I’m visiting doesn’t have personally-identifiable information? Does my Facebook URL personally identify me? Do my LinkedIn, blog, FriendFeed, Twitter URLs personally identify me? You bet they do. I have no idea whether RightMedia operates on those sites but if they do and if they can see links that can be tied to my account I’d certainly argue that their privacy policy is misleading.

Oh, one more thing. Do you know what’s the title of the privacy policy? “CONFIDENTIAL” Huh? Somebody better fix that.

<head>
<title>CONFIDENTIAL</title>
<link rel="stylesheet" href="http://my.yieldmanager.com/styles/css.php" type="text/css">
<link rel="shortcut icon" href="http://my.yieldmanager.com/images/default_icon.ico" type="image/x-icon" />
<script language="JavaScript" type="text/javascript">if (top.location != location) { // if in frame
	top.location.href = document.location.href; // break out!
}
</script>
</head>

Do you think most consumers care about their information being potentially misused? I haven’t seen the studies but I hope lots do. How many of them do you think will be able to find out that yieldmanager.com/RightMedia exist and get around to finding the privacy policy and understanding it? Right.

In a world where it’s hard to even know who these third parties are let alone what one of the them says they are going to do with your information, how can anyone one be certain whether they’ll actually do what they said they would? Is it too much to ask that consumers have an easy way to (a) find out who collects data as they browse the Web and (b) have the privacy policy of those entities at most one or two clicks away?

1. Get scale in usage and aggregated data
2. Redefine search via the social graph
3. Colonize the Web through Facebook Connect and OpenStream
4. Sell targeted ads everywhere

It is clear that the two companies are on a collision course and will both look for help and leverage in the upcoming battle. Beyond the well-identified brand advertising + search + communication tools + social graph battle lines there are two that merit a bit more analysis and some wild guesses:

• The real-time Web is becoming more important. That’s why both went after Twitter and, of course, why Twitter rebuffed both. (Nothing like a horsy race, especially when you don’t need to exit. Yes, Twitter knows they have to manage hosting/traffic costs.) Now both FB and Google are busy adding capabilities around real-time content contribution/sharing/search.
• More general search. If FB is to redefine search, it needs to be able to handle a reasonably set of search queries. Google’s core technology and reach have an overwhelming advantage here. FB has great data but even at their scale it is very difficult to create a great user experience for search. People come to a search box with diverse intent. Therefore, intent discovery is one of the most important and hardest problems for a search engine to solve. Through its richer UI and tracking capabilities FB has an advantage is discovering intent. However, if the intent and/or the search query cannot be satisfied easily within the agregated dataset of friend activity there is a problem. Also there is the expert gap–my friends may not be the experts. Wild guess: Facebook may partner with Microsoft, which is eager to give Bing more momentum, or another search player to create a more compelling search experience within FB.  Or they may really redefine search to establish a clear psychological contract with users about what FB search is all about. Either way, it will be interesting to watch.
• Desktop access. I was excited when Facebook acquired ParaKey. It was a brillian strategic move, Facebook’s Secret Plan v2.0. Desktop data and flexible offline access to online services is very important. Google clearly has the leadership now with Google Gears and Chrome. Google Apps help indirectly by moving the data to the cloud. I am disappointed that for two years we haven’t seen anything come out the Parakeyacquision. Blake Ross and Joe Hewitt are still at FB. Joe is doing some very cool work on the iPhone app and something secret (hopefully desktop-related). Blake’s been so busy he stopped paying his hosting fees.

Ultimately, the Net doesn’t want to be owned by anybody. We are witnessing the wearing off of Google’s goodwill as their hold on non-brand advertising solidifies. The same is likely to happen with Facebook if they aggregate tons more user data and continue to make it difficult to export/share/operate.

So, who’s going to balance out the power of Facebook and Google? One obvious answer is Microsoft. As the underdog in the online space, the company will be willing to play by new rules. When threatened MS rallies and responds well. Well, at least they did. A lot of people have left. Next is perhaps an unlikely candidate–the Mozilla Foundation. With its significant browser share Mozilla has a huge role to play in the evolution of data ownership, privacy and advertising. In fact, this will be the topic of my conversation with CEO John Lilly tomorrow. Apple has a role to play.

About to land. Enough for now.

A couple of weeks ago, we revised our terms of use hoping to clarify some parts for our users. Over the past couple of days, we received a lot of questions and comments about the changes and what they mean for people and their information. Based on this feedback, we have decided to return to our previous terms of use while we resolve the issues that people have raised.

That one was easy to predict.

Three big trends define how social infrastructure will evolve in the next few years: the opening up of communities, the battle for ownership of user data and the introduction of social economics.

Well, Facebook just drew the line in the sand in the battle for control over user data. The Web is aflame with stories about the recent change in the Terms of Service.

The short summary is that if you close your account, Facebook owns your data and can do anything it wants with it, including sub-licensing it. Imagine you are a professional photographer who uploads lots of pictures on Facebook. Oops, you just lots ownership of those pics.

There are some arguments as to why Facebook updated the ToS having to do with their ability to control how user data is shared and used across various services using Facebook Connect and other mechanisms. While this may be true, I don’t buy the blanket change to ToS as proportional. Lawyers need to go back to work and find a more precise way to eliminate reasonable risk to Facebook without essentially forcing Facebook users to lose control of their data.

What I find particularly offensive is that, from what I understand, users have no choice over the matter. Under the current terms of service, if I were to protest and close my account then they’ll own my archived data. But if I stay and close my account some years from now, well, they might own my data. Hmm……

This is an important step in the global battle for data ownership. If Facebook gets away with this, I expect to see these types of policies quickly spread through other services. I don’t expect this to happen, however. My prediction is that this will be another Facebook Beacon-type experience and another lesson to the company about the level of sensitivity and care with which a large online player must behave.

Side note: this ties to the broader issue of forced upgrades in SaaS, PaaS and IaaS (the broad set of cloud-based services). It shows how not just software companies but normal users are exposed to random, retroactive changes to ToS. Another relevant thread is changes to privacy policies–see FTC’s recent update to their behavioral targeting principles, which specifically calls out retroactive privacy policy updates.

Congrats to the Animoto team. They’ve built a very cool service and deserve to get a little breathing room.

The AIR / Flex 3.0 launch is a big deal for three reasons:

• The AJAX ecosystem is heavily fragmented
• Flex 3 is a big step forward
• AIR is a disruptive innovation enabler

There are just too many pure AJAX frameworks out there. This may be great for some abstract notion of innovation but is pretty bad for market adoption. It causes confusion and friction. Developers hate that. They want to feel like they are developing skills for the next cool technology which will be around for a decade. The tooling is also pretty immature, although my friend Paul Colton @ Aptana is doing some pretty cool work. I have also seen a number of new startups trying to make things better in this space but for now the situation is not great. After initially missing the AJAX boat in a big way, Adobe has recently gained traction in repositioning from Flash vs. AJAX to Flash vs. DHTML, both of which use AJAX. As proof, I’ve seen more startups mixing Flash/Flex with DHTML/AJAX.

There seems to be a rule that really big software companies get things right by version 3.0. That has been the case for years with Microsoft and now Adobe may be large-enough to join the list. Flex 1.0 shipped late and left a lot to be desired, which made Flex 1.5 more or less a requirement. Flex 2.0 was a big step forward but there was much more left to do to build a great RIA platform. Flex 3 steps up the innovation pace on a number of fronts. Better developer tools. Improved frameworks. Smaller application size. Better data handling throughout. And the Flex SDK as well as the BlazeDS remoting and messaging services are now open-source. Everything is in place to accelerate Flex adoption. There is a massive shortage of Flex developers in the market, always a good sign for a platform. The main challenge for Adobe remains monetizing the platform. I’m not 100% on board with the Flex Builder tool as the main revenue driver although the team recently added some great people from the ColdFusion team who really understand Web developers. For examples on what Flex can do, check out the app showcase. Warning: you may spend more time there than you previously  imagined.

AIR will become the biggest enabler for media-heavy Web applications. To get an idea about what I mean, check out the Allurent Desktop Connection video. This only scratches the surface of what one can do with a cross-platform, always on, richly interactive application with Net access and plenty of local storage. Adobe isn’t the only company who gets this–Google has plans based on Google Gears and Facebook has also moved in this direction with the Parakey acquisition–but Adobe has so far executed the best. Not surprisingly, Microsoft remains paralyzed by the Web <-> desktop strategy tension.

AIR is truly disruptive–in Clayton’s exact sense of the term–for two reasons. First, it given Web apps access to the desktop. Second, it does so while subverting the native operating system installers (using the Flash player executable as a conduit) which gives Adobe and its ecosystem substantial freedoms. One caveat–security. Adobe has done a pretty good job with the Flex 3 security model but it hasn’t been extensively tested in the wild. Also, the anti-virus and anti-spyware vendors have often been trigger happy in the past to blacklist totally legitimate software. I’m not expecting them to blacklist the AIR runtime but who knows how they’ll treat some of the apps… I hope the Adobe platform marketing team and evangelists have spent enough time with the security vendors. This is something I advised them to do at the Engage event last year based on some bad experiences from our portfolio.

With this launch, Adobe cements its platform leadership in Web 2.0 and makes life easier and more interesting for millions of developers and designers. A shout out to all my friends at Adobe who worked hard to make it happen and to everyone who put skin in the game during the beta process.

An interesting twist to the model comes when you combine it with social incentives. The latest arrival (I saw the news in PaidContent today) is ArcheType Media / SocialVibe. They let brands sponsor your social network profile. Here is an example (on their site and on MySpace). The bait is doing good and getting goods.

Max’s first post covers the launching of a social developer platform with ten common-sense rules. (My only question is: why always ten rules?) Two key takeaways:

• A platform will only succeed if it makes its developers successful. I wholeheartedly agree and hence my Ecosystem Test.
• Be humble, open and embracing of the developer community. That’s how we built the developer community at Allaire. It was and still is, one of ColdFusion‘s greatest assets to this day.

The second post, really an essay, goes deeper into developer incentives in social networking platforms starting with the hypothesis that “designing a social platform is in some ways similar to designing a competitive multi-player game.” Max offers a Darwinistic/behavioral view of ecosystem management which makes sense in a social environment where multiple developers/apps compete for the attention of consumers.

Perhaps because of the Slide perspective, the incentive analysis is primarily focused on valuable distribution.

It’s worth pointing out that ultimately, until non-advertising business models are devised for social applications (and probably even after they are) valuable distribution (reach + frequency) is going to be the main underlying goal for all developers, commercial and otherwise.

My take is that non-advertising business models are just around the corner. The key is social advertising/social commerce where one is leveraging the distribution power of millions of consumers. There the developer controls a social platform puts in place are somewhat less important.