I am roaming the halls of Google I/O 2013 and wondering whether Google’s platform passes the ecosystem test.
… no platform has become hugely successful without a corresponding ecosystem of vendors building significant businesses on top of the platform. Typically, the combined revenues of the ecosystem are a multiple of the revenues of the platform.
So much activity but what’s the combined revenue of the businesses building on top of Android, Chrome & Apps?
I’ve been asked to explain how online ads are delivered many times and every time I’m surprised by the complexity of covering even the most basic elements of how ads appear on Web pages. Since Wikipedia’s article on ad serving is not much help, I’ll try to explain one common way ads are delivered using a concrete example.
Side note: this is not how Swoop works. At Swoop we use a much simpler and more efficient model because we’ve built an end-to-end system. This eliminates the need for lots of different systems to touch (+ cookie + track) users. It also allows us to create deeper and more relevant matches by placing Swoop content not in arbitrary fixed slots but in dynamic slots right next to the part of a page it relates to. If you are looking for an analogy, think about Google Adwords on SERPs. It’s an end-to-end system where Google has complete control over ad placement and no ads are shown if there are no relevant ads to show.
Tools of the trade
If you want to know how adtech works, there is no better tool than Ghostery. Ghostery was created by my friend David Cancel and, later, when he was starting Performable (now part of Hubspot), my previous startup, Evidon, became the custodian of the Ghostery community. Ghostery will show you, page-by-page, all the different adtech players operating on a site. For example, on Boston.com’s sports page, there are 35 (!) separate adtech scripts running today.
Ghostery will show you what is happening but not how it happened. If you are technical and want to understand the details of how ad delivery works, there no better tool than a debugging proxy such as Charles or Fiddler. Just be prepared for the need to use code deobfuscators. If you don’t have time for wading through obfuscated code and you really want to know what’s going with your sites(s) or campaign(s), it is worth taking a look at Evidon Encompass. It’s an advanced analytics suite built on top of the Ghostery data.
The example
The example we’ll use is the arthritis page on Yahoo!’s health network. We will focus on the leaderboard ad at the top, which is a Celebrex (arthritis drug) ad from Pfizer.
What Yahoo! sent the browser
The initial response my browser got from Yahoo!’s server included the following chunk of HTML about the leaderboard ad unit, which I’ve formatted and added comments to. (Not sure what’s up with the empty lines that WP is adding to the bottom of the gists–they’re not on GitHub).
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
This content was mostly likely emitted by the Yahoo publishing system without direct coordination with the Yahoo ad server but instead using conventions about categories, page types, etc. and hence parameters like rs=cnp:healthline that you see on the URLs.
Display advertising units use standard IAB formats. In this case, we are dealing with a 728×90 leaderboard unit. The DIV with id yahoohealth-n-leaderboard-ad sets up the location where the ad unit will be displayed. The DIV under it serves the dubious function of controlling some styling related to the ad content.
Beyond this there are two things going on here. The first is the delivery of the ad script and the second is the delivery of a tracking pixel via a tracking pixel script.
Tracking pixels
Tracking pixels are 1×1 invisible images served from highly-parameterized URLs. They are not used for their content but for the request they generate to a server. The request parameters are used for record-keeping and the response could be used to cookie the user, though this did not happen in this case.
The tracking pixel is delivered via the script inside the <center> tag. It’s contents are shown below.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
The script uses the JavaScript document.write function to write some HTML into the page. In this case the HTML is for an invisible image (display: none, height: 0, width: 0) whose URL is that of the tracking pixel, whose unencoded value is the long URL:
As you can see, lots of data getting sent, most likely to record the impression opportunity parameters.
Yahoo! ad delivery script
There are two ways to deliver an ad unit. The preferred way is via a script. If scripting is disabled in the browser, however, Yahoo doesn’t want to lose the ad impression opportunity and so there is the <noscript> option to show the ad in an iframe, probably as an image.
The code for the Yahoo! ad delivery script, which comes from the Yahoo! ad server, is shown below with reformatting and comments from me.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
AdChoice came about in 2010 as the online advertising industry’s response to FTC pressure to reign in some poor privacy practices and provide consumers with more transparency and choice when it comes to interest-based advertising, a.k.a., behavioral targeting (BT in adtech parlance).
The AdChoice icon is a triangle with an i in it. Its color can vary. Yahoo!’s is gray (). Next time you see an ad with it, click on the AdChoice notice. You should see information about who targeted the ad at you and get some options to opt-out of interest-based advertising. We started Evidon back in 2009 to bring more transparently to adtech and we helped create AdChoice. Evidon is now the leading independent player in this space.
In the case of the Celebrex ad from our example, the AdChoice icon is tied to a very long URL:
If you click on the AdChoice icon, Yahoo! will record information about which ad you are selecting to learn more about and then redirect you to the page at the end of the URL, which is the Yahoo learn more about this ad page. The long URL is just for bookkeeping.
BTW, the reason why you don’t see AdChoice notice with Swoop is because Swoop does not do any behavioral targeting at this time. Still, because we want to make it clear that Swoop is serving content, you’ll see our logo on our units.
Cache busting
After the AdChoice notice setup comes a line of script that creates a random number. This is used for cache busting.
A cache-buster is a unique piece of code that prevents a browser from reusing an ad it has already seen and cached, or saved, to a temporary memory file.
Adding a random number to a URL does that nicely.
Google ad delivery script activation
The following script tag loads Google’s ad delivery script from ad.doubleclick.net. We will look at this later on.
Yahoo impression tracking
Remember how Yahoo already fired one tracking pixel to record the impression opportunity. Well, here, at the end of the script they are going to fire another tracking pixel but this time the purpose will be to record the impression of the Google ad. As before, you can see lots of data being passed.
As before, in the case that the browser does not have JavaScript enabled, Yahoo doesn’t want to miss the opportunity to deliver an ad, which is why they have the option to display the Google ad as an image.
In that case, Yahoo is also positioned to capture the click and then redirect to Google. This is achieved by wrapping the image (<img>) in a link (<a>). Getting click feedback data would be valuable for Yahoo as it allow is to optimize better. If the unit is sold on a cost-per-click (CPC) basis, then getting click data is a requirement for good record-keeping.
Google/DoubleClick ad delivery script
It’s time for us to take a look at what Google’s ad delivery script does. Alas, the guys at Google don’t want to waste bandwidth so they’ve packed everything into a single unreadable document.write call. You can scroll to the right for a very long time…
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Here is what Google is actually trying to write into the HTML page (with my comments added):
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Don’t worry about the volume of code. There are basically two things going on here: delivering a Flash ad and lots of third party ad verification.
Delivering a Flash ad
Flash ads are richer and more interactive but there are browsers where Flash ads don’t do so well. The first part of the Google/DoubleClick ad delivery script is about carefully determining whether a Flash ad unit can be used and falling back to images otherwise. As before, all clicks are tracked via redirects.
Third party verification
We saw Yahoo! attempting to fire three types of tracking pixels: (a) for impression opportunities, (b) for actual impressions and, in the case of no scripting, (c) for clicks. This is to help optimize the performance of Yahoo!’s ad network. This is first party verification. Google/DoubleClick does the same with its own systems.
Third party verification happens when the advertiser asks the delivery network, in this case Google/DoubleClick, to include additional verification tags (scripts) to prove that the campaign is delivered based on its configured parameters.
In the case of this Celebrex campaign, Pfizer is using four separate verification vendors. At the top level we have only Nielsen NetRatings and DoubleVerify, however NetRatings’s script loads AdSafe as well as Facebook in the pattern we are familiar with: a script that writes out <script> tags to load more scripts.
Putting it all together
Let’s try to piece together the requests that allow this one single ad unit for Celebrex to appear:
Yahoo ad unit delivery script
Google ad delivery script
Flash movie
??? (not easy to track Flash traffic)
Nielsen NetRatings tracking script
AdSafe pixel
Facebook iframe
Facebook tracking
??? (did not analyze)
DoubleVerify script
Tracking script (like a pixel)
Yahoo impression tracker
Tracking pixel
Yahoo impression opportunity tracker
Tracking pixel
All in all, 13 separate HTTP requests to 6 separate companies, not counting redirects and cacheable scripts. With this much complexity and independent parties doing their own accounting, it’s no surprise the display advertising value chain is in a bit of a mess right now.
One of the fastest ways for a startup to grow has always been to ride on the shoulders of a successful platform: from Microsoft/OSS in software to AWS in cloud computing to iOS/Android in mobile to Facebook/Twitter/Pinterest in social to IAB/Google in advertising and the many SaaS players. Betting on a platform focuses product development both because of technology/API choices and because of the automatic reduction in the customer/user pool. Also, platforms that satisfy the ecosystem test help the startups that bet on them make money. That is, until they don’t.
I’ve been involved with three startups that have been significantly helped by platforms initially and then hurt by them. Two cases involved Microsoft. One case involved Twitter. The first time it happened, our eyes were closed and it hurt. It prompted me to learn more about how platform companies operate and how they use and abuse partners—companies small and large—to help them compete with other platforms. The basic reality is that platform companies will do whatever it takes to win and they typically don’t care much about the collateral damage they cause.
Just like hacking fast & loose, which accumulates technical debt, accelerating the growth of a startup by leveraging a platform may come with substantial platform risk.
Note: links to undocumented anti-patterns will take you to the main list.
Startup Anti-Pattern: Platform Risk
What it is
Platform risk is the debt associated with adopting a platform. Platform risk becomes an anti-pattern when three conditions are met:
The platform dependency becomes critical to company operations.
The company is unaware of the extent of the risk it has assumed.
There is increased likelihood of adverse platform change.
Here are the top 10 sub-patterns of platform risk hurting startups that I’ve seen:
Lock-in. Startups that adopt a closed platform can be locked into their choice typically for the duration of the company’s life. This is not a problem until the need arises to support another platform. At that point the time & cost associated with the work could be substantial, especially if the core architecture was not designed with this in mind. In many cases, it is cheaper to start from scratch.
Forced upgrades. When software came in boxes, if you didn’t like the new version or if it was incompatible with your own software, you and your customers did not need to upgrade. You could take the time to make things work and upgrade on your own schedule. In the platform-as-a-service world, you do not have this option. Instead, forced upgrades are the norm. You have to deal with them on the platform vendor’s schedule, which may be quite inconvenient and costly. You do not have the option to ignore the update. Vendors vary widely in how they manage their partner ecosystems with respect to forced upgrades. Google has been pretty good when it comes to its APIs and has acted like a not-so-benevolent dictator when it comes to non-API-related behaviors of services such as search and advertising. Facebook and Google have both been accused of manipulating the behavior of their systems to force businesses to spend more money in their advertising platforms. In the case of Facebook, the issue has been pay-to-play for likes. Google has come repeatedly under fire for manipulating the search user experience to (a) shape traffic away from large publishers it competes with and (b) reduce advertiser choices and drive more ad dollars to AdWords. If your business depends on SEO or SEM, these changes can be very significant. The former CEO of a large advertising agency once summarized this as “Google giveth and Google taketh away.”
Forced platform switch. A forced platform switch usually comes as a side effect of platform vendors playing turf wars. For example, Apple severely hurt Adobe’s Flash platform as a way to limit write once, run anywhere options in mobile, thus also slowing Android’s adoption a bit. Thousands of small game & other types of content developers in the Web & Flash ecosystem were affected and had to either abandon iOS development or find new costly talent.
The partner dance. The partner dance is most commonly seen in enterprise software. It was popularized by Microsoft. As one former MS exec described it to me: “first you design your partners in and then you design you partners out.” During the design-in phase, a platform vendor partners with and, in some cases, spends meaningful resources helping an innovative startup company with solutions that compete with the solutions of another platform vendor. As the platform company’s own product roadmap matures, it designs its partners out starts directly competing with them.
Swinging. Swinging is a variation of the partner dance where rather than competing directly with a startup, the platform vendor partners with one of the startup’s competitors. Some years ago I was on the board of a European company that was Microsoft’s preferred partner in a fast-growing market. After winning against much bigger players such as EMC and IBM, the startup convinced MS that there was a big business to be built in this market. At that point MS promptly terminated the startup’s preferred status and partnered with a much bigger competitor. We were expecting the move: Microsoft now wanted to move hundreds of millions of dollars of its platform products in this space and the startup, despite closing significant business, could not operate at this scale. The Facebook/Zynga saga is an example from the online world.
Hundred flowers. The name of this sub-pattern comes from the famous Chairman Mao quote “Let a hundred flowers blossom.” Mao fostered “innovation” in Chinese socialist culture—open dissent—and then promptly executed many of the innovators. It seems that Twitter, Facebook and other social platforms have studied the Chairman quite well, judging by how efficiently they have moved from relying on the adopters of their APIs for growth and traffic to restricting their access and hurting their businesses. The prototypical example is Twitter driving much of its traffic from third party clients and then moving against them.
Failure to deliver. Startups pick platforms not just because of their current capabilities and distribution but also because of their expected future capabilities and distribution power. If the platform does not deliver, the startup’s ability to execute can be significantly hampered. One of the most common use cases of this sub-pattern relates to open-source platforms where the frequent lack of a single driving force behind a product or service could lead to substantial delays. At various points, teams I’ve been involved with have had to dedicate significant resources to accelerate development of OSS, e.g., Apache Axis, which turned out to be the most popular Web services engine, and Merb, whose adoption turned out to be a bad platform decision for my startup. It’s rewarding work but it also usually is plumbing work that generated little business value.
Divergence. Divergence is a form of failure to deliver rooted in a change of strategic direction of the platform. Divergence can be very costly over time and difficult to diagnose correctly because it happens very slowly. The analogy that comes to mind is of a frog in a pot of water on the stove. I knew a startup with a neat idea on how to provide significant value on top on the Salesforce platform APIs. They just needed one improvement that was “on the roadmap.” The improvement remained on the Salesforce roadmap for more than two years as the startup ran out of money. The hidden reason was that Salesforce had grown less interested in the use case. Another Salesforce-related example is the recent hoopla about the unannounced changes in Heroku’s routing mechanism, which cost RapGenius a lot of money. In this case, the reason was Heroku moving from being a great place to host Ruby apps to being a great place to host any apps and in the process becoming a less great place to host Ruby apps.
Poison pill. A platform choice made years ago could turn out to be a poison pill when it comes to selling your company to another larger platform vendor. As an example, consider the case of Google buying a company whose products are built on Microsoft’s .NET platform or Microsoft buying a SaaS collaboration solution that runs on Google Apps. Alas, most startups do not think about the exit implications as they make platform decisions early on.
Exit pressure. Platform companies may sometimes exert substantial pressure on partners when they want to acquire them. When Photobucket did not want to sell to MySpace they somehow experienced “integration problems” with MySpace, which affected their traffic. The sale soon completed. This goes to show that talking softly while controlling the source of traffic tends to deliver results. This week we learned that Twitter’s acquisition of social measurement service Bluefin Labs involved some threats, which must have been perceived as credible since 90% of Bluefin’s data came from Twitter.
Diagnosis
Good diagnosis of the platform risk anti-pattern is exceptionally difficult because it requires predicting the future path of a platform as well as those of the platforms it competes with. The basic strategy for diagnosing this anti-pattern involves three parts:
Investment in ongoing deep learning about the platform and its key competitors. This should cover the gamut from history to technology to business model to the personalities involved.
Developing relationships with industry experts with a deep perspective of the platform, whose businesses, like telltales on a sailboat, in some way provide leading indicators of platform change. You don’t want just smart people. You want people with proprietary access and data. For enterprise software try preferred channel partners. For open-source software try high-end OSS consultants. For advertising, find the right type of agency.
Network into the group(s) responsible for the platform, both involving people currently on the job as well as senior people who’ve recently left. This latter group has been the most helpful in my experience.
Ignorance is the most common anti-pattern that makes the diagnosis of platform risk difficult.
Misdiagnosis
A common misdiagnosis stems from failure to consider the effects of competitive platforms on the platform a startup has adopted. Sometimes it is these competitors’ actions that trigger the negative consequences, as was the case of Apple’s decisions hurting the Adobe Flash developer ecosystem.
Refactored solutions
Once diagnosed, the key question regarding the platform risk anti-pattern is whether anything at all should be done about it. Most companies choose to live with the risk, though very few fully use the diagnosis strategies to get an accurate handle of the net present value of the risk.
The refactoring of platform risk is typically very, very expensive as well as very distracting. For example, some would argue that Zynga’s fight on two fronts (a) trying to refactor its platform risk related to Facebook and (b) ship new games is what hurt the company’s ability to execute.
In the case of platform risk, prevention is far better than any cure. In the words of Fred Wilson (an investor in Twitter): “Don’t be a Google Bitch, don’t be a Facebook Bitch, and don’t be a Twitter Bitch. Be your own Bitch.” Being your own bitch doesn’t mean not leveraging platforms. It means getting in the habit of doing the following three things in a lightweight, continuous process:
Explicitly evaluate platform adoption decisions, once you have sufficient information. Having sufficient information usually involves more than reading a few blogs. For example, at Swoop we recently had to make a search platform choice. We decided to go with Elastic Search but not before I had talked to the company, not before Benchmark invested significantly in ES, and not before I’d talked to friends who ran some of the largest ES deployments to get the lowdown on what it was operate ES at scale.
Invest the time to learn about the platform and develop the relationships that would help you have special access to information about the platform. Here is my simple rule of thumb with respect to any platform critical to your business: someone on your team should be able to contact one of the platform’s leaders and get a response relatively quickly. This is especially important if you are dealing with new or not super-popular open-source projects. The best way to achieve this is to think about how you and your business can help the platform.
Every now and then spend a few minutes to honestly evaluate your company’s level of platform risk and think about how you’d mitigate it and when you’d have to put mitigation in action.
Remember, the goal is not to eliminate platform risk. You cannot do this while at the same time taking advantage of a platform. The goal is to efficiently reduce the likelihood of Black Swan-like events related to the platform hurting your business. If you understand the mechanics of how platforms operate and how platform risk accrues, you will be able to predict and prepare for events that take others by surprise. These are sometimes the best times to scale fast and leapfrog competitors.
When it could help
Betting on a platform can be hugely helpful to a startup, despite some level of platform risk. There is never a benefit from platform risk increasing to the anti-pattern level.
***
The startup anti-pattern repository is work in progress helped immensely by the stories and experiences you share on this blog and in person. Subscribe to get updates and share this anti-pattern with others so that we can avoid more preventable startup failures.
Listening to a startup pitch a few weeks ago I had to exercise effort not to shake my head in disbelief. The presenters were describing the products and business model of an established company I was very familiar with and doing it with what seemed to be blatant disregard of reality. They forgot to mention an entire product line that directly competed with their would-be product. They claimed the company had a different business model than the one it did. They misrepresented its scale, number of customers, etc. The obvious question was whether they were knowingly misrepresenting their competitor or whether they were confused or simply unaware. After the Q&A it became clear that they were just ignorant.
The event took place during a startup competition. What struck me was that, without exception, following any one startup’s presentation, at least one of the reviewers who was an expert in the startup’s space would point out some way in which the entrepreneur(s) had either forgotten to investigate something of critical importance to their business or deeply misunderstood it following a typically brief investigation. The startups were gearing to spend time & money based on flawed conclusions. They had all taken on significant risk because of their ignorance.
Note: links to undocumented anti-patterns will take you to the main list.
Startup Anti-Pattern: Ignorance
What it is
Ignorance is not knowing what you don’t know. It may be the most common startup anti-pattern. It should not be confused with making an explicit, considered decision to ignore something (knowing what you don’t know).
Ignorance hurts startups in countless ways, of which the following are pretty common:
Ignorance creates an invisible bias in decision-making, which often results in significant time and resources being wasted.
Ignorance sends a very early negative signal that turns experienced talent away. Would be co-founders, executives, employees, investors, board members and advisors may determine it is not worth their time to engage, especially if they see ignorance co-occurring with anti-patterns such as arrogance and unrealistic expectations.
Ignorance is self-perpetuating: it attracts equally or more ignorant talent to the company.
When equally or more ignorant investors join an ignorant company, they can experience a very rapid & destructive “falling out of faith” when they have to come to grips with reality. I have seen several companies hit the wall at high speed because expected financings did not come together as planned for this exact reason.
Diagnosis
Ignorance is easy to diagnose but it takes work. There are two main strategies.
The first strategy is introspective in nature. It involves comparing observed outcomes with clearly recorded expected outcomes and then analyzing the root cause of the difference. Ignorance creates an omitted variable bias (OVB) in decision making and will likely cause reality to not match expectations. If there is no obvious known root cause for the observed difference the likelihood of OVB increases and ignorance should be the suspect. The benefit of this approach is that anyone can practice it. The main problem with it is that it is a lagging indicator: it detects problems after they have already occurred, which is inefficient. The biggest reasons why this diagnostic cannot be applied are: (a) that expected outcomes are not clearly recorded or (b) that an incorrect root cause is identified (see the scapegoat anti-pattern).
The second strategy is a leading indicator. The idea is simple: seeks external feedback to diagnose and eliminate ignorance early, before it costs you. External feedback can come through materials or through people. In the latter case, the trick is to be humble and genuinely interested in people’s opinions or you may not get them or, worse, you may get an artificially positive opinion whose goal is to get you off their back. Another thing to watch out for when talking to experts is the mentor whiplash anti-pattern.
Arrogance and confirmation bias are the most common anti-patterns that make the diagnosis of ignorance difficult.
Misdiagnosis
When exogenous forces such as luck and timing affect the success of a startup, ignorance is often confused with genius, vision, and perseverance. As the saying goes, sometimes people can do something just because they don’t know it couldn’t be done. Statistically-speaking, this is not a good strategy for startup success.
Refactored solutions
Once diagnosed, the refactoring of the anti-pattern very much depends on the nature of ignorance involved and its root cause.
Some methodologies can fix the symptoms by making it very difficult to remain ignorant. For example:
Agile development methodologies make it difficult to hide issues related to engineering execution.
Fixing the symptoms is not the same as identifying and fixing the root cause. If the root cause is simply lack of knowledge or mis-understanding it is relatively easy to fix. If the root cause is deeply character-related, e.g., a fundamental lack of curiosity or excessive narcissism, then the fix typically has to involve finding more suitable roles for the people involved or transitioning them out of the company.
Effective startup execution requires agile handling of uncertainty. Eliminating ignorance has a cost, which needs to be considered relative to its potential benefit. A particular form of the analysis paralysis anti-pattern involves spending far more time than it is worth attempting to diagnose and eliminate ignorance.
When it could help
As with most things, ignorance happens on a scale and there are many cases in a startup’s life when measured amounts of ignorance can have positive effects, at least temporarily:
Ignorance can facilitate focus by artificially simplifying planning & execution.
Ignorance can be motivational by creating artificial certainty and hiding potentially bad, demotivating news.
Ignorance can bring resources to the company, e.g., investment capital from investors who might be put off by reality.
***
The startup anti-pattern repository is work in progress helped immensely by the stories and experiences you share on this blog and in person. Subscribe to get updates and share this anti-pattern with others so that we can avoid more preventable startup failures.
I’ve been doing a lot of work on startup anti-patterns recently so it seems fitting that I look at other things through that lens.
Every few months I fall into the same debugging anti-pattern:
I start using a debugger to track down a difficult-to-pin-down issue.
I set up a watch expression to look at some complex data structure.
The debugger evaluates the watch expression outside its intended scope, which (sometimes) creates a side effect that cannot be observed directly at the time it happens.
Yesterday, it happened when I was debugging a VBA script that’s part of my automation of tax preparation. Don’t ask: it seemed like a good idea a long time ago and is probably the main reason I occasionally use my old Windows machine.
The VBA Dictionary class will auto-create elements for any key it doesn’t have a value for. Therefore, a watch expression such as:
myClass.myDictionary(myKey)
can easily create a Nothing value with an Empty key when myClass is in scope and when myKey is a Variant not in scope.
The previous time I experienced a similar problem was in Ruby when the watch expression had a side-effect on a cache whose logic I was trying to debug.
Eventually, I hope to develop a sixth sense for this debugging anti-pattern. In the meantime, I don’t like my options: (a) not use a debugger or (b) live with paranoia about watch expressions.
Alternatively, I could embrace functional programming languages without side effects. I wonder what their debuggers are like…
Following my post on the value of startup anti-patterns, with the help of readers of this blog and friends from the startup ecosystem, I put together a list of the more common ones that I encounter. The list used to be here but I moved it to a page on startup anti-patterns for easier linking and sharing.
The TechStars Boston spring startups are looking very promising. At the mentors dinner tonight I had conversations with several of the companies and a common question came up: “What makes startups succeed?” My usual answer to this question is along the lines of “strong teams, hussle, luck & timing.” While in my experience these are the things that actually, casually make startups successful, it’s not a very satisfying answer to the question because luck and timing are not in a startup’s control.
Driving home, I was thinking about this and about a better way to share what I’ve learned about startups over the years as an entrepreneur, angel investor and venture capitalist. I’ve been lucky to have had access to the inner workings of dozens of startups through their lifecycles, often from founding until exit (good or bad). Still, beyond “strong teams, hussle, luck & timing” I don’t know what makes startups succeed. Maybe I’m not as smart as the people who confidently talk about the patterns of behavior that make startups successful. Maybe I’m in too much agreement with Guy Kawasaki about why good strategies fail. Who knows?
I do, however, have a pretty good idea about what makes startups fail. There is much more data about startup failure, especially if, you could see not just the external spin and the post-rationalizations but real, unadulterated startup stories unfolding over time. So, while the Law of Large Numbers cannot help identify strategies for startup success, it can definitely help identify strategies for startup failure that are repeated over and over again.
An anti-pattern “is a pattern … that may be commonly used but is ineffective and/or counterproductive in practice.” Commonly observable strategies of startup failure are startup anti-patterns. This is the startup community’s form of collective insanity, using the definition that “insanity is repeating the same mistakes and expecting different results.” I think this has a lot to do with the bias of talking about what makes startups successful as opposed to what prevents startups from failing. For better or worse, it is only the startups that repeatedly don’t fail that have the chance to take advantage of luck & timing.
I don’t mean failing in the lean startup sense of always making new mistakes quickly & cheaply. The process of learning through small failures is similar to the process of walking through controlled falling or to getting a few bruises and bumps on the playground. It’s good stuff and there is plenty to laugh about at the end. No, when I say failing I am referring to a startup wasting significant time & resources, to walking becoming tripping, the controlled fall becoming uncontrolled, the trip to the playground ending up in the emergency room.
Over the next few months I will document some of the key startup anti-patterns I see frequently. I’d love to learn about your experience with them. Subscribing is the easiest way to ensure you get the updates.
In school, I used to pick up new programming languages for fun. I could take my time to learn & explore them in whichever way felt right, without pressure. I implemented for fun and the code rarely ended up in production. By contrast, at work, I pick up a new language because something urgent & important is best done in it and, typically, the time between learning and production deployment is less than a month. Going through this several times over the past two years at Swoop (most recently with R) I think I’ve finally cracked the code on how I can help myself learn a new language in the fastest possible way.
I’ve lost count of how many programming languages I’ve written code in over the years (the most obscure one is probably ROBASIC). I think about programming using meta-patterns that transcend languages. When I encounter a new language, I go through the following phases:
Curious George. During this phase, which usually lasts only a few hours and involves semi-random exploration of tutorials, reference manuals, blog posts and StackOverflow posts, I get my bearings straight, find analogies between the new language and ones I’m more familiar with and choose my initial tooling.
Mario Bros. -> Super Mario Bros. In this phase, which usually lasts several days, I try to bring the meta-patterns I’m comfortable working with from familiar environments (the sewers of New York) to the new environment (the Mushroom Kingdom). It involves diving head first into advanced language features and building various utilities that I find lacking in the environment, e.g., debugging tools, all in the context of early prototypes of whatever it is that I need to work on.
Bull in a china shop. This is the phase where my noble goal of bending the language to the way I solve problems meets the harsh reality of me being a neophyte in its intricacies. The observable attributes of this phase are lower velocity, increased use of expletives and more time on StackOverflow. The amount of time in this phase varies. The “Eureka!” moments are fun but overall it’s a dip in the experience curve.
Singin’ in the Rain. With newly acquired knowledge and improved language-specific testing/debugging skills, the bull gently transforms into Fred Astaire. Coding is a lot of fun again. It’s time to go to production.
Obi-Wan Kenobi. Over time, the interaction with the new language improves the meta-patterns I use for problem solving. I tend to use less and simpler code using natural language idioms as opposed to generalized utilities & abstractions. It’s like changing from using the Force to allowing the Force to do things through you. It takes a long time to get here. More often than not, I never do.
In trying to optimize the way I learn languages, I focused on the obvious problem: the bull in the china shop phase. My guess as to the root cause of the problem was simple: I didn’t know what I didn’t know and it did have a tendency to bite me. Here are some examples of the types of advanced language features that have helped me in a big way but not without some very frustrating moments along the way:
In C++ template meta-programming turned out to be awesomely useful and powerful but rich in undocumented cross-platform compiler bugs at a time when my company was shipping on four different OSes.
In Java it had to do with dynamic bytecode generation for decorators of untrusted third party code in the days before J2EE was mature.
In Python a logically simple approach to behavior injection ran into edge cases where stateful decorators didn’t mix well with inheritance. To boot, it crashed the debugger, which made it all the more fun to fix.
In Ruby metaprogramming for dynamic domain specific language definition my approach ran afoul of the behind-the-scenes trickery of ActiveModel::Callbacks. To prove that Ruby is just like Python in all ways that matter, this was another situation where the debugger crashed liberally.
In JavaScript inside MongoDB, all hell broke loose when we tried to do some seemingly trivial map-reduce work. Forget crashing the debugger (there is none): it crashed the whole database server.
In R, whose best and worst quality is that it is designed by statisticians, it was custom environment chaining.
In reflecting on this after my recent experience with R, I felt stuck between a rock and a hard place. If I use advanced language features to gain productivity I risk wasting time debugging bizarre edge cases. If I do not use the types of language features that make it easier to map how I think about problem solving to a particular language I lose productivity. The obvious approach was to look for a way to discover the unknown unknowns sooner but I didn’t have an easy way to do this without creating waste in the agile sense. Even test-driven development didn’t help much as the majority of the problems that cost me the most time reared their ugly head deep into implementation.
What I now think is an optimal approach to learning a new programming language–at least for me personally–required two serendipitous discoveries. The first came because of some work we are doing at Swoop with thoughtbot. I’ve often preached that it’s important to invest in learning the intricacies of whichever testing framework you use so that you are able to express the way you think about testing more naturally in code. I thought I knew how to do this reasonably well but seeing those guys work challenged me to think about testing abstractions in a new way. The second serendipitous discovery came as I was trying to extend the testthat R package with some new primitives for the types of tests I wanted to write. I ended up hitting several unknown unknowns in that process but, perhaps due to the natural way tests isolate state & dependencies, it happened early and in ways where a solution or a workaround was easy to discover.
With the help of serendipity and reading the code of several testing frameworks, I’ve come to the hypothesis that either reading and extending existing test frameworks or building new ones (perhaps as a tool to go along with early prototype work) is the best way for me to both efficiently learn the advanced features of a new programming language and to efficiently discover in practice the unknown unknowns that will likely bite me. I expect the same to be true for other experienced developers but I would not recommend this approach for developers who are not at ease with abstract & pattern-driven problem solving.
I’m going to test this hypothesis with Lua, which I started learning tonight because of some Redis scripting work we are exploring. Here is the result of my first hour with Lua. I’m not trying to re-create TDD frameworks such as telescope or busted. (In fact, I’ll probably transition to busted as it is the most RSpec-like thing in the Lua universe and I find contextual test writing to be far easier and more enjoyable than xUnit-style test writing.) This was simply the fastest way for me to dive into learning the language as opposed to using the language. This approach also yielded the first unknown unknown: the curious fact that the built-in mechanism for determining the length of a table (Lua type that doubles as a array and hash/dictionary) seems to only work for arrays (not hashes) without nil values. I’m sure every Lua programmer learns this sooner rather than later but I also found no mention of this in the handful of tutorials & blogs posts I explored during my Curious George phase or the section on tables in the definitive Programming in Lua. I am left with the hope that this time around my bull in the china shop phase will be shorter.
Startup founder agreements has been the top post on my blog for a long time. The post has become much more valuable due to the many interesting questions & situations described in the comments. Every now and then, a comment merits its own post:
I have a question regarding IP rights.
I came up with a concept for a new start-up and the main feature. That particular feature I came up with was during a chat to one of my start-up partners.
The question is whose idea is it and who owns an IP for it? I feel I came up with it, but it did not happen by itself, it was triggered by conversation.
I would like to be fair.
This situation comes up frequently in one of the following two forms:
A friend helps you come up an idea and start a company. She joins in with you. How much was her help worth? Does she own part of the idea or not?
Continuing from the previous example, after unsuccessfully trying to raise money, you decide to shut down the company. Who would own the idea after the company is shut down?
I call it The Muse Problem. In Greek mythology, the Muses were the goddesses of inspiration in arts & science, catalysts of creativity and invention. If a muse helped a songwriter create a great song, how much of the song’s royalties should the muse get?
Is this and the question of how much of an idea a startup founder’s muse owns there are at least four separate ways of looking at the situation:
Letter of the law. There is a legal standard as to what constitutes full IP ownership. I will not pretend to fully understand it but the gist is that it requires non-trivial input for someone to co-own IP with you. Just being in the same room is not good enough. What about if the idea came from a conversation? What about many conversations? If the conversation(s) were specifically about IP/idea generation, I can tell you that in every single case I’ve approached a lawyer for advice the answer was, “it’s depends.” In every other case they’ve said, “no worries.”
Practice of the law.In theory, theory and practice are the same. In practice, they are not. Assume for a moment that you thought you owned 100% of the idea and that, given perfect information, the letter of the law would be that your friend co-owns the idea with you. She finds out about it and she gets really upset and she hires a badass lawyer and they come after you or, if the IP is now owned by your company, your company. What’s the worst that can happen assuming they can prove their case? If the idea has generated significant value, there could be some liability. How much would depend on how the case goes. If you’ve filed a patent on the idea without disclosing your (now former) friend as a co-inventor then the patent could be invalidated. In practice, however, there are so many ifs and buts along the way starting with the likelihood that your friend probably doesn’t think she owns a part of the idea, going to the fact that most ideas don’t generate enough value to justify lawsuits and ending with the observation that proving a case of wrongdoing may not be easy.
Industry practice. In industry practice, for better or worse, most high-tech founders do not share any portion of their IP with the muse or muses who do not join their startup. It’s just part of how high-tech operates. I don’t have great data about other fields.
Your practice. In your life, you have to decide what’s fair and how you will feel about your actions down the road. There is no right answer and, most likely, nobody will call you out if you do nothing.
The second case, when a company shuts down, is essentially equivalent. If company intellectual property is not a collateral someone can lay a claim to, it reverts to it owners prior to incorporation. If the intellectual property was created inside the company, e.g., when three employees file a patent, the IP goes to the inventors and is shared equally. The equal sharing I honestly find a little random but it’s very rare that the IP would be worth anything. If it were, someone would try to buy the company for the IP.
Back in 2008, I worked with a friend for many months to incubate a company and then, after the crash and after failing to acquire another company’s technology for what we thought was a fair price, we decided that it didn’t make sense to raise money. I had created a significant amount of intellectual property during that period. My friend was not technical and took no part in the actual IP creation but he was my muse and our conversations helped me think through things. I reached out to him and offered him a significant share of future IP proceeds. I did it for two reasons. First, it was the right & fair thing to do. Second, our agreement eliminated legal risk for any third parties interested in the IP down the road and, consequently, increased the value of the IP for me and my friend.
Here is the template for the agreement letter that we used:
Dear FRIEND:
As you know, I am now planning to go forward with trying to pursue protection for the PERPETUAL MOTION concept that we’ve discussed. I wanted to recognize your contribution and to confirm that it is my intention to have you share X% of proceeds from the sale or license of the patent I file on this concept.
I plan to seek patent protection on the high level concept that I presented to you originally. At this point I do not believe I need to have you execute any patent application. However, I would ask that you agree to cooperate in this regard (at my expense) in the event it becomes necessary.
I believe it is also the case that your prior contribution to the project was done on your own time and expense, and further that all information and ideas you have provided me were done outside of any employment or other obligations you might have or owe to any third party.
Please indicate your confirmation of these terms by signing this letter and returning it to me at your earliest convenience.
Caveat: I am not a lawyer and if you are in a situation like this, you may want to seek professional advice. This is not just CYA language, it’s what I’ve done more than once in the past.
The Muse Problem comes up in one other situation: when co-founders are wondering how to split up a cap table. In this case, however, the real question is not how much of the idea anyone owns but how much of the total capitalization of the company is the idea worth. Ideas without smart & hard-working people who can execute them and adapt them to new circumstances are not worth that much.