I’ve been asked to explain how online ads are delivered many times and every time I’m surprised by the complexity of covering even the most basic elements of how ads appear on Web pages. Since Wikipedia’s article on ad serving is not much help, I’ll try to explain one common way ads are delivered using a concrete example.

Side note: this is not how Swoop works. At Swoop we use a much simpler and more efficient model because we’ve built an end-to-end system. This eliminates the need for lots of different systems to touch (+ cookie + track) users. It also allows us to create deeper and more relevant matches by placing Swoop content not in arbitrary fixed slots but in dynamic slots right next to the part of a page it relates to. If you are looking for an analogy, think about Google Adwords on SERPs. It’s an end-to-end system where Google has complete control over ad placement and no ads are shown if there are no relevant ads to show.

Tools of the trade

If you want to know how adtech works, there is no better tool than Ghostery. Ghostery was created by my friend David Cancel and, later, when he was starting Performable (now part of Hubspot), my previous startup, Evidon, became the custodian of the Ghostery community. Ghostery will show you, page-by-page, all the different adtech players operating on a site. For example, on Boston.com’s sports page, there are 35 (!) separate adtech scripts running today.

Ghostery will show you what is happening but not how it happened. If you are technical and want to understand the details of how ad delivery works, there no better tool than a debugging proxy such as Charles or Fiddler. Just be prepared for the need to use code deobfuscators. If you don’t have time for wading through obfuscated code and you really want to know what’s going with your sites(s) or campaign(s), it is worth taking a look at Evidon Encompass. It’s an advanced analytics suite built on top of the Ghostery data.

The example

The example we’ll use is the arthritis page on Yahoo!’s health network. We will focus on the leaderboard ad at the top, which is a Celebrex (arthritis drug) ad from Pfizer.

What Yahoo! sent the browser

The initial response my browser got from Yahoo!’s server included the following chunk of HTML about the leaderboard ad unit, which I’ve formatted and added comments to. (Not sure what’s up with the empty lines that WP is adding to the bottom of the gists–they’re not on GitHub).

This content was mostly likely emitted by the Yahoo publishing system without direct coordination with the Yahoo ad server but instead using conventions about categories, page types, etc. and hence parameters like rs=cnp:healthline that you see on the URLs.

Display advertising units use standard IAB formats. In this case, we are dealing with a 728×90 leaderboard unit. The DIV with id yahoohealth-n-leaderboard-ad sets up the location where the ad unit will be displayed. The DIV under it serves the dubious function of controlling some styling related to the ad content.

Beyond this there are two things going on here. The first is the delivery of the ad script and the second is the delivery of a tracking pixel via a tracking pixel script.

Tracking pixels

Tracking pixels are 1×1 invisible images served from highly-parameterized URLs. They are not used for their content but for the request they generate to a server. The request parameters are used for record-keeping and the response could be used to cookie the user, though this did not happen in this case.

The tracking pixel is delivered via the script inside the <center> tag. It’s contents are shown below.

The script uses the JavaScript document.write function to write some HTML into the page. In this case the HTML is for an invisible image (display: none, height: 0, width: 0) whose URL is that of the tracking pixel, whose unencoded value is the long URL:

http://us.bc.yahoo.com/b?P=s4i.Bjc2LjFQGJ6UTYvtUx1LMTczLlFt_1n__8MU&T=18f3d55it/X=1366163289/E=96843138/R=he/K=5/V=8.1/W=0/Y=YAHOO/F=4264990850/H=YWRjdmVyPSI2LjQuNCIgc2VydmVJZD0iczRpLkJqYzJMakZRR0o2VVRZdnRVeDFMTVRjekxsRnRfMW5fXzhNVSIgc2l0ZUlkPSI0NDUzMDUxIiB0U3RtcD0iMTM2NjE2MzI4OTI1ODA1NyIg/Q=-1/S=1/J=69060D4C&U=12d690f6l/N=lv51DmKImng-/C=-1/D=FSRVY/B=-1/V=0

As you can see, lots of data getting sent, most likely to record the impression opportunity parameters.

Yahoo! ad delivery script

There are two ways to deliver an ad unit. The preferred way is via a script. If scripting is disabled in the browser, however, Yahoo doesn’t want to lose the ad impression opportunity and so there is the <noscript> option to show the ad in an iframe, probably as an image.

The code for the Yahoo! ad delivery script, which comes from the Yahoo! ad server, is shown below with reformatting and comments from me.

There are several things going on here:

• AdChoice notice
• Cache busting
• Google ad delivery script activation
• Yahoo impression tracking
• No script handling

Let’s consider them one at a time.

AdChoice notice

AdChoice came about in 2010 as the online advertising industry’s response to FTC pressure to reign in some poor privacy practices and provide consumers with more transparency and choice when it comes to interest-based advertising, a.k.a., behavioral targeting (BT in adtech parlance).

The AdChoice icon is a triangle with an i in it. Its color can vary. Yahoo!’s is gray (). Next time you see an ad with it, click on the AdChoice notice. You should see information about who targeted the ad at you and get some options to opt-out of interest-based advertising. We started Evidon back in 2009 to bring more transparently to adtech and we helped create AdChoice. Evidon is now the leading independent player in this space.

In the case of the Celebrex ad from our example, the AdChoice icon is tied to a very long URL:

http://clicks.beap.bc.yahoo.com/yc/YnY9MS4wLjAmYnM9KDE1aGtrdjRldShnaWQkVHdWQTBqYzJMakZRR0o2VVRZdnRVd1VTTVRjekxsRnRfMGJfXzcwayxzdCQxMzY2MTYzMjcwNTgzNjQ0LHNpJDQ0NTMwNTEsc3AkOTY4NDMxMzgsY3IkMzM1MTkyOTA1MSx2JDIuMCxhaWQkMENPUU9rd05QZlEtLGN0JDI1LHlieCR2ZFJfX19Ed2xLd2liQ0k4SEdHUVVBLGJpJDE3Mzk2MTcwNTEsdyQwKSk/1/*http://info.yahoo.com/relevantads/

If you click on the AdChoice icon, Yahoo! will record information about which ad you are selecting to learn more about and then redirect you to the page at the end of the URL, which is the Yahoo learn more about this ad page. The long URL is just for bookkeeping.

BTW, the reason why you don’t see AdChoice notice with Swoop is because Swoop does not do any behavioral targeting at this time. Still, because we want to make it clear that Swoop is serving content, you’ll see our logo on our units.

Cache busting

After the AdChoice notice setup comes a line of script that creates a random number. This is used for cache busting.

A cache-buster is a unique piece of code that prevents a browser from reusing an ad it has already seen and cached, or saved, to a temporary memory file.

Adding a random number to a URL does that nicely.

Google ad delivery script activation

The following script tag loads Google’s ad delivery script from ad.doubleclick.net. We will look at this later on.

Yahoo impression tracking

Remember how Yahoo already fired one tracking pixel to record the impression opportunity. Well, here, at the end of the script they are going to fire another tracking pixel but this time the purpose will be to record the impression of the Google ad. As before, you can see lots of data being passed.

http://csc.beap.bc.yahoo.com/yi?bv=1.0.0&bs=(134i64f2m(gid$TwVA0jc2LjFQGJ6UTYvtUwUSMTczLlFt_0b__70k,st$1366163270583644,si$4453051,sp$96843138,pv$0,v$2.0))&t=D_3&al=(as$12rveflk1,aid$0COQOkwNPfQ-,bi$1739617051,cr$3351929051,ct$25,at$0,eob$-1)

Noscript processing & click tracking

As before, in the case that the browser does not have JavaScript enabled, Yahoo doesn’t want to miss the opportunity to deliver an ad, which is why they have the option to display the Google ad as an image.

In that case, Yahoo is also positioned to capture the click and then redirect to Google. This is achieved by wrapping the image (<img>) in a link (<a>). Getting click feedback data would be valuable for Yahoo as it allow is to optimize better. If the unit is sold on a cost-per-click (CPC) basis, then getting click data is a requirement for good record-keeping.

Google/DoubleClick ad delivery script

It’s time for us to take a look at what Google’s ad delivery script does. Alas, the guys at Google don’t want to waste bandwidth so they’ve packed everything into a single unreadable document.write call. You can scroll to the right for a very long time…

Here is what Google is actually trying to write into the HTML page (with my comments added):

Don’t worry about the volume of code. There are basically two things going on here: delivering a Flash ad and lots of third party ad verification.

Delivering a Flash ad

Flash ads are richer and more interactive but there are browsers where Flash ads don’t do so well. The first part of the Google/DoubleClick ad delivery script is about carefully determining whether a Flash ad unit can be used and falling back to images otherwise. As before, all clicks are tracked via redirects.

Third party  verification

We saw Yahoo! attempting to fire three types of tracking pixels: (a) for impression opportunities, (b) for actual impressions and, in the case of no scripting, (c) for clicks. This is to help optimize the performance of Yahoo!’s ad network. This is first party verification. Google/DoubleClick does the same with its own systems.

Third party verification happens when the advertiser asks the delivery network, in this case Google/DoubleClick, to include additional verification tags (scripts) to prove that the campaign is delivered based on its configured parameters.

In the case of this Celebrex campaign, Pfizer is using four separate verification vendors. At the top level we have only Nielsen NetRatings and DoubleVerify, however NetRatings’s script loads AdSafe as well as Facebook in the pattern we are familiar with: a script that writes out <script> tags to load more scripts.

Putting it all together

Let’s try to piece together the requests that allow this one single ad unit for Celebrex to appear:

• Yahoo ad unit delivery script
  • Google ad delivery script
    • Flash movie
      • ??? (not easy to track Flash traffic)
    • Nielsen NetRatings tracking script
      • AdSafe pixel
      • Facebook iframe
        • Facebook tracking
          • ??? (did not analyze)
    • DoubleVerify script
      • Tracking script (like a pixel)
  • Yahoo impression tracker
    • Tracking pixel
• Yahoo impression opportunity tracker
  • Tracking pixel

All in all, 13 separate HTTP requests to 6 separate companies, not counting redirects and cacheable scripts. With this much complexity and independent parties doing their own accounting, it’s no surprise the display advertising value chain is in a bit of a mess right now.