Today is expected to be the biggest online shopping day ever.

Once a minor side show to brick and mortar Black Friday shopping, Cyber Monday has grown into a multi-billion dollar cyber bonanza for retailers. Without question, Cyber Monday is the most important online shopping day of the year.

With billions of dollars at stake, companies have raced to build out and promote their online shopping presence.

This blog post explores the rapidly changing online shopping landscape. In particular, we look across millions of Internet users and end devices (e.g. phones, mobile apps, web sites, etc.) to analyze which Internet sites and behind the scenes infrastructure lure the most traffic and visitors.

Over the last several months, we conducted a large-scale study of online shopping infrastructure. As in other research reports, we use data from an ongoing large scale study of Internet backbone traffic across a large cross section of North America and multiple collaborating infrastructure and Internet providers (although based on a different dataset, more information about our basic methodology is available here). We believe this is the largest ongoing study of its kind.

The below chart shows the results of our study. The graphic lists some of the top Internet retailers and shopping sites along with the average percentage of Internet users that interact with each site either directly or through third-party sites or back-end infrastructure.

Amazon tops the list with an amazing 14% of all Internet users interacting with Amazon managed shopping sites every day. This includes www.amazon.com as well as a growing empire of Amazon owned branded sites such as www.myhabit.com. For the purposes of this report, we include all amazon owned retail sites under amazon.com with the exceptions of zappos.com and quidsi.com which are large enough to warrant their own entries. The Amazon entry does not include AWS or other Amazon advertising and cloud infrastructure.

What is truly impressive is how much larger Amazon shopping is compared to any other other online site. Amazon is almost double the next largest shopping competitor, Ebay, which enjoys 8.8% of daily Internet users.

While most of the Cyber Monday media attention (including this Wall Street Journal article) focuses on the big names in retail, Cyber Monday also represents a major portion of the yearly sales for tens of thousands of smaller web sites. To demonstrate the growing power this market segment, we we include e-commerce hosting sites like www.shopify.com in the above chart. With an impressive 5.4% market share of daily users, Shopify provides a complete web storefront platform for more than 30,000 sites. As in the brick and mortar world, online small businesses account for a major portion of the US e-retail economy.

While not a household name (or at least no one in my family recognized the name), sites like quidsi.com represent new players that have quickly amassed web conglomerates encompassing dozens or hundreds of smaller sites. The Quidsi empire (ranked #6 at 3.1% of daily Internet users) includes dozens of popular shopping sites such as soap.com (where you can literally buy soap as well as hundreds of other daily sundries), diapers.com (baby products), and yoyo.com (toys). Note that Quidsi was acquired by Amazon for a whopping $540 million in 2010.

Our list also includes web shopping comparison sites like shopzilla.com at nearly 3% of all Internet daily users. Originally a failed Wharton student business plan competition entry, Shopzilla (and similar sites like pricegrabber.com) have grown to direct millions of daily online shoppers to the best deals in other online shopping sites.

For larger retailers, Cyber Monday represents nothing short of an all out price war. Struggling electronics retailers like bestbuy.com have slashed prices and spent millions on promoting their holiday deals.

The below graphs shows some of the tangible results of this online war. In the days following Thanksgiving, BestBuy traffic grew from an average of 1.5% of Internet users to more than 5%.

With that, I’ll wrap up my brief tour of Cyber Monday shopping. I’m off to try and find deals on a Nintendo D3 for my son. Anyone have suggestions?

Is Kayak really nearly twice as valuable as say, an Instagram?

The below graph shows how Kayak stacks up in popularity against other large travel sites. As in other blog posts, we use data from an ongoing large scale study of Internet backbone traffic across a large cross section of North America and multiple collaborating infrastructure and Internet providers (although based on a different dataset, more information about our basic methodology is available here).

The second column shows percentage of unique Internet users visiting a site or related third-party content at least once a day on average. The third, “percent of traffic”, column provides the daily percent of global Internet traffic to the sites and the last column shows 10 day patterns in the percentage of unique Internet visitors.

Based on data across several million Internet users, Expedia dominates the travel industry with 1.5% of all Internet users visiting at least once a day. At just over 1%, Kayak is second followed by HotWire at a distant third.

Oh, and in case you were wondering, Instagram averages a massive 9.5% of Internet users a day and a thousand-fold the volume of traffic.

The below graph shows normalized iTunes traffic across a random sample of several North American Internet providers over the last week. On Wednesday, iTunes backbone traffic spiked to consume an amazing average 7-12 percent of backbone traffic. This iTunes surge is roughly equivalent to abruptly switching on a new Internet service on the scale of YouTube or Netflix. The numbers are even larger if we just look at traffic consumer Internet providers.

Most of the update traffic came from edge CDN infrastructure or direct peering with CDN distribution infrastructure (mainly Akamai). iTunes traffic volumes remained elevated through late Thursday night (EST).

In many Internet backbones, the IOS6 release traffic spike handily outpaced the surges seen during previous IOS updates. Mostly, the millions of downloads appear to have gone without incident. But in a few networks, the IOS traffic flood overwhelmed backbone circuits leading to brief outages and periods of degraded Internet performance.

Within the industry, Netflix’s announcement Monday was old news. Netflix has been privately and publicly discussing their caching plans for the better part of a year (e.g. see this April Netflix presentation at RIPE). By the start of this week, almost every North American provider of even modest size had entertained multiple discussions with Netflix about deploying their caches. From a DeepField commercial and research perspective, we’ve enjoyed a front-row seat watching significant Netflix traffic volumes migrate from CDN to Netflix dedicated infrastructure over the last six months.

Apparently, though, Netflix’s caching plan was a surprise to the market which reacted to Netflix’s announcement by abruptly punishing CDN stocks.

In addition to a somewhat peculiar disregard of previously publicly available information, the market also seems to have significant misconceptions about Netflix’s cache impact on the broader CDN ecosystem.

As other analysts have written (e.g. again see Dan Rayburn’s post), Netflix’s caching move likely will not have a significant, long-term impact on the CDN market. Mainly, over the last several years low cost entrants have driven down pricing and commoditized video delivery — the margins for CDNs today on bulk video are terrible. As the chief driver of peak hour bandwidth (along with negotiated bottom of the barrel pricing), Netflix exacerbated the margin issue. Most CDNs would rather focus on higher value (and more profitable) services like analytics, acceleration, and DRM (e.g. see recent Limelight product announcements).

Overall, Netflix follows a trend amongst the select few “hyper giants” to gain market efficiencies through controlling or building their own low-level infrastructure. But, only a handful of companies have the market need and resources to build their own servers and datacenters (e.g. Google and Facebook) or deploy their own caching infrastructure (Netflix). For all other companies, it makes far more sense to use cost effective third-party services like the established CDNs.

Though, I’ll also observe that Netflix is not alone. At last count, on the order of 30 different content companies and CDNs were aggressively promoting their own hardware for deployment in provider networks. In many ways, the edge caching market resembles the early period of Internet peering. The heady days of open interconnection policies (circa 1996) were quickly replaced by ISPs employing objective (well, somewhat) metrics on settlement-free peering. In later years, only carriers or content providers of sufficient scale and economic benefit met these “free” criteria. We are now seeing the same sort of criteria emerge around provider decisions to deploy third-party caching hardware within their networks.

With all of the above said, it is worth looking at the CDN market in North America today. The below graph shows a break-down of CDN by percentage of aggregate subscriber traffic volume during the month of April 2012. As noted earlier, traffic volume provides limited insight into profitability of a CDN or the distribution of customers.

The graph uses data from an ongoing research collaboration with multiple large North American Internet providers. We analyze anonymized backbone data encompassing a geographically diverse set of several million subscribers (see earlier blog posts for more details). We believe this collaboration represents the largest ongoing study of cloud and Internet evolution.

Overall, the “big three” (Akamai, Level3 and Limelight) dominate Internet CDN traffic volumes. All other CDNs combined represent less than 10% of the traffic volume and typically focus on market niches such as gaming or low-cost bulk file updates. The exact percentages can vary by several points depending on timeframe and whether you look at things like percentage at peak time or overall average.

Netflix started the year more or less evenly distributing the traffic across the three big CDNs with a trickle of DRM / management traffic also going to Amazon’s EC2 cloud. In the below graphic, we visualize average Netflix traffic across several North American providers. The width of the flow corresponds to the relative percentage of traffic and the color either video traffic (green) or DRM / control (red). Again, I note the exact percentages vary across different networks and time scales.

By the time of Netflix’s cache announcement this week, you can see the significant change in average video traffic distribution across these networks. Now 70% of Netflix traffic on average comes directly from Netflix distributed caches or other infrastructure with Netflix datacenters (AS2906).

I emphasize that Netflix traffic patterns vary significantly across providers. In some networks, the Netflix cache transition appears nearly complete while in others not yet started. The below visualization shows several networks where the migration has already occurred (or is ongoing) and does not represent a synopsis view of all Netflix North American traffic.

So what does this all mean?

As I’ve observed in earlier work, the Internet is in the midst of a fundamental shift from connectivity to content. In the past, carriers saw their role as delivering arbitrary bits between their customers and many millions of web sites. Today, most customers care about a shrinking number of video, cloud and content sources. Our most recent data finds that more than 70% of all Internet traffic (on average) comes from just 150 CDN, hosting, cloud and content companies.

At the same time the number of content sources is shrinking, the volume of “hyper-giant” traffic is growing astronomically (especially HD video). The Internet simply can not cost effectively meet these burgeoning traffic demands without additional growth in CDN infrastructure and embedding additional server capacity and content directly into the last-mile network.

This hyper-giant content evolution has changed the way Internet and content providers build their networks and monetize their infrastructure. This is a good thing for the market and consumers (I will save a more detailed discussion on the nuances of these benefits for a later blog post). While we will continue to see disintermediation in the market as “hyper giant” companies like Netflix pursue direct relationships with subscriber networks, I also expect the CDNs to play a significant and growing role in the Internet / cloud evolution.

At a staggering $100 billion dollar valuation and reported 900 million users, Facebook is clearly a massive player in the market and Internet economy. From an Internet infrastructure perspective, Facebook also ranks amongst the largest of the “hyper giants” generating a significant share global Internet traffic.

With a PE ratio of 95:1, Facebook is also an incredibly expensive stock.

This blog post attempts to put some concrete numbers behind Facebook’s enormous Internet presence and evaluate the company’s valuation in terms of its Internet traffic contribution.

As in previous posts, we use data from an ongoing research collaboration with multiple large North American Internet providers. We analyze anonymized backbone data encompassing a geographically diverse set of several million subscribers. More details on the research methodologies used in our prior work is available here. We believe this is the largest ongoing study of its kind.

On average, our analysis finds Facebook contributes nearly one percent of all Internet traffic (the actual number is 0.75%). This includes traffic both to Facebook’s private datacenters as well as third-party edge CDN caches (over 85% of Facebook traffic relies on CDNs).

While one percent is an awesomely huge number, the really, really impressive statistic is 45%. More specifically, we estimate 45% of all Internet subscribers send traffic to Facebook servers at least once every day. This includes traffic sent directly to www.facebook.com as well as the indirect connections made by tens of thousands of third-party web sites that include Facebook content or APIs (most users are likely  unaware of all the traffic their browsers send to Facebook when they visit these third-party sites).

Given our estimate of Facebook’s traffic volume and today’s IPO price, we now calculate the first ever estimate of the value of Facebook’s Internet traffic.

We first use data from Cisco to estimate the overall size of Internet traffic (37 Exabytes per month). At 0.75% and a $104.2B valuation, this means that Facebook uses 824,000 Mbps of bandwidth continuously. When you put their valuation in terms of this bandwidth you get a staggering $124,000 per Mbps. (Amazing, considering that over the course of each day there’s over 9,000,000,000 Megabytes delivered.)

Overall, an amazing company with an equally astounding market valuation and now very, very valuable traffic.

In this blog, we ask the question how big is Amazon’s cloud?

Amazon is clearly big.

For example, Amazon’s press releases like to tout highly technical statistics like the rate of S3 requests (650,000 per second!) and number of objects (900 billion!). Somewhat more insightfully, outside analysts’s have estimated the number of servers and revenue ($200 million in 2010). But none of this really gives a picture of Amazon’s growing role underpinning the global Internet economy.

So with collaboration of several network provider research parters, we conducted one of the largest studies of its kind analyzing multiple weeks worth of network data to AWS from a broad cross section of a several million Internet end-users (mainly in North America). Our goal was to characterize AWS traffic, understand the major companies using AWS infrastructure, and ultimately gauge the importance of AWS to the Internet infrastructure and daily services / browsing of end-users.

The below chart summarizes some of our key findings.

One way to gauge the importance of Amazon is to ask how frequently will a typically Internet user visit a web site based on Amazon infrastructure? The answer: an amazing 1/3 of all users every day. This number is all the more impressive when you consider that our data includes millions of users and end devices of limited scope or activities, such as users who only check mail and home game consoles.

[Note: Since our study focused on subscriber traffic, we excluded servers (such as consumers hosting web sites) and Internet "background noise" including the nearly constant barrage of scanning / intrusion attempts from China, botnets, machine-to-machine communication for software updates, etc. Though a different dataset, our earlier academic papers provide more background on related methodology].

Traffic volume provides another metric, albeit indirect, of Amazon’s growing Internet presence. As of April 2012, Amazon contributes more than one percent of all consumer Internet traffic in North America. This is a huge number given that Amazon, unlike, say Google, does not typically host massive video content. Instead, this one percent represents the broad reach of Amazon infrastructure across hundreds of client companies. By comparison, we found all of Google’s sprawling YouTube infrastructure contributed six percent of Internet traffic in 2010.

Finally, we looked at Amazon’s growing content distribution network (CDN). Over the last several years, CDNs have evolved as the workhorse of the Internet, delivering the majority of images, video and other content to end users. Since its launch in 2008, Amazon’s CloudFront CDN and S3 distributed storage services have steadily gained in popularity. As of today, Amazon ranks as the fourth largest CDN by traffic volume (trailing behind Akamai, Limelight and Level3).

Now on to our final question: what companies are using Amazon cloud infrastructure?

In the below table, we show the 40 largest corporate users of Amazon’s cloud infrastructure (contact us for a complete list).

As an estimate of the importance of AWS to each company, we calculated the average percentage of all subscriber AWS connections that access one or more of each site’s AWS components each day. So, for example, in the top spot, 21% of subscriber connections to AWS go to truste.com. Like many of the top AWS corporate users, truste.com is an advertising / analytics company (as is InviteMedia, Chartbeat, Evidon, etc.). Although most consumers remain blissfully unaware, almost every web page they visit is tracked, analyzed and scored by dozens of analytics and marketing companies (a large number of them using Amazon infrastructure).

Many of the companies above are familiar consumer names like DropBox, Netflix, Instagram and Pintrest. Others, like Heroku, provide behind the scenes platform as a service (PaaS) to hundreds of other companies running cloud applications. And still many other companies (including my own), use Amazon infrastructure for their internal enterprise applications and back-office support.

Overall, Amazon enjoys a commanding lead in the much balleyhooed, mind-blowingly large $200 billion anticipated cloud computing market. But the war for cloud dominance is just beginning. Companies like Rackspace, CSC, Microsoft and Google are investing billions in datacenters and software to compete. In upcoming blogs, we’ll explore the infrastructure and Internet footprint of some of these other large cloud players.