This Wednesday for about 30 minutes  many Australians found themselves without Internet access. All these users were relying either directly of indirectly on the Telstra network, which at that point was isolated from the Internet. This story quickly hit the local headlines, in this blog we’ll look at the technical details of this event and what the cause of this outage likely was.

Telstra is one of Australia’s major Internet providers. It normally originates approximately 500 Ipv4 prefixes and 3 Ipv6 prefixes. Telstra also provides Transit for many ISP’s and enterprises such as for example AS38285 ‘Dodo’ an Australian ISP and AS10235 ‘National Australia Bank’. So how could such a large provider go down, surely it has lots of redundant hardware and multiple connections in and out of the country?

As it turns out Wednesday’s outage was caused by a routing error many network engineers have first hand experience with, a simple routing leak. A routing leak can happen when small ISP X buys transit from ISP A and also from ISP B. ISP X receives a full bgp routing table from A and because of incorrect filtering  relays these messages to ISP B. As a result ISP B now learns all Internet routes via ISP X to ISP B and ISP X (the customers) now became an upstream provider for ISP B.

The above is likely what happened last Wednesday between Telstra en Dodo (AS38285). Dodo a Telstra customer, re-announced all Internet routes to Telstra, which because it prefers customer routes now thinks the best way to the Internet is through Dodo. This post on the Ausnog mailings list shows how Telstra was using Dodo (a customer) as transit to reach a network in India.

This is not a new zero day attack scenario or anything like it. Instead it’s probably the number one mistake when configuring BGP routing. I remember when I was just learning about BGP my mentor always used to tell me.. Filter, Filter, Filter, filter!! Which is exactly what didn’t happen here.
Because it is so easy to accidentally leak routes in BGP you have to explicitly define filters that prevent this. In this case Dodo should have had filters to make sure they would only announce their prefixes and Telstra should have had these filters as well to prevent hijacks but more importantly to protect its own infrastructure. In this case these filters did not seem to be in place, which allowed this leak to happen.

However, this alone should not have brought down all of Telstra’s International connections. So what happened? It’s likely that Telstra now tagged all routes learned from Dodo (all 400,000 of them) as customer routes and faithfully announced this to all of its peers and upstream providers.

As keeping large filters up to date can be tedious we often see large providers use a mechanism known as max prefix limits. Instead of explicitly defining which prefixes to allow the number of prefixes expected plus some extra is set as the maximum number of prefixes allowed. This is useful to prevent a sudden spike in announcements, often caused by leaks. In case the limit is reached the BGP session is brought down to prevent the leak from spreading.

This is what likely happened with Telstra’s upstream providers. Telstra announced a full Internet routing table to its upstream providers, triggering the max prefix limit causing the BGP sessions to go down and as a result isolating Telstra and its customers from the rest of the Internet.

This outage is clearly visible when we look at the Telstra routes in the Internet BGP routing tables. The graph on the left clearly shows how all of Telstra’s prefixes suddenly disappeared at 2:40 UTC and came back  approximately 30 minutes later when the leak problem was resolved.

Our analysis shows that this affected approximately  1400 prefixes. This includes Telstra and Telstra customer networks and accounts for about 10% of all the Australian IPv4 prefixes. Worth pointing out is that non of the IPv6 prefixes were affected. This is because it’s common to use different BGP sessions for IPv4 and IPv6. In this case only the IPv4 session went down leaving the IPv6 networks unaffected.

And so it could happen that one simple customer leak can bring down a complete carrier network. So today’s lesson is Filter Filter Filter!

Often these anycast nodes are deployed to only serve specific network and/or regional areas, making them only reachable from the networks where they are deployed. In the case of the F-root server, operated by ISC, there are quite a few local servers. See this page for a detailed list: https://www.isc.org/community/f-root/sites One of these servers is the F-root located in China.

This weekend for about 25 hours this particular root server was reachable not only from China, but from several places in the world when using IPv6. Normally this shouldn’t be a real problem, as the root servers return the same DNS response regardless of their geographical location. However in the case of China, all traffic typically passes through the great firewall where DNS responses for requests such as facebook.com are often rewritten to different false IP addresses. Please note that I am not aware of any data to confirm if this is what happened in this particular case!

Technical details
This weekend between 01 Oct 2011 17:56:12 and 02 Oct 2011 19:01:09 GMT a number of providers thought that the best IPv6 path to the F-root server (AS3557), was to the F-root in China.

The prefix for the F-root server is 2001:500:2f::/48 and is originated by the following AS numbers:
3557 ISC-F-ROOT Internet Systems Consortium, Inc
1280 ISC-AS1280 Internet Systems Consortium, Inc.
30132 ISC-AMS1 Internet Systems Consortium, Inc

By looking at the next hop AS number we can get a good feeling of where the server for this particular announcement resides. When analyzing the announcement for 2001:500:2f::/48 we saw one new AS we had not seen before outside of China, AS55439.

When looking at all announcements for the F-root in China, we saw the following constant in the ASpath:
6939 23911 18344 37944 24151 55439 3557

In this case AS55439 is the AS for ISC-PEK2 Internet Systems Consortium, (Beijing, China). Next the announcement was passed along to AS24151 (China Internet Network Information Center), AS37944 (CHINA SCIENCE AND TECHNOLOGY NETWORK), AS18344 (CNCGROUP-BACKBONE-NORTH), it then ended up at the AS23911, the China Next Generation Internet Beijing IX. That’s where it was learned by AS6939 Hurricane Electric, the world’s largest IPv6 provider. Hurricane electric then advertised it to its customers.

The dataset I used showed that numerous Hurricane electric customers in countries worldwide, including for example Germany, South Africa, Brazil and the US selected this path to China. Some of the large providers include AT&T, Telia and Interoute.

What does all this mean
It’s likely that DNS queries over IPv6 to the F-root server from clients worldwide were answered by the F-root in China. Although this is in itself is not necessarily a problem, it does expose this traffic to the Great Chinese Firewall with the risk of altering the responses.

This is not the first time this happens, last year Renesys reported about a similar event involving the I-root server and China.

The only way to make sure that DNS responses are not rewritten is to use DNSSEC, this will allow the resolver to verify the authenticity of the response.
With regards to BGP, in this case all that seems to have happened was a BGP leak. Things like this are typically the result of an operator mistake. Best practice is to monitor for these kinds of things, so you’ll be alerted as soon as it happens and as a result allowing operators to limit the effect of such an event. BGPmon.net provides users with several flexible ways to monitor BGP announcement and to check if they match your BGP policies.

of all Syrian networks.

The Internet in Syria is dominated by “The Syrian Telecommunications Establishment”, which routes its networks from AS29256 and AS29386. Besides these providers there are AS6453 – Tata communications which routes 6 Syrian prefixes and AS39154 The Syrian Higher Education Network.

As of this morning only 19 of the normally 56 Syrian prefixes are routed.  Interestingly the prefixes that are no longer routed are normally  originated by AS29256 and AS29386, “The Syrian Telecommunications Establishment”.  The 6 Tata communication prefixes as well as the Syrian Higher Education Network have not been affected.

The table below show how many prefixes are normally routed by these providers, and how it has changed over the last hours.

The pie charts below show how the Syrian network are distributed over the different providers. Clearly visible are the reduced number of prefixes announced by “The Syrian Telecommunications Establishment”, AS29256 and AS29386.

Prefix Distribution June 1

Prefix Distribution June 3

Update June 4th, 2011
As of this morning (approximately 08:00 UTC) all Syrian networks routed by “The Syrian Telecommunications Establishment” are back online! Some prefixes came back earlier this morning. Also see Google’s Transparency report here.

Barrett Lyon reported on his blog that he noticed that AT&T was routing traffic to Facebook through a Chinese network (China telecom AS4134). In this blog post we will take a closer look at what exactly happened.

The raw data
By analyzing the raw data we can indeed see BGP announcements for several of Facebook prefixes with rather long and odd looking ASpaths. We also see several US based providers, such as AT&T, that selected a path through SK Telecom (Hanaro Telecom South Korea) and China Telecom.

It seems to all have started on Tuesday March 22, at 07:15:02 GMT. That’s when we see the first announcement for one of Facebook’s networks, routed through Korean provider SK Telecom. The last announcement via SK Telecom was at Tuesday, 22 Mar 2011 16:11:02 GMT, so about 9 hours in total.

The impact of this incident varied per provider. Some networks didn’t use the path through Korea, some only for a few of Facebook’s prefixes as some for (almost) all of Facebook’s prefixes.

Two Japanese providers, who peer directly with SK Telecom (Korea) routed the following Facebook networks trough AS9318, SK Telecom (Hanaro Telecom South Korea)

• 204.15.20.0/22
• 66.220.144.0/21
• 66.220.152.0/21
• 69.171.224.0/20
• 69.171.239.0/24
• 69.171.240.0/20
• 69.171.255.0/24
• 69.63.176.0/21
• 69.63.184.0/21
• 74.119.76.0/22

The providers affected by this were:
Internet Initiative Japan Inc. (IIJ)   ASpath:  2497 9318 32934 32934 32934
KDDI   ASpath: 7660 2516 9318 32934 32934 32934

Several other large providers such as Savis, AT&T, Tinet, KPN, Telia, Qwest, AboveNet and Telecom Italia (note this list is not inclusive, there are several more) were also affected, however in these cases only two prefixes were affected.

69.171.224.0/20
69.171.255.0/24

All of these providers learned this announcement via AS4134, which is China telecom. In all cases the Facebook prefixes were prepended three times. Different peers on different locations saw the announcements, the common part in the ASpath was:

4134 9318 32934 32934 32934

This proves that the announcements were not spoofed by China telecom, as others learned it directly from SK Telecom as well.

What exactly did AT&T see at the time of the incident?

This is a snapshot of the AT&T routing table at March 22^nd,  8AM, showing Facebook networks only. Note that only 2 networks were routed through Korea and China.

As can be seen from the snapshot above, the majority of the prefixes (as seen from AT&T) are routed through Level3 (3356). So what happened to the other 2 prefixes? Why was the path to those networks not through Level3. To answer that question we need to know the path to these networks from Level3′s perspective. When looking at the same datasource we see the following for 69.171.224.0/20:

This is interesting; it shows that Level3 learns the Facebook prefix with a rather long ASpath, Facebook’s AS is prepended 5 times. This explains why AT&T chose the path via China and Korea. Even though it’s long, it was shorter than the heavily prepended path via Level3. As a result providers who peer with China Telecom and had this shorter alternative, would have chosen the path via China.
Global Crossing, another large global Transit provider showed the same behavior as Level3, an ASpath with 5 times Facebook’s AS32934.

What could be the reason?

For some reason the path via providers such as Level3 or Global Crossing, Transit networks that are commonly used by the affected providers, were advertised with a very long ASpath. And as a result the providers that have a peering agreement with China Telecom (such as AT&T) started to use the shorter path via China and Korea.
SK Telecom normally is not one of Facebook’s transit providers, but likely just a normal peer.  SK Telecom then announced these Facebook prefixes to its peers (IIJ, KDDI and  China Telecom) which started to use them.

China Telecom did not filter these Facebook announcements learned via SK Telecom and re-announced them happily to its peers. As a result traffic from several major global providers started using the path through China and Korea to reach some of Facebook’s networks.

Why the providers such as Level3 and Global Crossing did not have a shorter path to these two Facebook networks is unknown. But it’s likely that it had to do with the Facebook BGP setup / infrastructure.

Keep in mind….
It’s likely that both China Telecom and SK Telecom have a presence in the US. And as such it’s not necessarily the case that traffic was actually routed through China.  If someone has a traceroute from AT&T or any of the other networks taken at the time of the incident, please post this in the comments, this can helps us determining how the actual traffic flowed.

Facebook has a lot of data and does all kinds of CDN (content delivery) tricks to get data to its users.  Depending on your location you get a different DNS result and send to a different server. This might have been the reason why there were no reports of widespread outage.

One of the blocks that did not came back immediately was 193.227.1.0/24. This block hosts FRCU.EUN.eg, one of the primary name servers for the EG domain. 193.227.1.0/24 is normally announced by AS2561 Egyptian Universities Network (EUN), which before these events announced 17 networks. Currently I only see 6 of these networks, some more specifics seem to be missing. I do see 193.227.0.0/18 which is an aggregate for this netblock. Reports are that the server is reachable again from most places, however I personally can’t reach it from the research network in Canada right now, from my Level3 server in Amsterdam it does work however. So I assume that the Egyptian Universities Network is only partly restored. This could result in slower DNS responses for Egyptian domain names for some users.

Both Ripe and Renesys have some other nice graphs visualizing Egypt coming back online.

Egypt has been offline for 5 days, this is truly unprecedented in these modern days.  It’s been interesting to see how alternative ways of electronic communications have been used and how Ad hoc Internet connections have been made available. I’ve read about amateur radio sending out morse code as well as lot’s of dial-up numbers that were distributed for Egyptians to use.

Different media are reporting that Internet and other forms of electronic communications are being disrupted in Egypt.  Presumably after a government order in response to the protests. Looking at BGP data we can confirm that according to our analysis 88% of the ‘Egyptian Internet’ has fallen of the Internet. In this post I’ll  share some observations I made with regards to the reachability of Egyptian networks and providers.

What’s different in this case as compared to other ‘similar’ cases is that all of the major ISP’s seem to be almost completely offline. Whereas in other cases, social media sites such as facebook and twitter were typically blocked.  In this case the government seems to be taking a shotgun approach by ordering ISP’s to stop routing all networks.

Networks affected

Egyptian Routes

When looking at the data it’s clear that many Egyptian networks have fallen off the Internet. Let’s start by looking at a quick summary. Yesterday there were 2903 Egyptian networks, originated from 52  ISP’s. Transit was provided via 45 unique isp’s.
Today at 2am UTC, the numbers look quite different, there were only 327 Egyptian networks left on the Internet. These were originated 26 by ISP’s.
So 88% of the Egyptian networks is unreachable!

Below you’ll find a table with the top 10 providers in Egypt. It shows how many Egyptian networks were announced earlier this week and how many are reachable today.

As you can see in the table below, right now most autonomous systems (ISP’s) are no longer announcing any, or at the very least, significantly less prefixes.

Interestingly the only provider that doesn’t seem to be impacted by this is AS20928  (Noor Data Networks).

Below is the list of providers that are still announcing networks (based on routeviews data):

When did this start?

By looking at some Egyptian websites and looking at when they became unreachable we are able to determine when the problem started.

At this point egypt.gov.eg is offline. This network, 81.21.104.0/24 was withdrawn at January 27^th at  22:28 UTC . Another example is www.ahram.org.eg an Egyptian news paper. This network 196.219.246.0/24, became unreachable at the exact same time, January 27^th at  22:28 UTC.

Update (Jan 28 18:36 PM UTC)
At this point only 239 Egyptian networks are reachable, this means that 91% of the Egyptian routes are unreachable. Noor networks remains the only provider that seems to be unaffected by this.

Vodafone has confirmed on their website, that they have been instructed to shutdown services in parts of the country
http://www.vodafone.com/content/index/press.html

All mobile operators in Egypt have been instructed to suspend services in selected areas. Under Egyptian legislation the authorities have the right to issue such an order and we are obliged to comply with it. The Egyptian authorities will be clarifying the situation in due course .

Update (January 29, 17:48 UTC)
It’s been reported that Mobile phone services have been restored. I found this confirmation on the Vodafone website

Vodafone restored voice services to our customers in Egypt this morning, as soon as we were able.
We would like to make it clear that the authorities in Egypt have the technical capability to close our network, and if they had done so it would have taken much longer to restore services to our customers.
It has been clear to us that there were no legal or practical options open to Vodafone, or any of the mobile operators in Egypt, but to comply with the demands of the authorities.
Moreover, our other priority is the safety of our employees and any actions we take in Egypt will be judged in light of their continuing wellbeing.

There has been no change in the Internet services situation. As of now Egypt has been cut-off the Internet for 36 hours.  In Egypt the work weeks starts tomorrow, so let’s see if the government will bring the services back online perhaps tonight.

Update January 29, 21:00 UTC

I’ve published a list of the Egyptian routes/prefixes that are still being routed. This list of  243 networks can be found here: http://bgpmon.net/egypt-routes-jan29-2011.txt.
This data is extracted from routeviews data (rib.20110129.1800). Note that routed does not necessarily mean that it’s reachable. Format of the list is:
|AS number| Prefix| Prefix description|.
A few examples of routes that are still announced:

• AT-Financial Holding routes
• Bibliothique of Alexandria  (http://www.bibalex.org/)
• Central Bank of Egypt Routes
• Egyptian National Scientific & Technical Information Network routes

Update January 31, 23:30 UTC

As of today thirteen more Autonomous systems (ISP’s) have disappeared. This includes the up to now unaffected Noor Data networks. The networks for Noor data network seemed to have disappeared at 2011-01-31 20:54 UTC.
As these ISP’s disappeared so did more routes. Using a routeviews dump of 22:00 January 31^st,  we now only see 12 origin Autonomous systems and a 134 routes. This means that now only 5% of the Egyptian routes remain reachable.
A list of all remaining routes can be found here:  http://www.bgpmon.net/egypt-routes-jan31-2011.txt

number of advertised egyptian routes

As this will become an important technology in securing inter-domain routing, I’ve decided to try to get some hands-on experience with this. In this article I’ll describe my understanding of the current state of the RPKI and introduce the BGPmon ROA validation service.

RPKI building blocks
The main building blocks in the RPKI infrastructure are trust-anchors, ROA’s and validators. Trust-anchors used today are the RIR’s (RIPE, APNIC, LACNIC, AFRINIC). These are the organization that hand out the network resources and therefor are our point of trust. Route Origin Authorization or ROA’s are documents used to link a set of prefixes with an origin ASN. These ROA’s are created by ISP’s, or more accurately the maintainers/administrators of an Autonomous system or Network. The result is signed by the resource holders private key and this signing than creates a chain of trust which allows us to authorize (validate) BGP announcements. Below you’ll find an example ROA, note that a single ROA can contain multiple prefixes and has only one origin AS.

Today, in order to verify BGP announcements, we can only rely on the numerous IRR databases and hope that this information is valid. The RPKI system is not intended to replace the current IRR systems, instead it complements this system.
As you probably know many IRR databases allow you to add any route object to the database. For example you could add a route object for 8.8.8.0/24 with your own origin AS. If your upstream uses the IRR to automatically create prefix-filters, you would now be allowed to announce 8.8.8.0/24. With this new RPKI infrastructure providers, or scripts, can do an additional validations step, to check if the AS in question is authorized to announce this network.

BGP router validation
Eventually the goal is to have BGP capable routers validate prefixes against ROA’s. Based on the result one might change the local-preference of this route add a tag, or whatever you think is best. The key thing is that you’ll have the ability to distinguish authorized announcement from non-authorized routes. So if you have a choice between multiple paths, one with a authorized origin AS and non with a non authorized origin AS you can prefer the path with the authorized origin AS.
Routers won’t do the actual certificate validation, this would take to much resources. Instead they’ll talk to a remote validator that will answer the router’s question:
I just received an announcement for 8.8.8.0/24 with origin AS 15169, is this an authorized announcement?
The remote server will answer this with one of the answers below (as specified in the RFC’s / IETF Drafts):
Code 0: Roa found, validation succeeded
Code 1: No Roa found (resource is not yet signed)
Code 2: Roa found, but validation failed. For example because of an incorrect origin AS or prefixlength.

Validating ROA’s, A proof of concept
In order to gain some experience with RPKI and ROA’s, I’ve decided to develop a ROA validator.
The current implementation relies heavily on the validator written by ripe. This validator fetches all ROA’s from the 4 trust anchors and stores the results locally. As a first implementation I’ve extended the current BGPmon whois service with ROA support. For each query the found prefixes will be checked for a ROA. The output of this whois query will now show and additional line with the results of the ROA validation. See example below:

$ whois -h whois.bgpmon.net 200.7.86.0

Prefix:              200.7.86.0/23
Prefix description:  LACNIC
Country code:        UY
Origin AS:           28001
Origin AS Name:      LACNIC - Latin American and Caribbean IP address
RPKI status:         ROA validation successful

If you would like to check just the ROA or receive more information about a ROA, you can use the -–roa feature. This will return the validation code as specified in the RFC’s / IETF Drafts.
In the case of a valid validation it will also print the details of the ROA. In the case of failed validation (2) it will show why it failed. For example it could be an incorrect prefix length or perhaps an invalid origin AS.
Below an example of how to use the bgpmon whois service to validate ROA’s.

$ whois -h whois.bgpmon.net " --roa 28001 200.7.86.0/23"
0 - Valid
------------------------
ROA Details
------------------------
Origin ASN:       AS28001
Not valid Before: 2011-01-07 02:00:00
Not valid After:  2012-08-05 03:00:00
Trust Anchor:     repository.lacnic.net
Prefixes:         200.7.86.0/23 (max length /24)
                  2001:13c7:7012::/47 (max length /47)
                  200.3.12.0/22 (max length /24)
                  2001:13c7:7002::/48 (max length /48)

I believe that this service will be useful for those who just want to play around with ROA’s and get some experience, but also for those who are be implementing a proof of concept ROA validation in routers or other real time validation software, perhaps even in the IRRToolSet

The next step is to add ROA functionality to the BGPmon.net monitoring service. This will allow BGPmon.net users to monitor in real-time if the ROA for your prefix is still valid and will alert you in case we detect any announcements that causes the ROA validation process to fail.

So are we safe now?
ROA’s will help us make the Internet routing infrastructure more secure. By having the ability to check if the origin AS is really authorized to announce this prefix, we should be able to prevent most of the accidental hijacks. However it’s also important to realize that it’s only a first step. For example it’s still possible to bypass this form of security by spoofing (prepending) the origin AS. This works because complete path attestation against the AS_PATH is not (yet?) possible.

Last but not least, this technology and the available software is still relatively new, so please feel free to play and test with it. Let me know if you find any bugs or if you have any suggestions for improvement.

On Friday January 14th AS4761, INDOSAT-INP-AP, started to originate a large number of new prefixes. A quick check show that AS4761 originated approximately 2800 new unique prefixes of 824 unique Autonomous systems. Whereas normally they originate approximately 100 prefixes.
The announcements happened between 12:19 and 12:57 PM UTC. Some prefixes were affected longer than others,

The geographic impact of these announcements varies per prefix. Some were seen by only a few peers, where others were seen by up to 50 peers geographically dispersed all over the world. Some of the networks affected are 8.8.8.0/24 (Google open resolver), a number of AS20940 Akamai prefixes, Amazon prefixes, Cisco, DoD, US Senate, American Express, General Electric and many others.

Wondering if your network was affected by this? Here you’ll find a list of all affected networks.

A number of the transit providers of AS4761 accepted these prefixes. This is the distribution:

Residential IPv6 services in Canada

Today TekSavy, an independent Canadian service provider, is the only ISP in Canada offering experimental IPv6 services to customers. This is pretty much a global trend, as we see most IPv6 deployments and service offerings are mainly offered by large carriers, down to regional ISP’s, business providers and lastly residential users.

The reason for this is easy, high costs.  Customer premise equipment (CPE), i.e. cable/ DSL modems provided by residential providers, are typically low end devices. They do not support newer technologies such as IPv6 or even the use DNSSEC.  As a result, in order to provide IPv6 services to residential users all of the routers/modems at the customer have to be replaced. This of course is a significant investment for the providers.  In the core of these provider networks, more high-end and typically more modern equipment is used, the majority of these routers have supported IPv6 for quite a while.

So if IPv6 development mainly happens in the core, let’s take a look at the Canadian IP Transit market and how IPv6 has change the market relationships. We’ll start by taking a look at the IPv4 market followed by more detailed analyses of the IPv6 market.

Top 10 IPv4 providers
Let’s first take a look at the view of the Canadian transit market with our IPv4 glasses on. For those familiar with the Canadian market, the list below contains few surprises. The list consists of major Canadian Telco’s and Cable providers as well as most of the global Tier1 providers.

According to the table below Cogent the leads the pack, by providing transit to 159 Canadian networks (autonomous systems).  However we should probably take into account that AS577 (Bell Canada) and AS6539  (GT-BELL) are both owned by Bell.  After GT, Group Telecom, was bought by Bell both networks operated as separate networks, but they’re all Bell customers. Having said that, the real market leader is bell with 197 transit customers.

From the list below the only Transit provider that’s not a Telco / Cable or Tier1 provider is Peer1.  Peer1 is a hosting / transit provider based out of Vancouver and provides transit to 49 Canadian Networks.

The table below shows a list of providers that provide IPv4 Transit to Canadian IPv4 networks.

Top IPv6 providers

Now let’s take a look at the Canadian transit market with our IPv6 glasses on. The situation now looks quite different.  The first thing to notice is that non of the Canadian Telco’s or Cable providers are represented in this list. With the exception of Peer1 and Canarie this list has no Canadian providers at all.

Hurricane Electric is the market leader in the Canadian IPv6 Market. No real surprise, as they are the global leader.  Globally, Hurricane Electric provides transit to three times as many networks as the runner up Global Crossing. Other major players are AS3257 Tinet (formerly known as Tiscali), Tata, Global Crossing, Level3 and NTT America.

The only Canadian Transit networks on this list are Peer1 and AS6509, Canarie. Canarie is the Canadian Research and Education (R&E) network, comparable to for example Internet2 in the US and Geant/Dante in Europe. Although TATA communication has some Canadian roots (it used to be TeleGlobe) It’s fair to say that the Canadian IPv6 transit market is dominated by non Canadian companies, specifically large global carriers.

In fact most of the Canadian transit providers that are market leaders in IPv4 do not provide transit to any IPv6 network (Telus, Bell, Allstream).  The exceptions are Peer1 (providing transit to 271/BCNET, 11290/ Cogeco Cable and 36483/ gossamer Threads) and Shaw that provides transit to AS271/BCNET.

The table below shows a list of providers that provide IPv6 Transit to Canadian IPv6 networks.

Research and Education
Research and Education (R&E) networks globally have been early adaptors and in Canada Canarie is no exception as they have been running IPv6 for many years. If we zoom in a little more into the Canadian R&E community we see that in the world of IPv4, fourteen Canadian networks appear behind AS6509 (Canarie).  In the world of IPv6 that’s seven networks. This would make for a score of 50% in our IPv6 deployment scale.  Significantly better than the average Canadian Ipv6 deployment score of 8%.

Please keep in mind that these results are based on BGP data, this means that if the AS is not visible in the data,  it’s considered as not IPv6 ready. If a network uses provider aggregated address space, the actual more specific might not be visible because of aggregation. In those cases the networks will be considered as not IPv6 ready.

Conclusion

The good news for Canada is that compared to the rest of the world, Canada’s IPv6 deployment ratio of 8% is on par with the global average. It’s however significantly lower than countries it normally likes to compare itself with.
Traditionally the IP Transit market in Canada was heavily dominated by Canadian Companies. However these companies have missed the boat in the new world of IPv6. Most of the IPv6 ready Canadian networks are now forced to by transit from the larger global carriers. As more and more transit RFP’s have IPv6 as a mandatory requirement, this could very well result in loss of IPv4 customers as well.

Earlier reports
I’ve published similar reports earlier, one in October 2009, and one in April 2010. In 2009 we saw that the global deployment rate was around 4,4%, by April 2010 this had grown to 5%. We can see similar growth by just looking at the growth of the IPv6 routing table.

In the graph below we see the number of IPv6 prefixes in the global routing table over time. It’s clearly visible that over the last year the growth has increased quite a bit as compared to earlier years.

IPv6 awareness
Over the last year the push for IPv6 has become more apparent.  I believe that within the global IT industry it has gained more awareness. Likely because of the attention from industry media and mailing lists. Every now and then IPv6 related stories even make it into the mainstream media.

In addition, a number of countries have started IPv6 working groups and some have been brave enough to set IPv6 mandates.  The US government for example set mandates quite a while ago, while India has recently set some mandatory IPv6 goals.
How successful these mandates are remains to be seen, but if anything it increases awareness and a push to implement IPv6.

IPv6 deployment ratio.
Each network in the global Internet has a unique Autonomous System (AS) number. An Autonomous System can be an Internet Service Provider (ISP), Enterprise network, content provider or any other sort of network. Each AS number announces one or more prefixes.  By using ‘Geo IP’ libraries we are able to determine a country for each prefix. This in turn allows us to determine the unique number of networks (AS numbers) per country.   Doing this for both IPv4 as well as IPv6 will result in the IPv4/IPv6 deployment ratio.

Assuming that at some point the majority of the networks will be dualstack, the eventual results should be around 80 to 90%. We’re not there quite yet.

Global IPv6 deployment December 2010

IPv6 Deployment worldwide

The global IPv6 deployment rate increased from 5% in April to 7.95 % December 2010.
If we look at the results for individual countries, we see that Greenland is the winner, with a deployment rate of a 100%. According to our statistics there’s exactly one ASn active in Greenland (Tele Greenland) and it’s announcing both an IPv4 as well as IPv6 prefix.  Second place is shared by Cuba and Fiji. Both with a deployment rate of 75%. In both cases 3 out of 4 networks (ASns) doing business in Cuba and Fiji announce both an IPv4 as well as an IPv6 prefix.

If we look further in this list we can see that the top 10 is taken by relatively small (measured in Internet availability) countries with a low diversity of ASn’s, making it relatively easy to score high.

So I decided to zoom to the countries that have at least one hundred autonomous systems announcing IPv4 address space. The table below shows the results.

IPv6 deployment - 100+ Networks

Clearly visible is that European countries together with Japan, New Zealand and Taiwan are populating the top is this list.  In North America we see Canada leading the pack, with 8% pretty much equal to the global IPv6 deployment rate. The US and Mexico both score 5%. Interested in how your country scored? The complete list of results per country can be found here.

Are we on the right track?

The short answer for this is no… At this point only 8 out of a hundred autonomous systems (8%) are announcing an IPv6 prefix. Which really is only the first step in successfully deploying IPv6. It’s only going to be a number of weeks before IANA will hand out the last to /8’s to the lucky RIR(s). After that the last 5 will automatically be distributed between all 5 RIRs. At that point the global pool of IPv4 addresses will be empty. After that we’ll likely have a year or so before the RIR pool dries up and then it’s really gone.  So we still have a long way to go.

I guess there is some good news for “IPv6 consultants” as there will be lots of demand for last minute IPv6 implementations.  There are also opportunities for Transit and Hosting providers, as in this world the availability of IPv6 is not always widely available, so it’s a great way of distinguishing your product from your competitors.

Who are leading the way into the IPv6 era
So can we say something about the types of networks that are deploying IPv6? Are these mainly Transit providers, small ISP’s or stub networks. Who are leading the way into the IPv6 era?

To answer this question I’ve categorized the Autonomous Systems into the following 6 categories:

• Stub networks are network do not provide transit to other Autonomous Systems.
• tiny transit ISP’s are networks that provide transit to less than 5 (IPv4) Autonomous Systems (<5)
• small transit ISP’s are networks that provide transit to less than 10 (IPv4) Autonomous Systems (>4 && <10)
• Medium transit ISP’s are networks that provide transit to less than fifty (IPv4) Autonomous Systems (>9 && <50)
• Large transit ISP’s are networks that provide transit to less then hundred Autonomous Systems (>49 && <100)
• Super Large transit ISP’s are networks that provide transit to 100 or more Autonomous Systems (>99)

According to our analysis it’s primarily the larger ISP’s that are originating IPv6 networks and hence are taking the lead in IPv6 deployments. The distribution is actually quite nice, as can be seen in the table below. It seems that the larger the network, the higher the chance that they’ve deployed IPv6.