Large hijack affects reachability of high traffic destinations

Posted by Andree Toonk - April 22, 2016 - Hijack - 10 Comments

April 23, Update: NOC Team at innofield posted an explanation of the Incident in the comments section below.

Starting today at 17:09 UTC our systems detected a large scale routing incident affecting hundreds of Autonomous systems. Many BGPmon users have received an email informing them of this change.

Our initial investigation shows that the scope of this incident is widespread and affected 576 Autonomous systems and 3431 prefixes. Amongst the networks affected are high traffic prefixes including those of Google, Amazon, Twitter, Apple, Akamai, Time Warner Cable Internet and more.

All these events have either AS200759 “innofield AG” or private AS 65021 as the origin AS. In the cases where AS65021 appears as the origin AS, AS200759 is again the next-hop AS.

AS200759 “innofield AG”  is a provider based out of Switzerland and normally only announces one IPv4 and one IPv6 prefix.

These are 2 example events:

Prefix  66.220.152.0/21 Is normally announced by Facebook AS32934 and during this event was announced by AS200759 as a more specific /22

Detected prefix: 66.220.152.0/22
Example aspath: 4608 24130 7545 6939 200759

And AS origin: 65021 behind AS 200759

Detected prefix: 66.220.152.0/22
Example aspath: 133812 23948 4788 6939 200759 65021

We saw the announcements via the following peers of  AS200759 “innofield AG”:

  • 20634 “Telecom Liechtenstein AG”
  • 6939 “Hurricane Electric, Inc.”
  • 16265 “LeaseWeb Network B.V.”

Not surprisingly, as HE is a major provider most of our probes (BGPmon peers) detected this path via their provider HE (6939).
It appears things have been resolved as of 17:30 UTC.

This event affected the reachability of many high traffic destinations, some good examples are posted on the outages.org mailing list. In this example posted by Frank Bulk we see how in his case amazon.com (54.239.16.0/20) is unreachable. The traceroute posted demonstrates how his traffic is routed via HE (6939) to towards Zurich, Switzerland where it eventually stops.

Since AS200759 (innofield AG) is connected to the SwissIX it’s likely they announced these prefixes to the route server there and as a result other peers such as HE picked it up from there. Since these are more specific announcements they are preferred over the original route even if the AS path is longer.

Below you’ll find an example email and screenshot that BGPmon users would have received alerting them of the incident in near real time.

====================================================================
Possible Prefix Hijack (Code: 10)
====================================================================
Your prefix: 199.16.156.0/23:
Prefix Description: twitter
Update time: 2016-04-22 17:10 (UTC)
Detected by #peers: 19
Detected prefix: 199.16.156.0/24
Announced by: AS65021 (-Private Use AS-)
Upstream AS: AS200759 (innofield AG)
ASpath: 58786 9957 6939 200759 65021

bgpmon-alert

10 comments

  • […] was caused by a very large-scale network outage at a major Internet backbone company. It affected traffic to a significant percentage of the Internet, including as Amazon, Facebook, […]

  • Chris says:

    Another route optomizer gone wrong?

  • alnc says:

    thank for the monitoring and for the explanation 🙂

  • Hi all

    Here is the NOC Team from innofield.

    First of all: we sincerely apologize for the inconvenience this have caused.

    Background of this incident:
    – Yesterday (2016-04-22) SwissIX (Swiss Internet Exchange) has performed a maintenance on our port and we had to de-activate and re-activate the BGP sessions.
    – After re-activating the sessions at approx. 17:10 UTC, for a currently unknown reason, we have redistributed other prefixes into SwissIX.
    – This definitely shouldn’t happen, as a matter of course we have filters in place for SwissIX (which only allow to send our prefixes).
    – Immediately after we received the first complains about the route-hijack, we have de-activated the BGP sessions again (at approx. 17:25 UTC)

    Next steps at our side:
    – We are in permanent contact with the router and the route-optimizer vendors. Together, we are deeply analyzing the system/logs why the filters were not considered after re-activating the BGP sessions.
    – Currently (but not confirmed yet) it looks like a code-bug in the router software.
    – We will definitely not re-activate the sessions until the root-cause is found!
    – With this planned actions we will prevent this from happening again.

    Again we are very sorry for this incident and we will do everything you can think of to avoid this situations in the future.

    Respectfully,
    NOC Team from innofield AG

  • Init7 AS13030 is one of the transit suppliers of Innofield AS200759. We double-checked the filters and it seems that we didn’t leak the hijacked prefix. Therefore we wonder why the path _13030_200759_ was seen during the hijack.

    We also see only an increase of about 1gig inbound traffic during the hijack. I wasn’t even detected as this is below threshold. Considering the importance of the hijacked prefix of Facebook, I think the malicious traffic went mostly through SwissIX infrastructure.

    • Andree Toonk says:

      Hi Freddy,

      Thanks for reaching out. Yes you are correct, it appears AS13030 was erroneously mentioned, our apologies. AS13030 was detected as an upstream for legitimate AS200759 prefixes only. Good job filtering! I’ve updated the list accordingly.

  • Lukas Tribus says:

    At least HE and Core-backbone don’t usually peer with routeservers, so its more likely that we are talking about dedicated peering session. In fact, the peering with HE is documented in innofield’s aut-num object (meaning it was manually put there).

    Also, the aut-num references a AS-SET that doesn’t exist: AS-FLOW.

    Init7 is actually a transit of innofield, so it appears they don’t filter from their customers.

  • […] 前幾天 BGPmon 偵測到大規模的 routing-hijacking 事件:「Large hijack affects reachability of high traffic destinations」,範圍應該是最近最大的: […]

Leave a Reply

Your email address will not be published. Required fields are marked *