Dev.to Security 🔐 Cybersecurity 👁 0 📖 13 min read

Peering vs Transit vs IXPs: How Networks Actually Connect

Peering vs transit comes down to who pays and which routes cross the link. Transit is paid access to the whole internet: a provider carries your traffic anywhere. Peering is a swap: two networks exchange traffic for thei

Peering vs Transit vs IXPs: How Networks Actually Connect

Peering vs transit comes down to who pays and which routes cross the link. Transit is paid access to the whole internet: a provider carries your traffic anywhere. Peering is a swap: two networks exchange traffic for their own customers, usually for free. An IXP is the building and the switch where that swap often happens.

Here's the part most explanations skip. On the wire, transit and peering look identical: a BGP session over a cable. The difference lives in contracts and route filters that outsiders never see, so every "upstream" or "peer" label in a dataset is a guess. (Not AWS VPC peering, by the way. Same word, different topic.)

TL;DR

  • Transit: you pay a provider for reachability to the wider internet. Depending on the service, it may send you a default route, selected routes, or a full routing table.
  • Peering: two networks exchange only their own routes and their customers' routes. Usually no money moves, though paid peering exists.
  • An IXP is a place, not a third kind of relationship. Networks peer there directly or through a route server, and the route server stays out of the AS path.
  • The rules are written down. RFC 9234 defines Provider, Customer, and Peer by what each side may send. Break the rules and you get a route leak.
  • Public relationship data is inferred. In the ASN data used below, the peers list holds every directly connected network, upstreams and customers included. The short version: transit buys reach, peering trades it, and IXPs make trading cheap. Everything you see from outside, in BGP data or an ASN lookup, is an inference about deals nobody publishes. Read the labels as well-informed guesses, and you'll avoid most of the mistakes people make with this data.

Peering vs transit in one table

IP transit Settlement-free peering
Who pays The customer pays the provider, usually on 95th-percentile usage or a committed rate Nobody pays the other side. Each covers its own port and cross-connect costs
Routes you receive A default route, selected routes, or a full table, depending on the service Normally the peer's own prefixes and its customers' prefixes
Routes you send Your prefixes and your customers' prefixes Same
What you can reach The whole internet Only the peer and the networks behind it
Where it happens Anywhere the provider has a port A private cross-connect or an IXP

The "routes you receive" row is the one that matters. Money is the reason the two relationships differ, but route export rules are how the difference actually shows up in the network. A transit provider hands you a path to everywhere. A peer hands you a path to itself and its own customers, and that's all.

That's also why "IP transit vs peering" isn't really a choice between two options. Almost every network that peers also buys transit, because peering only covers the destinations your peers can reach. Transit is the fallback for everything else.

Transit: renting the rest of the internet

What a transit provider actually sells

A transit provider sells two things in one contract. It carries your outbound traffic to any destination, and it announces your prefixes to the rest of the internet so traffic can find its way back. Without the second half, you'd have an outbound-only network.

Billing is usually by 95th-percentile usage: the provider samples your traffic, throws away the top 5% of samples, and bills on the highest one left. Burst for a few hours a month and you don't pay for it. Burst every evening and you do.

Networks that can afford it buy from two or more providers. Two upstreams means one provider's outage doesn't take you offline, and it gives you some bargaining power at renewal time. When you see a network with several upstreams in ASN data, that's what you're looking at. One wrinkle: an upstream in that data is an ASN, not a company, and one company can run several. If the provider vs. ASN distinction is fuzzy, there's a good explainer on how an ISP differs from an ASN.

Tier 1 is a peering club, not a size

A Tier 1 network reaches every destination on the internet without paying anyone for transit. It gets there by peering settlement-free with every other Tier 1, and by selling transit to everyone below. Lumen (AS3356), Arelion (AS1299), and NTT (AS2914) show up on most Tier 1 lists.

The definition is about contracts, not size. A huge eyeball network with millions of subscribers can still be a transit customer somewhere, and that makes it Tier 2. There's no registry that certifies Tier 1 status either. Like everything else in this article, outsiders infer it.

Peering: trading your customers' routes, not the internet

Settlement-free, paid, and why big networks refuse to peer down

Settlement-free internet peering works when both sides gain about equally. Two regional ISPs in the same city peer so their subscribers' traffic stays local instead of crossing a transit provider they're both paying for. Neither sends a bill, because both are saving money.

When the traffic is lopsided, the deal can turn into paid peering. A content network pushing far more traffic into an eyeball network than it receives back may pay for the privilege. It's still peering, because routes are still exchanged only for each side's own customers. Money flows, but the export rules don't change.

The refusal is the interesting part. A large transit network has no reason to peer with a network that could be paying it for transit. Every peer is a customer it didn't get. That's why peering policies at big carriers come with traffic-volume minimums and geographic requirements, and why small networks get told to buy transit.

There's also a performance argument, and it's measured. A 2017 IEEE study by Ahmed et al. compared peering and transit paths at scale and found peering paths outperformed transit paths for 91% of the ASes in their data. Peering paths had smaller propagation delays for more than 95% of ASes, mostly because they're shorter.

Bilateral, private, and route-server peering

Two networks with a lot of traffic between them often set up a private network interconnect (PNI): a dedicated cross-connect in a shared data center, used by nobody else. Everyone else peers over an IXP, either bilaterally (one BGP peering session per partner) or multilaterally through the IXP's route server.

Bilateral sessions give you control, one partner at a time. A route server gives you scale: one session, and you're peering with every other member that uses it.

An IXP is a place, not a relationship

An internet exchange point is a switching fabric inside one or more data centers. Members rent a port, plug in, and set up whatever BGP sessions they agree on with other members. The IXP itself carries traffic between ports at layer 2, and it isn't anyone's peer or anyone's provider. That's why "peering vs transit vs IXP" is a slightly broken question. Peering and transit are relationships. The IXP is where a lot of the peering happens.

Two networks peering through an IXP route server, with an AS path that skips the route server's AS number

What a route server does

Peering bilaterally with 300 members means maintaining 300 sessions. A route server collapses that to one. RFC 7947 defines it: a box that collects routes from the members connected to it and redistributes them to the others. It brokers routes but never forwards traffic. Packets still go directly from one member's router to the other's across the fabric.

Members still choose who they exchange routes with. Route servers usually let a member filter out specific networks, so "multilateral" doesn't have to mean "everyone".

Why IXPs vanish from AS paths

RFC 7947 recommends that a route server not prepend its own AS number to the routes it passes along. In a standards-compliant deployment, the AS path therefore shows the two participant ASes next to each other, much like a direct bilateral session. Legacy or non-compliant route-server behavior can differ.
The IXP switching fabric itself is not an AS hop, so it does not appear in the AS path as another transit network.
The AS path alone usually cannot tell you whether two networks interconnect through a route server, a bilateral session at an exchange, or a private cross-connect. Next-hop information and BGP communities can sometimes provide clues, while PeeringDB and the IXP's own member lists provide useful external evidence.

The rules that make it work: valley-free routing

RFC 9234 turns the export rules into protocol. It defines five BGP Roles (Provider, Customer, Peer, Route Server, and Route Server Client) and what each may send:

  • A provider may send any route it has to a customer.
  • A customer may send its provider only routes it originates itself or learned from its own customers.
  • A peer may send another peer only its own routes and its customers' routes. A route server client follows the same rule toward the route server. Notice what's missing: money. The RFC says outright that its relationship terms "do not necessarily represent business relationships based on payment agreements." They describe what may propagate where. That's the cleanest evidence for the point this article keeps making: the relationship is the route filter, and the bill is just why the filter exists.

Follow those rules and every path has the same shape: up through providers, across at most one peer link, then down through customers. Researchers call that valley-free. A path that goes down and back up, or crosses two peer links in a row, breaks the rules. That's what a route leak is: RFC 9234 defines it as, for example, announcing a route learned from one transit provider to another provider or a peer.

Route export rules: providers send all routes down, customers and peers send only their own and customer routes

RFC 9234 also adds a BGP Role capability. When both sides advertise roles and the pair does not match one of the permitted relationships, the connection is rejected. If one side does not advertise a role, the session may still be established unless strict mode is enabled. The RFC also defines an "Only to Customer" attribute for routes learned from providers, peers, or route servers, allowing compliant routers to prevent or detect leaks even several hops away.

You never see the contract

CAIDA publishes a public AS relationship dataset, and the paper behind its method puts it plainly: business relationships between networks are typically confidential, so they're inferred from BGP paths. Their 2013 method validated 34.6% of 126,082 inferred links and found the customer-provider inferences 99.6% accurate and the peer inferences 98.7% accurate.

Those are good numbers, but they come with two caveats. First, only about a third of the links could be checked against ground truth. Second, the three-way model is a simplification. CAIDA's follow-up work on complex AS relationships found 4.5% of 90,272 provider-customer links were complex: 1,071 hybrid relationships (transit in one place, peering in another) and 2,955 partial-transit deals. CAIDA calls these lower bounds, because most peering relationships are invisible.

They're invisible because of the export rules above. A peer sends a peering route only to its own customers, never up to its providers. So a peering link appears in public BGP data only if a route collector gets a feed from one of the two peers, or from somewhere in their customer trees. Every dataset that lists peers is undercounting them, and none of them can tell you by how much.

Reading upstreams, downstreams, and peers in ASN data

The ASN lookup is where all of this lands for most developers. CAIDA's AS Rank and Hurricane Electric's BGP Toolkit show relationship data in a browser. For programmatic access, I'll use ipgeolocation.io's ASN endpoint, because one call returns all three lists. It's on the paid plans and costs one credit per lookup.

The NYU example, field by field

curl -sS --max-time 15 \
  "https://api.ipgeolocation.io/v3/asn?apiKey=$IPGEO_API_KEY&asn=12&include=peers,upstreams,downstreams"
{
  "asn": {
    "as_number": "AS12",
    "organization": "New York University",
    "country": "US",
    "type": "EDUCATION",
    "domain": "nyu.edu",
    "date_allocated": "1984-07-05",
    "asn_name": "NYU-DOMAIN",
    "allocation_status": "",
    "num_of_ipv4_routes": "12",
    "num_of_ipv6_routes": "1",
    "rir": "ARIN",
    "downstreams": [
      { "as_number": "AS394666", "description": "NYU Langone Health", "country": "US" }
    ],
    "upstreams": [
      { "as_number": "AS3269", "description": "Telecom Italia S.p.A.", "country": "IT" },
      { "as_number": "AS8220", "description": "COLT Technology Services Group Limited", "country": "GB" },
      { "as_number": "AS286", "description": "GTT Communications Inc.", "country": "US" },
      { "as_number": "AS3257", "description": "GTT Communications Inc.", "country": "US" },
      { "as_number": "AS3754", "description": "NYSERNet", "country": "US" },
      { "as_number": "AS3356", "description": "Level 3 Parent, LLC", "country": "US" },
      { "as_number": "AS6461", "description": "Zayo Bandwidth", "country": "US" },
      { "as_number": "AS137", "description": "Consortium GARR", "country": "IT" }
    ],
    "peers": [
      { "as_number": "AS3269", "description": "Telecom Italia S.p.A.", "country": "IT" },
      { "as_number": "AS8220", "description": "COLT Technology Services Group Limited", "country": "GB" },
      { "as_number": "AS286", "description": "GTT Communications Inc.", "country": "US" },
      { "as_number": "AS394666", "description": "NYU Langone Health", "country": "US" },
      { "as_number": "AS3257", "description": "GTT Communications Inc.", "country": "US" },
      { "as_number": "AS3754", "description": "NYSERNet", "country": "US" },
      { "as_number": "AS3356", "description": "Level 3 Parent, LLC", "country": "US" },
      { "as_number": "AS6461", "description": "Zayo Bandwidth", "country": "US" },
      { "as_number": "AS137", "description": "Consortium GARR", "country": "IT" }
    ]
  }
}

Eight upstreams for a university is a lot, and the list explains itself once you read the names. Lumen (AS3356), Zayo, COLT, and Telecom Italia are commercial transit. NYSERNet is New York's research and education network, and GARR is Italy's. GTT appears twice, as AS286 and AS3257: one company, two ASNs. The single downstream is NYU Langone Health, a separate network that routes through NYU.

Two field quirks will bite you. as_number carries the AS prefix, so it's a string. num_of_ipv4_routes and num_of_ipv6_routes are strings too, despite being counts.

Why your upstream is also in your peer list

Look at peers again. It has nine entries, and every one of them is either an upstream or the downstream. The docs' field reference describes peers as directly connected ASes, and the live data matches that exactly. In this dataset, peers means "neighbors", not "settlement-free peers".

To find lateral peers, subtract the upstreams and downstreams. For NYU that leaves nothing. That doesn't prove NYU has no peers. Given the previous section, it more likely means none of its peering links are visible from where the data was collected.

Lumen's record shows how far the labels drift at the top of the hierarchy. AS3356 lists 6,544 downstreams and 185 upstreams. Its peers list opens with Gilead Sciences, Aflac, SpaceX, and ExxonMobil, which look a lot more like transit customers than settlement-free peers. And a Tier 1 network shouldn't have any upstreams at all. Either the inference is reading Lumen's peers as providers, or some of those links are the hybrid relationships CAIDA describes. I don't know which method produces these lists, and the docs don't say. From the outside, you can't tell the two apart.

This script runs the subtraction for any list of ASNs:

import os
import sys

import requests

API_URL = "https://api.ipgeolocation.io/v3/asn"
API_KEY = os.environ.get("IPGEO_API_KEY")


def neighbor_sets(asn):
    """Return (upstreams, downstreams, lateral peers) as sets of AS numbers, or None."""
    try:
        resp = requests.get(
            API_URL,
            params={"apiKey": API_KEY, "asn": asn, "include": "peers,upstreams,downstreams"},
            # A Tier 1 record lists thousands of neighbors, so allow a long read.
            timeout=(2.0, 15.0),
        )
        resp.raise_for_status()
        record = resp.json().get("asn") or {}
    except requests.HTTPError as exc:
        # Do not print the exception object: it may contain the URL and API key.
        status = exc.response.status_code if exc.response is not None else "unknown"
        print(f"AS lookup for {asn} failed: HTTP {status}", file=sys.stderr)
        return None
    except requests.RequestException as exc:
        # Log the exception type only, never the request URL.
        print(
            f"AS lookup for {asn} failed: {type(exc).__name__}",
            file=sys.stderr,
        )
        return None
    except ValueError:
        print(f"AS lookup for {asn} returned invalid JSON", file=sys.stderr)
        return None

    def numbers(key):
        # A set dedupes on as_number, so one AS listed twice is counted once.
        return {n.get("as_number") for n in (record.get(key) or []) if n.get("as_number")}

    upstreams, downstreams = numbers("upstreams"), numbers("downstreams")
    # `peers` holds every neighbor, so what's left after both subtractions is the lateral set.
    return upstreams, downstreams, numbers("peers") - upstreams - downstreams


if __name__ == "__main__":
    if not API_KEY:
        sys.exit("Set IPGEO_API_KEY before running this.")
    for asn in sys.argv[1:] or ["12"]:
        result = neighbor_sets(asn)
        if result is None:
            continue
        up, down, lateral = result
        print(f"{asn}: {len(up)} upstreams, {len(down)} downstreams, {len(lateral)} lateral peers")

Save it as neighbors.py and run python neighbors.py 12 3356. AS12 comes back as 8 upstreams, 1 downstream, and 0 lateral peers. The lateral count is the closest this data gets to "who does this network peer with". Read it as a floor, not a total.

The ASN Pivot Playbook covers the same arrays from the investigation side: which pivots carry weight and which don't.

What this tells you about a network

A few readings hold up even with inferred data:

  • One upstream, no downstreams: a stub network. It goes down when that one provider does, which matters if you depend on it.
  • Hundreds or thousands of downstreams: it sells transit. An IP announced by that ASN may belong to the transit provider itself or sit in a customer's space announced on its behalf, so check routes before blaming the provider.
  • Zero upstreams: a Tier 1, or a gap in visibility. And as Lumen's 185 shows, upstream counts near the top of the hierarchy tell you less than they seem to. The relationship lists also pair well with the type field, covered in Classify an IP by ASN Type: Hosting, Business, ISP. A HOSTING network with thousands of downstreams is a different animal from a HOSTING network with none.

Treat every upstream, downstream, or peer label as a dated hint, not a fact about a contract. Re-pull it before a decision depends on it, and when two datasets disagree about a relationship, assume both are partly right.

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.