ISP Tech Talk

by Avi Freedman

SATELLITE-BASED BANDWIDTH AUGMENTATION

EDITOR'S NOTE: This is a continuation of Avi Freedman's April column.

s ince most of inbound content is probably porn, I suppose you could call it breast augmentation and not be far off. Anyway, two of our T-3 customers practice it. Basically, in Europe and Australia, most ISPs get in 3-10 times as much data as they push out. And they pay between $10,000 and $40,000 per Mbps of bandwidth! "Satellite Half- duplex Bandwidth Augmentation" is the practice of taking satellite time (which is cheaper than transoceanic fiber) and sending data from the U.S. to the foreign ISPs, one direction. Because the data is only flowing to those ISPs via satellite, the overall round-trip time (latency) isn't too terrible.

TYPICAL ISP BANDWIDTH USE WHERE BANDWIDTH IS EXPENSIVE

Putting web farms where bandwidth is expensive is not a swift idea. So we're going to assume for this discussion that that web sites are presented off in (optionally replicated) web farms somewhere.

Now, if your bandwidth is expensive because of high local-loop costs through Billy Bob's telco in the U.S., or because of both transoceanic pipe cost and governmental extortion (ah, tariffs) in European and Asian countries, you're typically going to host content somewhere cheap, like the U.S., if you find your outbound bandwidth use creeping up.

But your core functionality, providing mail, news reading, and web browsing to dial-up and corporate customers - primarily the web browsing part - involves taking much more data than you send.

For each web request, you send out maybe 80-200 characters and can get back 1K, or more typically 10K or 100K! This is a huge inbound:outbound ratio.

So, let's say you're caching already, but you still need about 2 Mbps of inbound-from-the-Internet bandwidth. One approach is to get a 2 Mbps pipe to a local or national provider, costing anywhere from $5,000 (if you're in the boonies in the U.S.) to $20,000 to $80,000 or more outside of the U.S.

Or you can get a dial-up ISDN line or two, or a usage-based T-1 or E-1, to a provider and contract for $10,000 to $40,000/mo satellite inbound bandwidth. Your outbound data flows over your terrestrial pipe, and so does some of your inbound data.

But your satellite-bandwidth provider tries to be your "best path," and advertises your routes in the U.S., then aggregates traffic destined for you with traffic destined for other ISPs and sends it up to a satellite, usually in 8 Mbps "streams."

The receive-only gear costs us$10k or so, give or take, plus cabling costs. And you need a router that can receive an 8 Mbps data stream, which isn't cheap.

There are various ways of encoding those streams, ranging from Frame Relay to IP tunneling. And you have to be sure your provider isn't overselling capacity (or, at least, that you always have the bandwidth you're contracting for available to you).

But that's the basic idea.

I hope to have a powerpoint presentation available soon with a network architecture for OGN's (an Australian provider-to-providers) delivery of half-duplex satellite-delivery bandwidth augmentation available in a few months.

WHAT ABOUT LATENCY?

Geosynchronous satellites are way out there. Tens of thousands of miles out. It takes a long time, relatively speaking, to get data out to them and back. Each "hop" is about 120-150 ms, including equipment latency. So, to send data from the U.S. to Australia or Europe via satellite takes 300ms (150ms up and 150ms down). Then, to send it back over fiber or copper takes 100-150ms. The total is 400ms or so on average.

In a "full-duplex" satellite application, the basic latency is 600ms (150ms up and 150ms down, then on the return path 150ms up and 150ms down).

It turns out that 400ms as a total round-trip time is a lot better for TCP utilization and perceived throughput than 600ms.

Also, 400ms with almost-zero packet loss is a few thousand percent better than 80ms with even 5-10 percent packet loss, even for interactive applications!

WHAT ABOUT SECURITY?

So what about security? If the packets are uplinked to the satellite as raw IP out of a router speaking HDLC, PPP, or Frame, any standard router can receive the IP data stream.

Is this really a problem?

Well, if they are stupid, they'll flood their Ethernet and upstream connections of they monitor your bandwidth (if they transfer the packets from the serial port of a router to their Ethernet).

Even if they're smart, they're still seeing only one part of the communication - the inbound. So it's hard(er) to capture passwords and e-mail by sniffing, but still possible.

The larger problem is that someone could send packets into a data stream and then listen for those packets, thus stealing valuable bandwidth.

Two easy approaches are available, neither of them cheap.

(1) Get a compression card, and compress the data. (2) Get an encryption card, and encrypt the data.

In either case, it'll be hard to unpack by a thief.

Surprisingly, most satellite IP links to Australia, Asia, and Europe run unencrypted.

WHO DOES IT?

There are now others getting into the business, but (conflict of interest warning) two of our customers were early pioneers, which is how we gained knowledge and experience with the market.

Netsat, http://www.internetsat.net, serves primarily European ISPs, and actually does bidirectional satellite transit for many in addition to bandwidth augmentation. NetSat has most customer ISPs receive the data streams themselves, and shapes/guarantees inbound traffic with Frame Relay switches.

Ourworld Global Network (OGN), http://www.ogn.net, in Australia has a slightly different model. They receive the (currently) 8 MB data streams at their core POPs in Sydney and Melbourne, and redistribute to customers terrestrially, usually with PAPL (basically, the Australian version of DSL over alarm circuits) 2 MB links in each city.

Both companies have almost entirely ISPs as customers, and don't cater to the business marketplace currently.

TECHNICAL TRICKINESS IN THE LAND OF EXPENSIVE BANDWIDTH

I actually had a bit of fun setting up the OGN network for satellite bandwidth augmentation. In fact, the whole Australian ISP economics make for interesting technical `necessities', including caching, reverse caching, and other various hackeries.

It's tricky to get half-duplex links to come up, and even trickier to make them stop being used when they're down. Some ugly static routes and IP tunnels allow routing information to flow only when the satellite connections are up, and weighting the routes heard via various IP tunnels allow for dynamic rerouting to a different city when rain fade hits one city hard.

Also, it's cheaper to advertise your routes to Telstra, the dominant Australian carrier, in the U.S. and then beam it back over satellite - but that screws up Quake users pretty badly.

Anyway, sometimes U.S. routing gets "boring," so working with these customers keeps us on our toes...