John Smith's Blog

I've just uploaded a small personal project to GitHub here. It's basically a very crude webserver that allows me to share audio files on my Linux boxes to my iOS devices, using Mobile Safari.

The main reason for noting this is that the code may be of more general interest because it implements an improved version of Python stdlib's SimpleHTTPServer module, that implements basic support for the Range header in HTTP requests, which is necessary for Mobile Safari on some MP3 files.

During early development, I found that some MP3 files would refuse to play in Mobile Safari when served by SimpleHTTPServer. The same file would play fine if served by Apache. Because debugging mobile web browsers is a PITA (caveat: I've haven't kept up with the latest-and-greatest in this area), I ended up resorting to Wireshark to see what was going on.

Wireshark indicated that Mobile Safari would request chunks of the MP3 file (initially just the first couple of bytes), but SimpleHTTPServer would always serve the entire file, because it never checked for the existence of the Range header. On certain files, this wouldn't bother Mobile Safari, but on others it would cause the audio player widget to show an unhelpful generic error.

Once I understood what the problem was, I found that I'm not the first person to get caught out by this, and that Apple themselves state that servers need to support Range to keep Mobile Safari happy.

To solve the problem, I wrote a new class HTTPRangeRequestHandler that is a direct replacement for SimpleHTTPServer. In my app code proper, I then (try to) pull in my enhanced handler as follows: try: import HTTPRangeServer inherited_server = HTTPRangeServer.HTTPRangeRequestHandler except ImportError: logging.warning("Unable to import HTTPRangeServer, using stdlib's " + "SimpleHTTPServer") import SimpleHTTPServer inherited_server = SimpleHTTPServer.SimpleHTTPRequestHandler ... class MySpecificHandler(inherited_server): ... def main(port=12345): Handler = EnhancedRequestHandler httpd = SocketServer.TCPServer(("", port), Handler) Arguably it might be better for the code to die if HTTPRangeServer cannot be imported, but as the stdlib SimpleHTTPServer is good enough for many browser clients, it doesn't seem too unreasonable to use it as a fallback.

This code is currently OK for most uses, but currently it doesn't support all variations of the Range header as described at aforementioned W3C spec page. It does however support all the requests variants I've seen in my - admittedly very cursory - browser testing, and any requests that it can't parse will instead get the full file served, which is the same behaviour as SimpleHTTPServer.

The musicsharer application that's built on this class is even rougher, but as it's really just intended for my own personal use, you shouldn't hold your breath waiting for me to tart it up...

On an idle Sunday afternoon, I was browsing my Twitter stream, and came across the following tweet which looked of interest: I was actually using the iPad Twitter client at the time, and when I clicked - or rather pressed - on the link, noticed a number of changes in the URL/title bar that appears at the bottom of the app window.

This piqued my interest, so I did a bit more investigation about what was happening on a platform more conducive to my nosiness. Below is a cut-n-paste from the curl session - don't worry about the details, I summarize it further down.

First off, as I think most techie people know, the link addresses that the Twitter site and apps show aren't actually what they appear to be, they are in fact using Twitter's own t.co link shortening * service. This in turn redirects to the "real" URL, except in a case such as this, the redirect is to StockTwits own shortener.

Which in turn redirects back to a different URL on t.co.

Which then redirects to Business Week's link shortener.

Which only then redirects you to the real page.

i.e. there are four HTTP redirects before you actually get to what you want. When I did some tests with curl'ing the first t.co and then following the redirects, versus getting the page directly, I was finding that the redirects were adding between 1.5 and seconds on average to the overall time taken. (In a few cases it was closer to 6-7 seconds, although this could possibly be down to one of the servers in the chain throttling back multiple requests in a short time period.)

I got similar results from Pingdom - it loaded the page faster, but still had a non-trivial delay from the redirects: Obviously, my curl tests don't reflect the real browser experience, as they don't show the time taken to load images, CSS, JavaScript etc, but the Pingdom test showed that the redirects took up a second of an overall load time of 4.6 seconds - not a non-trivial delay in my opinion.

Given that there have been any number of articles posted and tools made available for analyzing and improving web page load times, it seems crazy to have external services slowing things down - or even breaking on occasions.

Given that all of the redirects in this example were permanent, maybe Twitter should get even more aggressive with t.co, by evaluating redirect chains like this, and just having the t.co link go straight to the final destination. Browsers seem to have been caching HTTP 301s for a few years now, so maybe web services should too?

Footnote: t.co isn't actually shortening the StockTwits link, as the latter is one character shorter: [john@hamburg tmp]$ echo "http://t.co/1mkYnU0v" | wc -c 21 [john@hamburg tmp]$ echo "http://stks.co/2v3c" | wc -c 20 I guess they're opting to build up a big analytics database of popularly-clicked-links that they'll hope to monetize, over making things better (albeit in a tiny way) for their end users.

A lame update in just about every respect...

I've belatedly noticed that the tweet panel on the right hasn't been updated for a week or so. Investigating this using App Engine's local development server though, the latest tweets were pulled in fine.

The logs on the live server indicated that Twitter was returning an HTTP 400 'Bad Request' status, leading me to suspect that maybe something within the Google infrastructure was mangling the request in some way. Only by dumping the HTTP headers returned from Twitter did I find that my requests were actually being refused due to a rate-limit being breached - which means that the 400 status was a complete red herring, something like 509 'Bandwidth Limit Exceeded' would be far more accurate about describing the true cause of the problem.

Digging up the Twitter API docs, it seems that unauthenticated GET requests are rate-limited by IP address. Given that thousands of App Engine apps will effectively all share the same IP, it's hardly surprising that the limit of 150 requests (per hour?) has already been reached every time I make a request. I've been caught out by this before with other APIs, but this is the first time I've been aware of any such problems with Twitter. I guess I'll have to upgrade my code to use the OAuth or something - assuming I can be bothered.

In this supposed era of cloud computing, rate-limiting by IP seems a bit of a crummy thing to do. It'll probably be reasonable if/when IPv6 becomes the default addressing method, but when just about everyone is still using IPv4, it's a colossal pain in the backside.