John Smith's Blog

Ramblings (mostly) about technical stuff

Twitter feed on this blog broken

Posted by John Smith on

A lame update in just about every respect...

I've belatedly noticed that the tweet panel on the right hasn't been updated for a week or so. Investigating this using App Engine's local development server though, the latest tweets were pulled in fine.

The logs on the live server indicated that Twitter was returning an HTTP 400 'Bad Request' status, leading me to suspect that maybe something within the Google infrastructure was mangling the request in some way. Only by dumping the HTTP headers returned from Twitter did I find that my requests were actually being refused due to a rate-limit being breached - which means that the 400 status was a complete red herring, something like 509 'Bandwidth Limit Exceeded' would be far more accurate about describing the true cause of the problem.

Digging up the Twitter API docs, it seems that unauthenticated GET requests are rate-limited by IP address. Given that thousands of App Engine apps will effectively all share the same IP, it's hardly surprising that the limit of 150 requests (per hour?) has already been reached every time I make a request. I've been caught out by this before with other APIs, but this is the first time I've been aware of any such problems with Twitter. I guess I'll have to upgrade my code to use the OAuth or something - assuming I can be bothered.

In this supposed era of cloud computing, rate-limiting by IP seems a bit of a crummy thing to do. It'll probably be reasonable if/when IPv6 becomes the default addressing method, but when just about everyone is still using IPv4, it's a colossal pain in the backside.

About this blog

This blog (mostly) covers technology and software development.

Note: I've recently ported the content from my old blog hosted on Google App Engine using some custom code I wrote, to a static site built using Pelican. I've put in place various URL manipulation rules in the webserver config to try to support the old URLs, but it's likely that I've missed some (probably meta ones related to pagination or tagging), so apologies for any 404 errors that you get served.

RSS icon, courtesy of RSS feed for this blog

About the author

I'm a software developer who's worked with a variety of platforms and technologies over the past couple of decades, but for the past 7 or so years I've focussed on web development. Whilst I've always nominally been a "full-stack" developer, I feel more attachment to the back-end side of things.

I'm a web developer for a London-based equities exchange. I've worked at organizations such as News Corporation and Google and BATS Global Markets. Projects I've been involved in have been covered in outlets such as The Guardian, The Telegraph, the Financial Times, The Register and TechCrunch.

Twitter | LinkedIn | GitHub | My CV | Mail

Popular tags

Other sites I've built or been involved with


Most of these have changed quite a bit since my involvement in them...