This article is a follow up to Invaluable command line tools for web developers, and covers some more great tools that can make your life as a developer that little bit easier.
This post first appeared, combined with the first command line tools post, on Smashing Magazine.
Netcat, or nc, is a self described networking swiss-army-knife. It's an very simple but also very powerful and versatile application that allows you to create arbitrary network connections. Here we see it being used as a port scanner:
$ nc -z example.com 20-100
Connection to example.com 22 port [tcp/ssh] succeeded!
Connection to example.com 80 port [tcp/http] succeeded!
In addition to creating arbitrary connections netcat can also listen for incoming connections. Here we use this feature of nc, combined with tar, to very quickly and efficiently copy files between servers. On the server run:
$ nc -l 9090 | tar -xzf -
And on the client:
$ tar -czf dir/ | nc server 9090
We can use netcat to expose any application over the network. Here we expose a shell over port 8080:
$ mkfifo backpipe
$ nc -l 8080 0<backpipe | /bin/bash > backpipe
You can now access the server from any client:
$ nc example.com 8080
uname -a
Linux li228-162 2.6.39.1-linode34 ##1 SMP Tue Jun 21 10:29:24 EDT 2011 i686 GNU/Linux
While the last two examples are slightly contrived (in reality you'd be more likely to use tools such as rsync to copy files, and ssh to remotely access a server) they do show the power and flexibility of netcat, and hint at all of the different things you can achieve by combining netcat with other applications.
Sshuttle allows you to securely tunnel your traffic via any server you have SSH access to. It's extremely easy to setup and use, not requiring you to install any software on the server, or change any local proxy settings.
By tunnelling your traffic over SSH you secure yourself against tools like firesheep and dsniff when on unsecured public wifi or other untrusted networks. All network communication, including DNS requests, can be sent via your SSH server:
$ sshuttle -r <server> --dns 0/0
If you provide the --daemon argument sshuttle will run in the background as a daemon. Combined with some other options you can make aliases to simply and quickly start and stop tunnelling your traffic:
alias tunnel='sshuttle --D --pidfile=/tmp/sshuttle.pid -r <server> --dns 0/0'
alias stoptunnel='[[ -f /tmp/sshuttle.pid ]] && kill `cat /tmp/sshuttle.pid`'
You can also use sshuttle is to get around the IP-based-geolocation filters that are now used by many services, such as BBC's iPlayer which requires you to be in the UK, and turntable.fm requires you to be in the US. You'll need access to a server in the target country. Amazon has a free tier of EC2 micro instances that are available in many countries, or you can find a cheap VPS in almost any country in the world.
In this scenario rather than tunnelling all of your traffic you might want to send only that for the service you are targeting. Unfortunately sshuttle only accepts IP address arguments rather than hostnames, so we need to make use of dig to first resolve the hostname:
$ sshuttle -r <server> `dig +short <hostname>`
mitmproxy is an SSL-capable man-in-the-middle HTTP proxy that allows you to inspect both HTTP and HTTPS traffic, and rewrite requests on the fly. The application has been behind quite a few different iOS app privacy scandals, including Path's address book upload one. It's ability to rewrite requests on the fly has also been used to target iOS, including setting a fake a high score in GameCenter.
Far from only being useful to see what mobile apps are sending over the wire or for faking high scores, mitmproxy can help out with a whole range of web development tasks. For example, instead of constantly hitting F5 or clearing your cache to make sure you're seeing the latest content you can run
$ mitmproxy --anticache
which will automatically strip all cache control headers and make sure you always get fresh content. Unfortunately it doesn't automatically setup forwarding for you like sshuttle, so after starting mitmproxy you still need to change your system wide, or browser specific proxy settings.
Another extremely handy feature of mitmproxy is the ability to record and replay HTTP interactions. The official documentation gives an example of a wireless network login. Exactly the same technique can we used as a basic web testing framework. For example, to confirm that your user signup flow is works you can start recording the session:
$ mitmdump -w user-signup
Then go through the user signup process, which at this point should work as expected. Stop recording the session with Ctrl-c. At any point we can then replay what was recorded and check for the 200 status code:
$ mitmdump -c user-signup | tail -n1 | grep 200 && echo "OK" || echo "FAIL"
If the signup flow gets broken at any point we'll see a FAIL message, rather than an OK. You could create a whole suite of these tests and run them regularly to make sure you get notified if you ever accidentally break anything on your site.