Recursive curl with Tor on Apple OSX

The Apple OSX TOR Bundle for Anonymous browsing

The Tor Apple OSX Tor Bundle is a stripped Firefox browser that uses a local SOCKS proxy to anonymize the requests.

The SOCKS proxy that is used is tor.real, located at /Applications//TorBrowser.app/TorBrowser/Tor/tor.real.

/Applications//TorBrowser.app/TorBrowser/Tor/tor.real

Mar 10 23:50:29.711 [notice] Tor v0.2.5.10 (git-13318a95ddfbbf8d) running on Darwin with Libevent 2.0.21-stable, OpenSSL 1.0.1l and Zlib 1.2.5.
Mar 10 23:50:29.711 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Mar 10 23:50:29.801 [notice] Configuration file "/home/ubuntu/install/etc/tor/torrc" not present, using reasonable defaults.
Mar 10 23:50:29.806 [notice] Opening Socks listener on 127.0.0.1:9050

SOCKS proxy

Anonymous browsing is good but I needed a command line tool to fetch a web page or web site. More specific, I want to recursively download a website from the command line. Ideally you use wget for this. Unfortunately wget does not support a socks server. The other command line tool that allows downloading a website is curl. Curl has SOCKS5 support. You can do this with

curl --socks5 127.0.0.1:9150 http://www.google.com

Unfortunately curl does not allow recursive downloading.

Recursive website download with curl

There is a perl script curlmirror.txt that allows recursive curl downloads. It’s written by Kjell Ericson. I adjusted the script to include socks support for a local Tor proxy (the -T option). You can download it from Github.

The usage is very simple

perl curlmirror.txt -T  http://www.google.be

This setup allows you to recursively download a website from an Apple OSX system with the Tor browser bundle installed.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.