SiteCrawler Editor's Review

SiteCrawler is a website downloader application that lets you capture entire sites or selected portions, like image galleries.

The fact that is a little bit different about this site downloader is the integration with Safari. By using it in this mode you can use session cookies and you may crawl in portions of the websites where cookies are required for authentication.

Another thing that I do like about this program is the fact that it can be set up very easily. The interface has just a few tabs where you can find all the options you need.

In the first tab you need to set the download path, the website URL, the crawling scope and the number of concurrent downloads. Setting the URL functionality is very well implemented since it has the support for URL patterns. The crawling scope gives you the possibility to pick where the program should crawl: same directory, same sub-domain, same domain or any domain.

In the second tab you need to set up the restrictions, such as: restrictions per session and restrictions per file. When you set up the restrictions per file you have the opportunity to set the maximum connection time and the maximum size of the file. The session restrictions are about the maximum time of connectivity, the number of files, the maximum size of the files and the crawling level.

In the next tab you can set advanced filters for the downloads. You can set there various filters in order to suit your needs.

In the last tab you can set post-processing options such as: archiving into a zip file the downloaded files and if you pick this option you can pick if the application should or shouldn't delete the original files. Other post-processing options are: run an Apple script or run a shell command. You can set in this tab the user agent that is reported to the hosting server and you can manually set the HTTP headers that the application sends to the HTTP server.

After setting the application all you need to do is to press the 'Start' button. If you would like to do the same job in the future you can save all your settings into a project file.

Pluses: is a strong application that can do a pretty good job when downloading a site, it has a good Safari integration in order to use session cookies, it is built as a universal binary so it runs both on PPC and Intel Macs.

Drawbacks / flaws:

In conclusion: a pretty good tool that has two advantages: the fact that is very easy to use but in the same time it is featuring advanced options.

version reviewed: 1.0.1