Web crawlers are not that hard to understand. Despite many technical presentations about what Web crawlers are, it mainly boils down to a very simple definition: a program that remembers all the visited pages on the Internet. Thus it is a helpful way to make browsing more efficient by creating copies of visited URLs and saves them for later use.
The likes of the Yahoo Crawler, Google Crawler, Fast Crawler, Methabot, Polybot and WebFountain are among the most common Web crawlers. During a Web crawl process, the Web administrator examines their Web logs to determine which Web crawlers visited their site and how often. Web administrators are often aware of the nature of such visits unless the crawler is a spambot that hides the true identity of the crawler.
The popularity of Web crawlers can be mainly attributed to the fact that these crawlers can retrieve data a lot quicker than the traditional click-and-search. Experts say that the average access intervals in using such increased speeds vary from 20 seconds to 2 minutes, thus making lesser waiting time for a typical researcher.
To avoid having to crawl the same pages multiple times, Web crawlers have a URL normalization processes, also known as URL canonization to avoid such redundancies. Thus in addition to choosing the best Web crawler to use, knowing how often the Web crawler normalizes will definitely save you bandwidth space and avoid server overload.
Other classic considerations in choosing the Web crawler are the amount of network resources you have, the method of how the Web crawler was written, and other personal crawlers already installed on your computer.
There are many Web sites offering free Web crawler downloads such as soft32.com, sharewareconnection.com, cs.cmu.edu and a lot more. Try looking it up over at you favorite search engines to find available sites to download it from. One quick way to know if the download is safe for your computer is to read on the site's forums. Check the user comments about the download and decide for yourself if it is worth your time.
To download your own Web crawler, make sure that your pop-up blocker is disabled. This can be done by right clicking on the yellow bar at the top of your screen and choosing the "Download File" option. You may use any download accelerator that you have or simply download one. The site offering you the Web crawler download will usually suggest one and provide you with a link to it.
A window will then appear prompting you to either Run or Save the file. Choose Save and for Mozilla Firefox users, another dialog box will appear. Just click OK and wait until the download finishes. Once the download finishes, simply look it up in your downloads directory or if you don't have one, just make one for your future downloads. If you experience any problems on the download or if it doesn't work, you may contact the Web administrator where you downloaded it from or simply choose another site.