# A1 Website Download / Help / Scan and crawl all website pages

##

We use our own website crawler engine called Miggi / MiggiBot.

- Choose if the website crawler accept cookies and redirects.
- Supply username / password to access protected site sections.
- Choose if crawler should obey the "robots.txt" file if found.
- Change user agent identification string to. Sometimes servers output data based on the user agent ID.
- Scan and crawl local and online pages on internet, localhost, LAN (local area network), CD-ROM / DVD, disks and UNC paths.
- Scan static and dynamic websites. Can also handle dynamic CMS / database generated sites.
- Crawler can handle both HTTP and HTTPS (HyperText Transmission Protocol / Secure) pages in website scan.
- Use multiple simultaneous connections / threads to optimize the website crawler speed.
- Crawl and index all kinds of references including frames, stylesheets, javascripts and images.
- Retrieve sitemap data across multiple domains by using "root alias" and multiple crawler paths.
- Use various options to limit and prevent the scanner from crawling certain pages and directories.
- Decide with filters which kind of files to include in generated sitemap output data.
- Automatic link checker / link validation. After site scan, view website data for internal and external broken links.

Copyright © 2005-2006 Thomas Schulz (see http://www.micro-sys.dk), all rights reserved.