# A1 Website Download / Help / Website crawler scanner filters

##

## General about website crawler filters

The website crawler filters differentiates between internal and external paths.

For internal paths, the crawler filters are applied to to the relative part.

If the root directory scanned is http://www.micro-sys.dk/ then http://www.micro-sys.dk/blogs/ is an internal path where blogs/ is the relative part.

## Allow and crawl internal links that match certain paths

As default, the allow list does not contain any items (and thus ignored).

However, if the allow list has any items, all links "outside" filter will not be crawled.
In those cases, you should also use: Initiate website crawling from multiple paths.

## Disallow and ignore links that match certain strings or paths

- Strings: blogs ignores all paths containing the text "blogs".
- Paths: :blogs/ ignores all paths starting with http://www.micro-sys.dk/blogs/.
- Subpaths: :blogs/* accepts http://www.micro-sys.dk/blogs/ but works like above in all other cases.
- Regex (regular expressions): ::blog(s?)/ accepts http://www.micro-sys.dk/blogs/ and http://www.micro-sys.dk/blog/.
- More examples
  - Strings
    - @ ignores all paths containing the text "@".
    - ? ignores all paths containing the text "?".
  - Special
    - : (when alone)
      - Relative links: Paths will be ignored if ":" is found before any "?".
      - Full links: All paths will be accepted.

If some parts of your website are only linked from filtered pages, you can hinder the website crawler in finding those parts. In such cases you may benefit from:

- Limit pages categorized as sitemap with output filters.
- Initiate website crawling from multiple paths.

## Webmaster crawler filters

- If checked the robots.txt option, A1 Website Download will respond to:
  - Exact user agent id match as defined in website crawler identification.
  - If the user agent id contains A1 Website Download that is matched as well
  - * (all crawlers)

Copyright © 2006 Thomas Schulz (see http://www.micro-sys.dk), all rights reserved.