Smart Document Crawler
Within Smart Document Access, you can configure the crawler to index the content of a website.
The crawler operates in three distinct phases: a first link navigation phase, which maps the site structure, a second content extraction phase, which identifies useful content, and a third PDF conversion phase, which generates indexed documents.
1. Link navigation
The crawler follows internal site links and maps the structure, without extracting content.
2. Content extraction
Navigated pages are analyzed to identify useful content.
3. PDF conversion
Extracted content is converted into indexed PDF documents. If a page contains no extractable content, no document is generated.
Activation and structure creation
Inside an already created subfolder, select the Configure Smart Crawler option.
Enable the Enable on subfolder toggle.
Select Add configuration.
Assign a name to the configuration and select Save.
At this stage you can import previously downloaded configurations or download existing ones.
Setting up the Crawler
Selecting Edit fields opens the Crawler configuration.
General
In the General section you define the URLs the crawler starts from.
Enter the starting URL of the site in the Address field.
In the Additional pages field you can enter other starting URLs:
- A single URL: the crawler starts from that URL and applies the Navigation and Extraction settings.
- Several URLs: the crawler starts from each of the URLs listed and applies the same navigation and extraction settings to each of them.
When the pages to extract are few and known, it can be useful to enter them directly as starting URLs, instead of starting from the sitemap, the home page or container pages that list them.
Navigation
In the Navigation section you tell the crawler which pages to visit to collect the links they contain.
The crawler starts from the URLs defined in the General section and collects their links. Among the links found, it visits only the pages that match the Addresses to include and, in turn, collects their links. The process repeats level by level, up to the maximum depth set.
The starting URLs must also appear in the Addresses to include: otherwise the crawler starts, but does not navigate even the starting page and collects no links.
A page therefore becomes reachable only if its link appears in a page the crawler has navigated. Likewise, a page can be converted into PDF in the Extraction phase only if its URL was collected during navigation.
Example — page hierarchy:
example.com/sitemap ← starting URL
└── example.com/products ← child: linked in the sitemap
└── example.com/products/algho ← grandchild: linked only in /products
You want to extract the page example.com/products/algho, which does not appear in the sitemap but is linked only in the page example.com/products:
- If the Addresses to include contain only
https://example.com/sitemapandhttps://example.com/products/algho, the crawler never reaches the page: from the sitemap it finds the link to/products, but does not navigate it and therefore does not collect its links. - If the Addresses to include also contain
https://example.com/products, the crawler navigates/products, collects the link to/products/alghoand the page becomes available for extraction.
Alternatively, example.com/products or example.com/products/algho can be entered directly in the Additional pages of the General section, remembering to also add them to the Addresses to include.
Enable the Exclude off-domain sites option to limit navigation to links within the main domain.
Enable the Include PDF, doc, docx documents option to also navigate documents attached to web pages.
In the Addresses to include field, enter the list of site pages to navigate.
In the Addresses to exclude field, enter the list of site pages not to navigate.
Both fields accept regular expressions (regex) to define URL patterns. Before creating regex, it is useful to analyze the site's path structure: path segments reveal page categorization.
Example — site structure:
example.com/use-cases
example.com/products/algho
example.com/products/sda
example.com/products/booking
example.com/products/pricing
example.com/blog/first-post
example.com/docs/getting-started
example.com/docs/getting-started-api
example.com/about
example.com/contact
Main pattern types:
| Pattern | Example | Matched URLs |
|---|---|---|
| Exact match | https://example.com/use-cases | https://example.com/use-cases |
| Alternation | https://example.com/products/((algho)|(sda)|(booking)) | example.com/products/algho, example.com/products/sda, example.com/products/booking |
Wildcard with .* | https://example.com/products/.* | example.com/products/algho, example.com/products/sda, example.com/products/booking, example.com/products/pricing |
| Shared prefix | https://example.com/docs/getting-.* | example.com/docs/getting-started, example.com/docs/getting-started-api |
The structure of each regex follows the schema: https://[domain]/[segment1]/[segment2]/...
| Component | Meaning |
|---|---|
https:// | HTTP or HTTPS protocol |
domain | Domain name |
.* | Any text (generic wildcard) |
(a|b|c) | Alternation: one of a, b, or c |
A regex should be tested against the entire set of site URLs, not just the known pages. This reveals false positives: unwanted pages that the regex captures anyway.
For example, https://example.com/products/.* captures both products/algho (desired) and products/pricing (false positive). If pricing is not desired, the alternation pattern is preferable.
Principles for choosing the best regex:
- Prefer simplicity: a short, readable regex is better than a complex one.
- Always test against the full crawl: a regex that works only on known pages is unreliable.
- Avoid overly generic regex:
https?://.*captures the entire site, which is not useful. - Consider future growth: a wildcard on a category will also capture pages added after the rule is defined.
Set the maximum depth and the maximum number of pages to navigate.
Extraction
In the Extraction section, you configure the rules the Crawler will follow to transform web pages into documentation.
In the Filter addresses to include field, enter the list of pages to extract and convert into documentation.
Both fields accept the same regular expressions described in the Navigation section. The pages to extract can coincide with the pages to navigate or represent a specific subset.
A page can only be extracted if its URL was collected during the navigation phase. Make sure the Addresses to include in the Navigation section cover all pages of interest.
In the Filter addresses to exclude field, enter the list of pages not to extract.
Removing elements from the page
Before converting a page into PDF, the crawler already performs a basic cleanup of the content. However, every site is structured differently and may contain elements that are not useful for documentation, such as menus, banners, footers or cookie boxes. The Remove elements field allows a more precise cleanup: the elements specified are removed from the page before the PDF is generated.
The field offers some preset selectors and also accepts manually entered selectors, in CSS, XPATH or Puppeteer-compatible format.
The most commonly used CSS selectors are:
| Type | Syntax | Example | Elements removed |
|---|---|---|---|
| Class | . + class name | .cookie-banner | All elements with class="cookie-banner" |
| ID | # + id | #newsletter | The element with id="newsletter" |
| HTML element | element name | header, footer, nav | All elements of that type |
To find the classes and ids to remove, open the page in the browser, right-click the element and select Inspect: the highlighted HTML code shows the element's class and id attributes.
If an element has several classes, specifying just one is enough. For example, for a menu with class="sidebar side-menu" the selector .side-menu is sufficient.
Removal applies to all pages extracted by the crawler. For this reason it is good practice to specify mainly the elements common to the whole site: if a side menu identified by the class .side-menu appears on every page, entering the selector once is enough to remove it everywhere. Elements present only on some pages must instead be identified and added individually.
Avoid overly generic selectors such as div: such a common element is present across almost the whole page, and removing it would also delete the content to be extracted.
The crawler already performs a preliminary cleanup of the page: manually entering the elements to remove is not mandatory, but it is recommended especially for older sites or sites with unusual layouts.
Scheduling
Select Enable scheduling and set the crawler execution frequency (weekly or monthly), then configure the day and start time; you can set multiple days of the week or month for repeated execution.
Viewing documents
To verify the PDF documents generated by the crawler, go back to the subfolder in Smart Document Access.
The list of indexed documents will be available in the section dedicated to the configured subfolder.
Each document has the URL from which it was generated associated as the Source file.
