Invantive Cloud Documentation offers the files that crawlers and agents read before any page, at fixed addresses. ## `robots.txt` The file `/robots.txt` allows search engines, AI crawlers and link previews, and refuses known scrapers and download tools. For every other crawler it allows the site except: - `/*/search`: the search result pages, which never end and are no content, - `/api/`, - `/*.md : the Markdown versions of the pages, which are meant for agents that read `llms.txt`, not for the index of a search engine. It carries the line `Content-Signal: search=yes, ai-input=yes, ai-train=yes` and names the sitemap. ## Sitemaps The file `/sitemap.xml` is the index of the sitemaps, one file per language and section of at most 10,000 addresses, compressed with gzip. Each address lists the other languages in which the page exists. The sitemaps hold the documentation and data model pages whose translation is complete, the platform pages and the poster PDFs; they leave out the search pages, the forum reading views and the Markdown versions. The date of the last change of an address is the date its content or translation last changed. ## Text and Data Mining Text and data mining of the site is allowed, also to train models. Every answer of the site carries the header `TDM-Reservation: 0`, and `/.well-known/tdmrep.json` says the same after the [TDM Reservation Protocol](https://www.w3.org/community/tdmrep/). ## `security.txt` The file `/.well-known/security.txt` after [RFC 9116](https://www.rfc-editor.org/rfc/rfc9116) tells how to report a vulnerability: the telephone number and e-mail address of Invantive security, the preferred languages, the disclosure policy and the acknowledgements. Its expiry date always lies about six months ahead. The address `/security.txt` redirects to it. ## Other Host Names A host name of the site other than `documentation.invantive.eu`, such as a test environment, answers `robots.txt` with `Disallow: /` and marks every page `noindex`, so only the production site is indexed.