Add your website to the Library

The quickest way to give an agent useful knowledge is your own website. help.ai crawls the pages you choose and prepares them so agents can answer from them.

Add a website#

1. Open Library and select Create Library Item.
2. Set Type to Website.
3. Optionally add a Title, Tags and a Description so your team can find the item later.
4. Under Links, enter the URL to start from, such as https://example.com/help.
5. Choose a Crawl mode. For Crawl links, also choose a Crawl boundary and, optionally, a Max depth.
6. Optionally set Max pages, Include paths and Exclude paths.
7. Select Create Website. The crawl starts right away.

To crawl another site or section in the same item, select Add another link. Each link has its own settings, and one Website item holds up to 500 links.

Crawl modes#

For Sitemap, your home page URL is enough: help.ai looks for the sitemap at the root of the site.

The Crawl boundary decides how far a Crawl links crawl can travel. Same domain, the default, can move across subdomains of the same root domain, such as from www.example.com to docs.example.com. Same hostname stays on the exact hostname, such as docs.example.com.

Limits#

A Max depth of 0 captures only the starting page. The numbers shown in empty fields are examples, not defaults.

Include and exclude paths#

Enter one path prefix per line, up to 200 of each per link. Include /help/ and /docs/ to keep a crawl focused, or exclude /blog/ and /careers/ to leave out pages that would muddle answers.

The crawler follows your site's robots.txt, so pages it disallows are skipped.

Status#

The Content Extraction column shows where each item stands:

A crawl that's still running after 2 hours is marked Failed. Failed items also show on the Dashboard under Needs attention → Failed Library items. See Fix and refresh Library items.

Keep it current#

When your site changes, open the item's row menu and choose Extract content. Pages fetched in the last 24 hours are reused; older pages are crawled again. Extraction summary shows what was captured, and View content shows the text agents read.

If you change the item's links or settings with Edit, saving crawls the site again on its own. A new title, description or tags doesn't.

Connect it to an agent#

An agent with nothing picked under Knowledge → Library uses every ready item, so a new website is picked up on its own. If the agent uses a selection, add the new item with Pick items. See Choose what each AI agent can read.