# Add your website to the Library

> Crawl your site so your AI agents answer from the pages your customers already read.

The quickest way to give an agent useful knowledge is your own website. help.ai crawls the pages you choose and prepares them so agents can answer from them.

## Add a website

1. Open **Library** and select **Create Library Item**.
2. Set **Type** to **Website**.
3. Optionally add a **Title**, **Tags** and a **Description** so your team can find the item later.
4. Under **Links**, enter the **URL** to start from, such as `https://example.com/help`.
5. Choose a **Crawl mode**. For **Crawl links**, also choose a **Crawl boundary** and, optionally, a **Max depth**.
6. Optionally set **Max pages**, **Include paths** and **Exclude paths**.
7. Select **Create Website**. The crawl starts right away.

To crawl another site or section in the same item, select **Add another link**. Each link has its own settings, and one Website item holds up to 500 links.

## Crawl modes

| Mode | What it does | Best for |
| --- | --- | --- |
| Crawl links | Follows matching links from the starting URL. This is the default. | Most sites |
| Sitemap | Reads the sitemap and queues the listed pages | Sites with an up-to-date sitemap |
| Individual link | Captures only this exact URL | One important page |

For **Sitemap**, your home page URL is enough: help.ai looks for the sitemap at the root of the site.

The **Crawl boundary** decides how far a **Crawl links** crawl can travel. **Same domain**, the default, can move across subdomains of the same root domain, such as from www.example.com to docs.example.com. **Same hostname** stays on the exact hostname, such as docs.example.com.

## Limits

| Setting | Range | If you leave it blank |
| --- | --- | --- |
| Max pages | 1 to 5,000 per link | The crawl stops at 500 pages for that link |
| Max depth | 0 to 100 link levels | No depth limit; only the page limit applies |

A **Max depth** of 0 captures only the starting page. The numbers shown in empty fields are examples, not defaults.

## Include and exclude paths

Enter one path prefix per line, up to 200 of each per link. Include `/help/` and `/docs/` to keep a crawl focused, or exclude `/blog/` and `/careers/` to leave out pages that would muddle answers.

The crawler follows your site's robots.txt, so pages it disallows are skipped.

## Status

The **Content Extraction** column shows where each item stands:

| Status | Meaning |
| --- | --- |
| Pending | Waiting to extract |
| Processing | Being crawled and prepared |
| Ready | Agents can use it |
| Failed | Something went wrong; check the URL and try again |

A crawl that's still running after 2 hours is marked **Failed**. Failed items also show on the **Dashboard** under **Needs attention** → **Failed Library items**. See [Fix and refresh Library items](https://support.help.ai/library-and-content/fix-and-refresh-library-items).

## Keep it current

When your site changes, open the item's row menu and choose **Extract content**. Pages fetched in the last 24 hours are reused; older pages are crawled again. **Extraction summary** shows what was captured, and **View content** shows the text agents read.

If you change the item's links or settings with **Edit**, saving crawls the site again on its own. A new title, description or tags doesn't.

## Connect it to an agent

An agent with nothing picked under **Knowledge** → **Library** uses every ready item, so a new website is picked up on its own. If the agent uses a selection, add the new item with **Pick items**. See [Choose what each AI agent can read](https://support.help.ai/library-and-content/choose-what-each-agent-knows).

<Tip>
A crawl can't reach pages behind a login. Add that material as a **File** or **Text Snippet** instead.
</Tip>