# Track AI crawlers and bots
URL: https://pieowl.com/docs/ai-bots
The **AI & bots** card on the overview shows which crawlers and AI assistants requested your pages, per company, over time. Bot requests are kept apart from your visitors: they never show up as sessions, page views or visitors.
The three categories [#the-three-categories]
Every recognised bot belongs to one category:
* **AI answers**: fetched live because someone asked an AI assistant something, for example ChatGPT, Claude or Perplexity looking up your site to answer a question.
* **Indexing**: search engines and AI search building their index, for example Googlebot, Bingbot, OAI-SearchBot or PerplexityBot.
* **Training**: collected to train AI models, for example GPTBot, ClaudeBot, CCBot (Common Crawl) or Bytespider.
Why you need a server-side snippet [#why-you-need-a-server-side-snippet]
Bots don't run JavaScript, so the Pieowl tracking script never sees them. Your server sees every request, so it reports the bot requests to Pieowl. The snippets below check the user agent first and only send requests that look like a bot, so normal visitors cause no extra network calls.
All options need your client secret. Keep it on the server and never put it in browser code. See [Authentication](/docs/api/authentication).
@pieowl/bots [#pieowlbots]
One package for every server that runs JavaScript: Node.js, edge runtimes and Cloudflare Workers. It uses only `fetch` and depends on no framework.
```bash
npm install @pieowl/bots
```
It reports only requests whose user agent looks like a bot, never blocks or changes your response, and never throws. Reports are sent in the background, in batches of up to 50 requests or every second. Each report has the path, host and method, the response status where the framework gives it, and the client IP for [verification](#verification): the first address in `X-Forwarded-For`, else `X-Real-IP`, else `CF-Connecting-IP`. Pass `apiUrl` to send to a different API URL (default `https://api.pieowl.com`).
Next.js [#nextjs]
```typescript title="proxy.ts"
import { trackBots } from '@pieowl/bots/next';
export const proxy = trackBots({
clientId: process.env.PIEOWL_CLIENT_ID!,
clientSecret: process.env.PIEOWL_CLIENT_SECRET!,
});
```
On Next.js 15 or older, name the file `middleware.ts` and use `export const middleware = trackBots({ ... })`. The request continues as if there were no proxy, and the report is kept alive with `event.waitUntil`.
Already have a proxy? Call `trackBotRequest` in it and keep your own response:
```typescript title="proxy.ts"
import { trackBotRequest } from '@pieowl/bots';
import { type NextFetchEvent, type NextRequest, NextResponse } from 'next/server';
export function proxy(request: NextRequest, event: NextFetchEvent) {
trackBotRequest(request, {
clientId: process.env.PIEOWL_CLIENT_ID!,
clientSecret: process.env.PIEOWL_CLIENT_SECRET!,
}, event);
return NextResponse.next();
}
```
If your proxy or middleware has a `matcher`, make sure it covers the pages bots request. Requests the proxy doesn't run for are not reported.
Express [#express]
```typescript
import { trackBots } from '@pieowl/bots/express';
app.use(trackBots({
clientId: process.env.PIEOWL_CLIENT_ID,
clientSecret: process.env.PIEOWL_CLIENT_SECRET,
}));
```
The request is reported after the response is sent, with its status. The IP is `req.ip`: behind a proxy or load balancer, set Express's [`trust proxy`](https://expressjs.com/en/guide/behind-proxies.html) so `req.ip` is the client and not the proxy. It also works with Connect and other `(req, res, next)` servers.
Hono [#hono]
```typescript
import { trackBots } from '@pieowl/bots/hono';
app.use(trackBots({
clientId: process.env.PIEOWL_CLIENT_ID!,
clientSecret: process.env.PIEOWL_CLIENT_SECRET!,
}));
```
On Cloudflare Workers, read the keys with `import { env } from 'cloudflare:workers'`. The report is kept alive with `executionCtx.waitUntil`.
Cloudflare Workers and Pages [#cloudflare-workers-and-pages]
Wrap the Worker's fetch handler:
```typescript title="src/index.ts"
import { env } from 'cloudflare:workers';
import { withBotTracking } from '@pieowl/bots';
export default {
fetch: withBotTracking((request) => fetch(request), {
clientId: env.PIEOWL_CLIENT_ID,
clientSecret: env.PIEOWL_CLIENT_SECRET,
}),
};
```
`withBotTracking` wraps any `(request, ...) => Response` handler (Deno, Bun, Vercel Edge Functions) and uses the first argument that has `waitUntil`.
For Cloudflare Pages, add a middleware:
```typescript title="functions/_middleware.ts"
import { trackBotRequest } from '@pieowl/bots';
export const onRequest: PagesFunction = (context) => {
trackBotRequest(context.request, {
clientId: context.env.PIEOWL_CLIENT_ID,
clientSecret: context.env.PIEOWL_CLIENT_SECRET,
}, context);
return context.next();
};
```
Astro, Nuxt, SvelteKit and others [#astro-nuxt-sveltekit-and-others]
Call `trackBotRequest(request, options, context?)` in your framework's server middleware. It takes a Fetch `Request`, or `{ url, headers, method?, ip?, status? }`. On serverless and edge platforms, pass anything with `waitUntil` as `context`.
```typescript title="Astro: src/middleware.ts"
import { trackBotRequest } from '@pieowl/bots';
import { defineMiddleware } from 'astro:middleware';
export const onRequest = defineMiddleware((context, next) => {
trackBotRequest(context.request, {
clientId: import.meta.env.PIEOWL_CLIENT_ID,
clientSecret: import.meta.env.PIEOWL_CLIENT_SECRET,
});
return next();
});
```
```typescript title="Nuxt: server/middleware/pieowl-bots.ts"
import { trackBotRequest } from '@pieowl/bots';
export default defineEventHandler((event) => {
trackBotRequest({ url: getRequestURL(event).href, headers: getRequestHeaders(event) }, {
clientId: process.env.PIEOWL_CLIENT_ID!,
clientSecret: process.env.PIEOWL_CLIENT_SECRET!,
});
});
```
```typescript title="SvelteKit: src/hooks.server.ts"
import { trackBotRequest } from '@pieowl/bots';
import { env } from '$env/dynamic/private';
export const handle = async ({ event, resolve }) => {
trackBotRequest(event.request, {
clientId: env.PIEOWL_CLIENT_ID,
clientSecret: env.PIEOWL_CLIENT_SECRET,
});
return resolve(event);
};
```
In Remix and React Router, call it at the top of your root `loader` with `request`.
HTTP API [#http-api]
From any other server, send bot requests to the API yourself:
```bash
curl -X POST 'https://api.pieowl.com/bot' \
-H 'Content-Type: application/json' \
-H 'pieowl-client-id: YOUR_CLIENT_ID' \
-H 'pieowl-client-secret: YOUR_CLIENT_SECRET' \
-d '{
"requests": [
{
"userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)",
"path": "/pricing",
"host": "example.com",
"method": "GET",
"status": 200,
"timestamp": "2026-01-15T10:30:00.000Z",
"ip": "20.171.206.17"
}
]
}'
```
The body has a `requests` array with 1 to 100 items. Each item:
| Field | Type | Required | Description |
| ----------- | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `userAgent` | string | Yes | The `User-Agent` header of the request, 1 to 1024 characters. |
| `path` | string | Yes | Requested path, 1 to 2048 characters. |
| `host` | string | No | Requested host, up to 255 characters. |
| `method` | string | No | HTTP method, up to 16 characters. |
| `status` | integer | No | Response status, 100 to 599. |
| `timestamp` | string | No | ISO 8601 date-time with an offset, for example `2026-01-15T10:30:00.000Z`. Defaults to the time Pieowl receives it. Times more than 1 hour in the past are moved to 1 hour ago, and future times to now. |
| `ip` | string | No | IPv4 or IPv6 address the request came from, without a port. Used to [verify](#verification) the bot, not stored. |
A valid request returns `202` with the number of stored and ignored items:
```json
{ "accepted": 1, "ignored": 0 }
```
Requests without a valid client secret get `401`. The endpoint allows 600 calls per 10 seconds per client ID, so batch bot requests (up to 100 per call) instead of sending one call each.
Verification [#verification]
Anyone can put `GPTBot` in a user agent. When you send the client IP, Pieowl checks it against what the bot's company publishes: its official IP list, or a reverse DNS lookup confirmed by a forward lookup. Each request gets one of three results:
* **Verified**: the IP belongs to the company. Shown with a check mark in the card.
* **Fake**: the company publishes a way to check its bots, and the IP fails it. Fake requests are left out of the card's charts and totals. A line under the tabs says how many were hidden and which bots they claimed to be.
* **Can't be verified**: the company publishes no IP list or DNS name for its bot, the request was sent without an IP, or the check could not run (for example a DNS timeout). These requests are counted as normal.
Meta, ByteDance (Bytespider), DeepSeek and xAI publish no addresses for their bots, so their requests are always shown as "can't be verified". Companies Pieowl verifies include OpenAI, Anthropic, Google, Microsoft (Bing), Apple, Amazon, Perplexity, Mistral, DuckDuckGo, Common Crawl and Yandex.
The IP is only used for this check. Pieowl stores the result, not the IP.
The IP must be the bot's, not your proxy's. `@pieowl/bots` takes it from the usual proxy headers (Express: `req.ip`); if you call the HTTP API yourself, forward the client IP your load balancer or CDN reports.
Notes [#notes]
* **Bot requests don't count toward your event limit.**
* **Requests are classified by user agent.** Pieowl matches the user agent against a list of known AI and search bots and assigns each one a company and a category. The IP then [confirms](#verification) the claim where the company makes that possible.
* **Unknown bots are ignored.** Requests from user agents that aren't on the list are not stored. In the HTTP API response they count as `ignored`.