Track AI crawlers and bots

The AI & bots card on the overview shows which crawlers and AI assistants requested your pages, per company, over time. Bot requests are kept apart from your visitors: they never show up as sessions, page views or visitors.

The three categories

Every recognised bot belongs to one category:

• AI answers: fetched live because someone asked an AI assistant something, for example ChatGPT, Claude or Perplexity looking up your site to answer a question.
• Indexing: search engines and AI search building their index, for example Googlebot, Bingbot, OAI-SearchBot or PerplexityBot.
• Training: collected to train AI models, for example GPTBot, ClaudeBot, CCBot (Common Crawl) or Bytespider.

Why you need a server-side snippet

Bots don't run JavaScript, so the Pieowl tracking script never sees them. Your server sees every request, so it reports the bot requests to Pieowl. The snippets below check the user agent first and only send requests that look like a bot, so normal visitors cause no extra network calls.

All options need your client secret. Keep it on the server and never put it in browser code. See Authentication.

@pieowl/bots

One package for every server that runs JavaScript: Node.js, edge runtimes and Cloudflare Workers. It uses only fetch and depends on no framework.

npm install @pieowl/bots

It reports only requests whose user agent looks like a bot, never blocks or changes your response, and never throws. Reports are sent in the background, in batches of up to 50 requests or every second. Each report has the path, host and method, the response status where the framework gives it, and the client IP for verification: the first address in X-Forwarded-For, else X-Real-IP, else CF-Connecting-IP. Pass apiUrl to send to a different API URL (default https://api.pieowl.com).

Next.js

import { trackBots } from '@pieowl/bots/next';

export const proxy = trackBots({
  clientId: process.env.PIEOWL_CLIENT_ID!,
  clientSecret: process.env.PIEOWL_CLIENT_SECRET!,
});

On Next.js 15 or older, name the file middleware.ts and use export const middleware = trackBots({ ... }). The request continues as if there were no proxy, and the report is kept alive with event.waitUntil.

Already have a proxy? Call trackBotRequest in it and keep your own response:

import { trackBotRequest } from '@pieowl/bots';
import { type NextFetchEvent, type NextRequest, NextResponse } from 'next/server';

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackBotRequest(request, {
    clientId: process.env.PIEOWL_CLIENT_ID!,
    clientSecret: process.env.PIEOWL_CLIENT_SECRET!,
  }, event);
  return NextResponse.next();
}

Express

import { trackBots } from '@pieowl/bots/express';

app.use(trackBots({
  clientId: process.env.PIEOWL_CLIENT_ID,
  clientSecret: process.env.PIEOWL_CLIENT_SECRET,
}));

The request is reported after the response is sent, with its status. The IP is req.ip: behind a proxy or load balancer, set Express's trust proxy so req.ip is the client and not the proxy. It also works with Connect and other (req, res, next) servers.

Hono

import { trackBots } from '@pieowl/bots/hono';

app.use(trackBots({
  clientId: process.env.PIEOWL_CLIENT_ID!,
  clientSecret: process.env.PIEOWL_CLIENT_SECRET!,
}));

On Cloudflare Workers, read the keys with import { env } from 'cloudflare:workers'. The report is kept alive with executionCtx.waitUntil.

Cloudflare Workers and Pages

Wrap the Worker's fetch handler:

import { env } from 'cloudflare:workers';
import { withBotTracking } from '@pieowl/bots';

export default {
  fetch: withBotTracking((request) => fetch(request), {
    clientId: env.PIEOWL_CLIENT_ID,
    clientSecret: env.PIEOWL_CLIENT_SECRET,
  }),
};

withBotTracking wraps any (request, ...) => Response handler (Deno, Bun, Vercel Edge Functions) and uses the first argument that has waitUntil.

For Cloudflare Pages, add a middleware:

import { trackBotRequest } from '@pieowl/bots';

export const onRequest: PagesFunction<Env> = (context) => {
  trackBotRequest(context.request, {
    clientId: context.env.PIEOWL_CLIENT_ID,
    clientSecret: context.env.PIEOWL_CLIENT_SECRET,
  }, context);
  return context.next();
};

Astro, Nuxt, SvelteKit and others

Call trackBotRequest(request, options, context?) in your framework's server middleware. It takes a Fetch Request, or { url, headers, method?, ip?, status? }. On serverless and edge platforms, pass anything with waitUntil as context.

import { trackBotRequest } from '@pieowl/bots';
import { defineMiddleware } from 'astro:middleware';

export const onRequest = defineMiddleware((context, next) => {
  trackBotRequest(context.request, {
    clientId: import.meta.env.PIEOWL_CLIENT_ID,
    clientSecret: import.meta.env.PIEOWL_CLIENT_SECRET,
  });
  return next();
});

import { trackBotRequest } from '@pieowl/bots';

export default defineEventHandler((event) => {
  trackBotRequest({ url: getRequestURL(event).href, headers: getRequestHeaders(event) }, {
    clientId: process.env.PIEOWL_CLIENT_ID!,
    clientSecret: process.env.PIEOWL_CLIENT_SECRET!,
  });
});

import { trackBotRequest } from '@pieowl/bots';
import { env } from '$env/dynamic/private';

export const handle = async ({ event, resolve }) => {
  trackBotRequest(event.request, {
    clientId: env.PIEOWL_CLIENT_ID,
    clientSecret: env.PIEOWL_CLIENT_SECRET,
  });
  return resolve(event);
};

In Remix and React Router, call it at the top of your root loader with request.

HTTP API

From any other server, send bot requests to the API yourself:

curl -X POST 'https://api.pieowl.com/bot' \
  -H 'Content-Type: application/json' \
  -H 'pieowl-client-id: YOUR_CLIENT_ID' \
  -H 'pieowl-client-secret: YOUR_CLIENT_SECRET' \
  -d '{
    "requests": [
      {
        "userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)",
        "path": "/pricing",
        "host": "example.com",
        "method": "GET",
        "status": 200,
        "timestamp": "2026-01-15T10:30:00.000Z",
        "ip": "20.171.206.17"
      }
    ]
  }'

The body has a requests array with 1 to 100 items. Each item:

A valid request returns 202 with the number of stored and ignored items:

{ "accepted": 1, "ignored": 0 }

Requests without a valid client secret get 401. The endpoint allows 600 calls per 10 seconds per client ID, so batch bot requests (up to 100 per call) instead of sending one call each.

Verification

Anyone can put GPTBot in a user agent. When you send the client IP, Pieowl checks it against what the bot's company publishes: its official IP list, or a reverse DNS lookup confirmed by a forward lookup. Each request gets one of three results:

• Verified: the IP belongs to the company. Shown with a check mark in the card.
• Fake: the company publishes a way to check its bots, and the IP fails it. Fake requests are left out of the card's charts and totals. A line under the tabs says how many were hidden and which bots they claimed to be.
• Can't be verified: the company publishes no IP list or DNS name for its bot, the request was sent without an IP, or the check could not run (for example a DNS timeout). These requests are counted as normal.

Meta, ByteDance (Bytespider), DeepSeek and xAI publish no addresses for their bots, so their requests are always shown as "can't be verified". Companies Pieowl verifies include OpenAI, Anthropic, Google, Microsoft (Bing), Apple, Amazon, Perplexity, Mistral, DuckDuckGo, Common Crawl and Yandex.

The IP is only used for this check. Pieowl stores the result, not the IP.

Notes

• Bot requests don't count toward your event limit.
• Requests are classified by user agent. Pieowl matches the user agent against a list of known AI and search bots and assigns each one a company and a category. The IP then confirms the claim where the company makes that possible.
• Unknown bots are ignored. Requests from user agents that aren't on the list are not stored. In the HTTP API response they count as ignored.