Curated, sourced list of AI crawler / training bot user agents, plus a small CLI to test whether a URL is reachable to each bot.
Crawler(robots) decision middleware for Express
AI agent permissions for Vercel — block training crawlers, allow search and AI assistants, enforced at the edge.
常见爬虫拦截koa中间件
AI-friendly SEO generator for modern web frameworks
Zero-code bot tracking for Node.js servers (Express, Fastify, NestJS, Next.js, pure Node)
AI bot detection middleware for Next.js — works on any host
Plugin for generating a `robots.txt` route for your Enhance application.
常见爬虫拦截egg插件
AI bot detection middleware for Cloudflare Workers & Pages — detect 26+ AI crawlers at the edge
Web Builder Module - Create robots.txt file
🛡️ Nuxt 3 middleware to block suspicious bots, protect SEO crawlers with reverse DNS checks, and enforce User-Agent rules.
Verify that a request is from Apple crawlers using DNS verification steps
Zero-code bot tracking for Node.js servers (Express, Fastify, NestJS, Next.js, pure Node)
Zero-code bot tracking for Node.js servers (Express, Fastify, NestJS, Next.js, pure Node)
koa detect googlebot, facebookbot ...