This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.
The fastest directory crawler & globbing alternative to glob, fast-glob, & tiny-glob. Crawls 1m files in < 1s
Device detection module for Nuxt
A triple-linked lists based DOM implementation
A list of common crawler agents used on Internet..
express middleware for serving prerendered javascript-rendered pages for SEO
A library to recursively retrieve and serialize Notion pages with customization for machine learning applications.
This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
Maps proxy protocols to `http.Agent` implementations
Analyzes license information for multiple node.js modules (package.json files) as part of your software project.
The OpenAI Agents SDK is a lightweight yet powerful framework for building multi-agent workflows.
Used to run a web crawler that checks for errors on specified pages.
Inspecting Node.js's Network with Chrome DevTools
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
Client side APIs for working with Salesforce agents
Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.
Distributed web crawler powered by Headless Chrome
The OpenAI Agents SDK is a lightweight yet powerful framework for building multi-agent workflows. This package contains the logic for building realtime voice agents on the server or in the browser.
Turn a function into an `http.Agent` instance
LangGraph
Promptbook: Create persistent AI agents that turn your company's scattered knowledge into action
A light weight JS library to check if a user agent is a web crawler.
The OpenAI Agents SDK is a lightweight yet powerful framework for building multi-agent workflows.
HTTP request module customized for crawlers.
A simple library for detecting bots/crawlers/spiders through User-Agent strings
CrawlerDetect is a library to detect bots/crawlers via the user agent
Checks a user agent for a web crawler
Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.