Results for redis_dedupe

Ergonomic Rust SDK for the Net mesh network

Ergonomic Rust SDK for the Net mesh network

High-performance, schema-agnostic, backend-agnostic event bus

High-performance, schema-agnostic, backend-agnostic event bus

A highly parallel Perl 5 interpreter written in Rust

Multi-core Tokio-native orchestration for LLM inference pipelines: dedup, circuit breakers, backpressure, MCP, and autonomous self-improvement

Data processing and transformation pipeline

Generic classifieds scraping framework with browser automation, photo deduplication, and S3 storage

Step-by-step install helper. Tells you what to type, doesn't run it.

Content deduplication utilities for web crawling

Redis-backed distributed cache stampede prevention with leases, fencing tokens, SWR, and tagging

Common types and utilities for the Argus web crawler

This is a weak deduper to make things like bulk email run safer. It is not a lock safe for financial/security needs because it uses a weak redis locking pattern that can have race conditions. However, imagine a bulk email job that loops over 100 users, and enqueues a background email for each user. If the job fails at iteration 50, a retry would enqueue all the users again and many will receive dupes. This would continue multiple times as the parent job continued to rerun. By marking that a subjob has been enqueued, we can let that isolated job handle its own failures, and the batch enqueue job can run multiple times without re-enqueueing the same subjobs.