Skip to content

Home

# General Scraper Engine
A modular, policy-aware Python engine for discovering, acquiring, extracting, validating, and storing structured leads from the public web.
Get Started View on GitHub
### Python Package Install directly with pip and integrate the engine into your Python workflows.
### Current Release Version **1.0.3** with release notes available in the documentation.
### Public Web Built around public-web acquisition and explicit access boundaries.
01
### Discovery Find relevant pages using pluggable discovery providers such as Brave Search and SearXNG.
02
### HTTP-first acquisition Use inexpensive HTTP fetching by default, with browser and external-provider fallback when required.
03
### Structured extraction Extract leads from JSON-LD, Schema.org, metadata, contact information, and visible page content.
04
### Policy-aware Respect public-web boundaries, robots.txt rules, crawl delays, request limits, and access restrictions.
05
### Quality and deduplication Validate, normalize, merge, and deduplicate extracted leads before output.
06
### Multiple outputs Write results to terminal, JSON, CSV, or DuckDB with optional run-level metrics.

Pipeline

The engine separates discovery, acquisition, extraction, validation, and persistence into independent stages.

Discovery
    |
    v
Relevance
    |
    v
Policy
    |
    v
Acquisition
    |
    v
Content Quality
    |
    v
Parse / Extract
    |
    v
Location / Lead Quality
    |
    v
Normalize / Deduplicate
    |
    v
Persistence / Output

HTTP remains the cheapest default acquisition method. Browser and external-provider strategies are introduced only when content or workload requires them.

Quick Start

Install the package:

pip install general-scraper-engine

Then run a basic scrape:

general-scraper-engine \
  --keyword "dentist" \
  --location "New York" \
  --output leads.json

See the complete setup in the Quick Start guide.

What it supports

Area Capabilities
Discovery Brave Search, SearXNG, provider protocol, caching
Acquisition HTTP, Playwright browser, external providers, Google Maps, Justdial
Crawling Same-site crawling, depth/page limits, domain controls
Extraction JSON-LD, Schema.org, metadata, contacts, addresses
Relevance Keyword, location, category, requirements
Quality Content quality, lead quality, location validation
Data processing Normalization, deduplication, merging
Persistence Terminal, JSON, CSV, DuckDB
Reliability Retries, throttling, bounded concurrency, caching

Documentation

### Getting Started Install the engine and run your first scrape. [Installation](getting-started/installation.md) [Quick Start](getting-started/quickstart.md)
### Guides Learn CLI usage, output formats, providers, crawling examples, and public-web policy. [Explore Guides](guides/cli.md)
### Reference Understand the architecture, lead schema, programmatic API, and error model. [Open Reference](reference/architecture.md)

Public-Web Boundary

General Scraper Engine is designed for public-web acquisition.

It does not provide:

  • Login bypass
  • CAPTCHA bypass
  • Private-page access
  • Authentication bypass
  • Access-control circumvention

Robots.txt and crawl-policy controls are part of the acquisition boundary.

See Public-Web and Robots Policy for details.

Project

**General Scraper Engine** Modular - Policy-aware - HTTP-first - Extensible