Skip to content

General Scraper Engine — V1 Tracker

Project Goal

Build a general-purpose public-web lead scraper engine that accepts a keyword, location, and optional requirements, discovers relevant public web pages, extracts lead data, normalizes and deduplicates the results, and outputs structured leads.

Scope

  • General-purpose web lead discovery
  • Public web pages only
  • Keyword + location + optional requirements
  • Discovery → Fetch → Parse → Normalize → Deduplicate → Output
  • No login, CAPTCHA, private-page, or access-control bypass
  • No UI, dashboard, or authentication in V1; optional DuckDB persistence is supported
  • No PyLage integration in V1

Roadmap

Phase 0 — Foundation

  • Create project structure
  • Create Python virtual environment
  • Install core dependencies

Phase 1 — Data Model

  • Create Lead model
  • Define structured lead fields

Phase 2 — Discovery Abstraction

  • Create SearchRequest
  • Create DiscoveredPage
  • Create DiscoveryProvider protocol
  • Create WebDiscovery
  • Add query generation
  • Add URL deduplication
  • Add discovery tests

Phase 3 — Public Web Fetcher

  • HTTP/HTTPS fetching
  • Redirect handling
  • Timeout handling
  • HTTP error handling
  • HTML content validation
  • robots.txt checking
  • Custom user-agent
  • Add fetcher tests

Phase 4 — Parser

  • JSON-LD / Schema.org extraction
  • Meta description extraction
  • OpenGraph/title fallback
  • Phone extraction
  • Email extraction
  • Address extraction
  • Category extraction
  • Add parser tests

Phase 5 — Normalization & Deduplication

  • Normalize text
  • Normalize names
  • Normalize phone numbers
  • Normalize email addresses
  • Normalize URLs
  • Generate lead identity keys
  • Deduplicate leads
  • Merge duplicate records
  • Preserve source URLs
  • Complete dedicated normalizer pytest coverage

Phase 6 — Engine Integration

  • Connect discovery to fetcher
  • Connect fetcher to parser
  • Connect parser to normalizer
  • Create ScrapeResult
  • Track fetch failures
  • Track parse failures
  • Add engine tests

Phase 7 — Real Web Discovery

  • Implement real public-web discovery provider
  • Keep discovery provider-based and replaceable
  • Support multiple discovery providers
  • Avoid brittle direct search-engine HTML scraping where possible
  • Respect public-web access rules

Phase 8 — Relevance Filtering

  • Keyword relevance scoring
  • Location relevance scoring
  • Category relevance filtering
  • Requirements matching
  • Remove irrelevant candidates

Phase 9 — Result Quality

  • Improve phone extraction
  • Improve address extraction
  • Improve location extraction
  • Improve business/category extraction
  • Distinguish business website from source URL
  • Add lead quality checks

Phase 10 — Rate Limiting & Crawl Policy

  • Implement crawl-delay handling
  • Add request throttling
  • Add per-domain limits
  • Add retry policy
  • Improve robots.txt failure handling

Phase 11 — CLI

  • Add keyword argument
  • Add location argument
  • Add optional requirements argument
  • Add result limit argument
  • Validate CLI input

Phase 12 — Terminal Output

  • Print result count
  • Print structured lead records
  • Print source information
  • Print fetch/parse summary
  • Keep terminal output readable
  • Optional JSON/CSV output

Phase 13 — Live Web Smoke Test

  • Run real doctor + Shahjahanpur search
  • Verify discovery
  • Verify fetching
  • Verify parsing
  • Verify normalization
  • Verify deduplication
  • Verify terminal output

Phase 14 — Robustness

  • Handle 404 responses
  • Handle 403 responses
  • Handle 429 responses
  • Handle 500 responses
  • Handle timeouts
  • Handle redirects
  • Handle empty HTML
  • Handle malformed HTML
  • Handle invalid JSON-LD
  • Handle non-HTML pages
  • Handle duplicate URLs
  • Handle duplicate leads
  • Handle dead domains
  • Handle robots-blocked pages
  • Ensure engine does not crash on individual failures

Phase 15 — Documentation

  • Write README
  • Document installation
  • Document quick start
  • Document CLI usage
  • Document architecture
  • Document discovery providers
  • Document Lead schema
  • Document public-web rules
  • Document robots policy
  • Add examples
  • Document limitations

Phase 16 — Packaging

  • Create/update pyproject.toml
  • Add package metadata
  • Add CLI entry point if appropriate
  • Verify package installation

Phase 17 — Final Test & Release Audit

  • Run full pytest suite
  • Run Python compile checks
  • Run CLI smoke test
  • Run live-web smoke test
  • Run git diff --check
  • Review documentation
  • Review scope boundaries
  • Commit final V1
  • Push final V1

Current Status

Foundation [x] Data Model [x] Discovery abstraction [x] Discovery tests [x] Fetcher [x] Fetcher tests [x] Parser [x] Parser tests [x] Engine [x] Engine tests [x] Pytest configuration [x] Normalizer tests [x] Real discovery [x] Relevance [x] CLI [x] Live smoke test [x] Documentation [x] Packaging [x] Final release [x]