Near-duplicate page clustering API

Cluster near-duplicate pages with Jaccard similarity on token shingles and connected-component clustering.

The Near-duplicate page clustering operation is part of the Content Keyword Analytics TF-IDF Readability API on BRAINIAC.software. Cluster near-duplicate pages with Jaccard similarity on token shingles and connected-component clustering. It is exposed as POST /api/v1/seo-content/near-duplicate-detect and priced at $0.03 USDC on Base. Payment uses the x402 exact scheme: an unpaid call returns HTTP 402 with payment requirements; after paying the exact USDC amount, retry the same request with a payment-signature header and Idempotency-Key. No account and no API key are required. Data sources for this product: Caller-supplied page texts, keywords and rank snapshots processed locally stopwords-iso English list (MIT) and Snowball English stemming (ISC / BSD-3-Clause) Liberation Sans Regular 2.1.5 advance widths (OFL-1.1) for snippet pixel estimates Limitations: Pure computation on caller input: no SERP scrape, keyword database or LLM is called at request time. English stopwords and Porter2 stemming only in v1; other languages return unsupported_language. Outputs are lookups or prechecks for human review. They are not legal, customs, tax, medical, or identity determinations. Outputs are compliance prechecks, drafts, or data lookups for human review. They are not official determinations, legal advice, identity confirmation, or certified filings. Related operations in this group include TF-IDF term gap vs competitor pages, Content brief from SERP page texts, English readability metrics, Internal link suggestions. Open the group documentation for the full request schema, field notes, and response examples before integrating.

POST$0.03 USDC
/api/v1/seo-content/near-duplicate-detect

Example request

curl -i -X POST "https://www.brainiac.software/api/v1/seo-content/near-duplicate-detect" \
  -H 'content-type: application/json' \
  -H 'Idempotency-Key: <fresh-64-character-random-hex-key>' \
  --data-binary @- <<'JSON'
{
  "pages": [
    {
      "id": "p1",
      "text": "the quick brown fox jumps over the lazy dog today"
    },
    {
      "id": "p2",
      "text": "the quick brown fox jumps over the lazy dog tonight"
    },
    {
      "id": "p3",
      "text": "the quick brown fox leaps over the lazy dog today"
    },
    {
      "id": "p4",
      "text": "lorem ipsum dolor sit amet consectetur adipiscing elit"
    }
  ],
  "shingleSize": 3,
  "threshold": 0.4
}
JSON

An unpaid probe returns HTTP 402 with x402 payment requirements. After paying the exact USDC amount on Base, retry the identical request with a payment-signature header.

FAQ

  • Do I need an API key? No. Pay the listed USDC amount with x402.
  • What network? Base (eip155:8453), USDC.
  • Is this an official determination? No. Results are for human review against the cited sources.

Related APIs

Content Keyword Analytics TF-IDF Readability APIFull docsHow it works