agentutility(7)$ man scrape-website
/scrape-website
PRICE / CALL
$0.04
USDC · base mainnet · scheme: exact
METHOD
POST
CLUSTER
webprobeCATEGORY
utilities
STATUS
● live
NAME
scrape-website — fetch a public webpage into clean markdown when an agent needs page content and metadata for a recurring watch or extraction job
SYNOPSIS
POST https://x402.agentutility.ai/scrape-website
Content-Type: application/json
X-PAYMENT: <signed-transferWithAuthorization>
{ ... }↳ first call →
402 Payment Required. Sign USDCtransferWithAuthorization, retry with theX-PAYMENT header.DESCRIPTION
Fetch a public webpage into clean markdown when an agent needs page content and metadata for a recurring watch or extraction job. Send one URL and select text, HTML, or markdown; receive the final URL, status, title, meta description, canonical URL, OpenGraph and Twitter cards, headings, body content, and optional links. It follows redirects and is suited to static pages and SSR sites; use website-screenshot for JS-heavy pages. Use it as a web scraping API, webpage-to-markdown converter, website metadata extractor, or URL content fetcher.
INPUT — request schema
| property | type | description | req? |
|---|---|---|---|
| url | string | Public URL to fetch and parse. Must include scheme (http/https). Follows redirects. | required |
| format | string | Body output format. 'text' (default, plain text), 'html' (raw HTML body), or 'markdown' (clean markdown — best for LLM ingestion). enum: text · html · markdown | optional |
| include_links | boolean | If true, also returns an array of all <a href> links found on the page. Default false. | optional |
| user_agent | string | Custom User-Agent header. Defaults to a modern desktop Chrome UA. | optional |
OUTPUT — response shape
| field | type | description |
|---|---|---|
| url | string | Final URL fetched after following redirects from the requested page. |
| final_url | string | Final URL after redirects. |
| status_code | number | HTTP status code returned by the target server for the scrape request. |
| title | string | Page title pulled from the <title> tag or og:title fallback. |
| description | string | Meta description from the page's description, og:description, or twitter:description tag. |
| canonical | string | Canonical URL declared by the page's <link rel="canonical"> tag. |
| lang | string | Language code from the <html lang="..."> attribute (e.g. en, es). |
| h1 | string | Text content of the page's first <h1> heading. |
| og | object | All og:* meta tags. |
| object | All twitter:* meta tags. | |
| text | string | Body when format=text. |
| html | string | Body when format=html. |
| markdown | string | Body when format=markdown. |
| format | string | Body format returned: text, html, or markdown depending on the requested format param. |
| links | array | Array of {href, text} when include_links=true. |
| body_chars | number | Character count of the extracted body content in the chosen format. |
EXAMPLES — two ways to call
EXAMPLE 1 · curl
curl -X POST https://x402.agentutility.ai/scrape-website \
-H 'Content-Type: application/json' \
-d '{ }'first response =
402 Payment Required with payment requirements; sign + retry with X-PAYMENT.EXAMPLE 2 · mcp
# Install the MCP package for this endpoint's cluster npx -y @agentutility/mcp-<cluster> # Required: EVM private key with USDC on Base export X402_PRIVATE_KEY=0x... # Then call the scrape-website tool from your MCP-aware agent.
MCP server handles payment automatically — your coding agent just calls the tool by name.
METADATA
- tags
- scrapefetchhtmlextractmetadataurl
- methods
- POST
- cluster
- webprobe
- price
- $0.04 USDC per call
ADJACENT — other endpoints in webprobe
| endpoint | description | price |
|---|---|---|
| answer-web | Answers natural-language questions with live web research, returning a synthesized answer with inline [N] citations and the source URLs. | $0.04 |
| arxiv-summarize | Turn an arXiv paper into a structured research summary without reading the PDF. | $0.04 |
| rss-from-anything | Turn any webpage into an RSS feed, even sites that never published one. | $0.04 |
| scrape | Scrape any webpage. | $0.04 |
| screenshot | Turn any public URL into a PNG or JPG image on demand. | $0.04 |
| webpage-diff | Detects changes on a webpage: fetches a URL, strips HTML to plain text, computes a SHA-256 hash, and (when given a previous hash or text)… | $0.04 |
| website-screenshot | Capture a screenshot of any live URL and get back a hosted PNG or JPG. | $0.04 |
| arxiv-search | Search arXiv's full paper catalog by query, author, category, or a specific id-list and get back structured results instead of scraping HTML. | $0.03 |
SEE ALSO