Skip to main content

Interface: ReadWebPageOptions

Defined in: packages/tools/src/web/web-fetch.ts:135

Properties​

allowedHosts?​

optional allowedHosts?: string[]

Defined in: packages/tools/src/web/web-fetch.ts:141

Only these hosts (exact, or .example.com for a domain and its subdomains).


allowPrivateNetworks?​

optional allowPrivateNetworks?: boolean

Defined in: packages/tools/src/web/web-fetch.ts:145

Allow localhost and private ranges. Default false.


blockedHosts?​

optional blockedHosts?: string[]

Defined in: packages/tools/src/web/web-fetch.ts:143

Never these hosts (same matching).


content?​

optional content?: "main" | "full"

Defined in: packages/tools/src/web/web-fetch.ts:167

full (default): all of an HTML page's readable text. main: its main content — the article body without navigation, header and footer chrome, sidebars, banners, ads, share bars and trailing link lists. When the cut leaves too little (an unusual layout), the full text is kept and WebPage.content says full. PDFs and plain text are unaffected.


fallback?​

optional fallback?: (request) => Promise<WebFallbackPage | null | undefined>

Defined in: packages/tools/src/web/web-fetch.ts:157

Read a page the plain read could not. Return null to keep the plain outcome.

Parameters​

request​

WebFallbackRequest

Returns​

Promise<WebFallbackPage | null | undefined>


fetch?​

optional fetch?: {(input, init?): Promise<Response>; (input, init?): Promise<Response>; }

Defined in: packages/tools/src/web/web-fetch.ts:151

The platform fetch, when there is no transport.

Call Signature​

(input, init?): Promise<Response>

MDN Reference

Parameters​
input​

URL | RequestInfo

init?​

RequestInit

Returns​

Promise<Response>

Call Signature​

(input, init?): Promise<Response>

MDN Reference

Parameters​
input​

string | URL | Request

init?​

RequestInit

Returns​

Promise<Response>


maxBytes?​

optional maxBytes?: number

Defined in: packages/tools/src/web/web-fetch.ts:137

Bytes downloaded at most. Default 5 MB.


readPdf?​

optional readPdf?: false | PdfTextReader

Defined in: packages/tools/src/web/web-fetch.ts:155

Read a PDF's text layer. Default: unpdf when installed. false: keep the bytes, read no text.


resolveHost?​

optional resolveHost?: (hostname) => Promise<string[]>

Defined in: packages/tools/src/web/web-fetch.ts:149

Unpinned: check a hostname's addresses before the platform fetch resolves it again.

Parameters​

hostname​

string

Returns​

Promise<string[]>


signal?​

optional signal?: AbortSignal

Defined in: packages/tools/src/web/web-fetch.ts:168


thinChars?​

optional thinChars?: number

Defined in: packages/tools/src/web/web-fetch.ts:159

An HTML page with less readable text than this is thin. Default 200 characters.


timeoutMs?​

optional timeoutMs?: number

Defined in: packages/tools/src/web/web-fetch.ts:139

One deadline for the whole read, redirects and body included. Default 20 s.


transport?​

optional transport?: WebFetchTransport

Defined in: packages/tools/src/web/web-fetch.ts:147

Pinned resolution and connection (see WebFetchTransport). Wins over fetch and resolveHost.


userAgent?​

optional userAgent?: string

Defined in: packages/tools/src/web/web-fetch.ts:153

Sent as User-Agent.