Interface: ReadWebPageOptions
Defined in: packages/tools/src/web/web-fetch.ts:135
Properties
allowedHosts?
optionalallowedHosts?:string[]
Defined in: packages/tools/src/web/web-fetch.ts:141
Only these hosts (exact, or .example.com for a domain and its subdomains).
allowPrivateNetworks?
optionalallowPrivateNetworks?:boolean
Defined in: packages/tools/src/web/web-fetch.ts:145
Allow localhost and private ranges. Default false.
blockedHosts?
optionalblockedHosts?:string[]
Defined in: packages/tools/src/web/web-fetch.ts:143
Never these hosts (same matching).
content?
optionalcontent?:"main"|"full"
Defined in: packages/tools/src/web/web-fetch.ts:167
full (default): all of an HTML page's readable text. main: its main
content — the article body without navigation, header and footer chrome,
sidebars, banners, ads, share bars and trailing link lists. When the cut
leaves too little (an unusual layout), the full text is kept and
WebPage.content says full. PDFs and plain text are unaffected.
fallback?
optionalfallback?: (request) =>Promise<WebFallbackPage|null|undefined>
Defined in: packages/tools/src/web/web-fetch.ts:157
Read a page the plain read could not. Return null to keep the plain outcome.
Parameters
request
Returns
Promise<WebFallbackPage | null | undefined>
fetch?
optionalfetch?: {(input,init?):Promise<Response>; (input,init?):Promise<Response>; }
Defined in: packages/tools/src/web/web-fetch.ts:151
The platform fetch, when there is no transport.
Call Signature
(
input,init?):Promise<Response>
Parameters
input
URL | RequestInfo
init?
RequestInit
Returns
Promise<Response>
Call Signature
(
input,init?):Promise<Response>
Parameters
input
string | URL | Request
init?
RequestInit
Returns
Promise<Response>
maxBytes?
optionalmaxBytes?:number
Defined in: packages/tools/src/web/web-fetch.ts:137
Bytes downloaded at most. Default 5 MB.
readPdf?
optionalreadPdf?:false|PdfTextReader
Defined in: packages/tools/src/web/web-fetch.ts:155
Read a PDF's text layer. Default: unpdf when installed. false: keep the bytes, read no text.
resolveHost?
optionalresolveHost?: (hostname) =>Promise<string[]>
Defined in: packages/tools/src/web/web-fetch.ts:149
Unpinned: check a hostname's addresses before the platform fetch resolves it again.
Parameters
hostname
string
Returns
Promise<string[]>
signal?
optionalsignal?:AbortSignal
Defined in: packages/tools/src/web/web-fetch.ts:168
thinChars?
optionalthinChars?:number
Defined in: packages/tools/src/web/web-fetch.ts:159
An HTML page with less readable text than this is thin. Default 200 characters.
timeoutMs?
optionaltimeoutMs?:number
Defined in: packages/tools/src/web/web-fetch.ts:139
One deadline for the whole read, redirects and body included. Default 20 s.
transport?
optionaltransport?:WebFetchTransport
Defined in: packages/tools/src/web/web-fetch.ts:147
Pinned resolution and connection (see WebFetchTransport). Wins over fetch and resolveHost.
userAgent?
optionaluserAgent?:string
Defined in: packages/tools/src/web/web-fetch.ts:153
Sent as User-Agent.