Skip to main content

​
Custom Strategies

Waterfall-Fetch allows you to create and use custom fetching strategies to tailor the HTML retrieval process to your specific needs.

​
Creating a Custom Strategy

A custom strategy is a function that follows the FetchStrategyFunction type: 
type FetchStrategyFunction = (url: string, evalFunction?: OnPageEvaluationFunction) => Promise<StrategyResponse>;
Here’s an example of a custom strategy: 
import axios from "axios";

async function customAxiosStrategy(url: string): Promise<StrategyResponse> {
	const strategy = { name: "custom-axios", cost: 0 };
	try {
		const response = await axios.get(url, {
			headers: {
				"User-Agent": "Custom User Agent",
				// Add other custom headers here
			},
		});
		return {
			success: true,
			strategy,
			html: response.data,
			error: null,
			status: response.status,
		};
	} catch (error) {
		return {
			success: false,
			strategy,
			html: null,
			error,
			status: error.response?.status || null,
		};
	}
}

​
Using Custom Strategies

To use custom strategies, pass them as an array to the set option when calling getHtml: 
import getHtml, { fetchWithNodeFetch, fetchWithStealthPuppeteer } from "waterfall-fetch";

const url = "https://example.com";

const customSet = [customAxiosStrategy, fetchWithNodeFetch, fetchWithStealthPuppeteer];

const result = await getHtml(url, { set: customSet });
This allows you to define the order and composition of strategies used in the waterfall approach.
⌘I