ScraperAPI

ScraperAPI enables data collection from any public website with its web scraping API, without worrying about proxies, browsers, or CAPTCHA handling. langchain-scraperapi wraps this service, making it easy for AI agents to browse the web and scrape data from it.

Installation and Setup

• Install the Python package with pip install langchain-scraperapi.
• Obtain an API key from ScraperAPI and set the environment variable SCRAPERAPI_API_KEY.

Tools

The package offers 3 tools to scrape any website, get structured Google search results, and get structured Amazon search results respectively.

To import them:

%pip install langchain_scraperapi

from langchain_scraperapi.tools import (
    ScraperAPITool,
    ScraperAPIGoogleSearchTool,
    ScraperAPIAmazonSearchTool
)

Example use:

tool = ScraperAPITool()

result = tool.invoke({
    "url": "https://example.com",
    "output_format": "markdown"
})
print(result)

For a more detailed walkthrough of how to use these tools, visit the official repository.