Freelance

Python Web Scraping Tutorial for 2026: Tools, Examples, and Best Practices

Web scraping with Python allows you to collect information from websites automatically and transform it into structured data for analysis, reporting, and applications.

Whether you want to extract product prices, collect job listings, monitor website updates, or gather article information, Python provides several libraries that make the process easier.

In this tutorial, you’ll learn how to build a web scraper from scratch using beginner-friendly tools such as Requests and Beautiful Soup. You’ll also explore more advanced solutions, including Playwright, Scrapy, and web scraping APIs.

We’ll cover practical examples, complete Python scripts, data extraction techniques, error handling, pagination, and responsible scraping practices.

By the end, you’ll understand which tools to use for different scraping tasks and how to build a reliable scraper that can handle real-world websites.

TL;DR: Python Web Scraping Explained

Most Python web scraping projects follow five basic steps:

  1. Send a request to a website.
  2. Download the page’s HTML.
  3. Parse the HTML using a library such as Beautiful Soup.
  4. Extract and clean the information you need.
  5. Save the results in a suitable format.

The most common tools include:

ToolMain purpose
RequestsDownloading website content
Beautiful SoupParsing HTML and extracting information
lxmlFast HTML and XML parsing
PlaywrightAccessing JavaScript-rendered content
ScrapyBuilding larger web crawlers
pandasCleaning and exporting extracted data
Scraping APIsHandling browser rendering and scraping infrastructure

A useful starting point: Use Requests and Beautiful Soup for static websites. Choose Playwright when content requires JavaScript or browser interaction. Use Scrapy when your project needs to crawl many pages.

Before writing a scraper, check whether the website provides an official API. An API is often more reliable than extracting information directly from HTML.

What Is Web Scraping With Python?

Web scraping is the process of automatically collecting information from web pages.

Instead of manually copying information from a website, you write a Python script that visits a page, reads its HTML, identifies the information you need, and extracts it into a structured format.

For example, a web scraper might collect:

  • Product names and prices.
  • Job titles and descriptions.
  • Article headlines and publication dates.
  • Business names and contact information.
  • Website links and metadata.
  • Tables and publicly available datasets.

Most basic Python scraping scripts have three main components:

1. Request: The script sends an HTTP request to retrieve a web page.

2. Parsing: A library processes the HTML and makes its elements easier to navigate.

3. Extraction: The script identifies the required information and converts it into structured records.

For beginner projects, Requests and Beautiful Soup are usually enough.

As your project becomes more complex, you may need additional tools:

  • Playwright: For JavaScript-rendered pages and browser interactions.
  • Scrapy: For crawling multiple pages and managing larger scraping projects.
  • pandas: For cleaning, organizing, and exporting extracted data.
  • APIs: For retrieving structured information directly from a data provider.

If your project involves submitting information to a website rather than collecting it, you may also need to understand how to send Python POST requests.

The main principle is simple: choose the smallest tool that can reliably complete your task.

The Python Web Scraping Tools Used in This Tutorial

We’ll begin with simple tools and introduce more advanced options only when they solve a specific problem.

ToolWhat it does
RequestsSends HTTP requests and downloads page content
Beautiful SoupSearches HTML and extracts selected elements
lxmlProvides efficient HTML parsing
PlaywrightOpens and interacts with browser pages
ScrapyManages large crawling projects
ScrapingBeeProvides a scraping API for rendering, proxies, and extraction
pandasOrganizes and exports structured data

Requests and Beautiful Soup

Requests handles communication with websites, while Beautiful Soup makes it easier to search through the returned HTML.

Together, they provide a straightforward way to extract information from static pages.

Playwright

Playwright automates a real browser. It can wait for JavaScript content, click buttons, scroll through pages, and interact with website elements.

Scrapy

Scrapy is a framework designed for larger crawling projects. It provides features for following links, handling pagination, managing requests, and exporting results.

Scraping APIs

A scraping API can handle some of the infrastructure involved in collecting website content, such as JavaScript rendering, proxy management, and browser execution.

pandas

pandas is useful after extraction. It allows you to organize records into DataFrames, clean values, remove duplicates, and export results.

Requests + Beautiful Soup vs. Playwright vs. Scrapy

Choosing the right tool can save time and prevent unnecessary complexity.

ToolBest forUse it whenAvoid it when
Requests + Beautiful SoupStatic websitesThe required information is already in the HTMLContent requires JavaScript or browser interaction
PlaywrightDynamic websitesContent loads after JavaScript, clicking, or scrollingA simple HTTP request is sufficient
ScrapyLarge crawling projectsYou need pagination, link-following, retries, and structured exportsYou’re building a small one-page scraper

A practical decision process looks like this:

  1. Check whether the website provides an API.
  2. If the data is available in the HTML, use Requests and Beautiful Soup.
  3. If JavaScript loads the data, consider Playwright.
  4. If you need to crawl many pages, consider Scrapy.

For additional examples, see our guide on how to scrape websites with Playwright.

If you’re already comfortable with Beautiful Soup, you can also explore techniques for improving HTML parsing and scraping performance.

Build Your First Web Scraper With Requests and Beautiful Soup

Let’s start with a practical example.

We’ll collect the first five stories from the Hacker News homepage. This is a useful beginner project because the required information is available in the page’s HTML.

Step 1: Install the Required Libraries

Open your terminal or command prompt and run:

pip install requests beautifulsoup4 lxml

Each package has a specific purpose:

  • requests downloads the web page.
  • beautifulsoup4 searches and extracts information from HTML.
  • lxml provides an efficient parser for Beautiful Soup.

Step 2: Inspect the Website’s HTML

Before writing code, open the Hacker News homepage in your browser.

Right-click a story and select Inspect to examine its HTML structure.

For this example, we need to identify:

  • The HTML element containing each story.
  • The link containing the story title.
  • The section containing the author, score, age, and comments.

The relevant structure uses:

  • tr.athing.submission for each story row.
  • .titleline > a for the title and URL.
  • .subtext for additional story information.

These selectors give us the information needed for extraction.

Step 3: Write the Python Scraper

Create a Python file named hacker_news_scraper.py and add the following code:

from __future__ import annotations

import re
from dataclasses import dataclass
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup
from requests import RequestException


HN_URL = "https://news.ycombinator.com/"
TIMEOUT_SECONDS = 10

HEADERS = {
    "User-Agent": "python-web-scraping-tutorial/1.0"
}


@dataclass
class Story:
    rank: int
    title: str
    url: str
    points: int
    author: str
    age: str
    comments: int


def extract_number(text: str | None) -> int:
    """Extract the first integer from a string."""
    if not text:
        return 0

    match = re.search(r"\d+", text.replace("\xa0", " "))
    return int(match.group()) if match else 0


def fetch_html(url: str) -> str:
    """Download a page and return its HTML."""
    try:
        response = requests.get(
            url,
            headers=HEADERS,
            timeout=TIMEOUT_SECONDS,
        )
        response.raise_for_status()

    except RequestException as exc:
        raise RuntimeError(
            f"Failed to fetch {url}: {exc}"
        ) from exc

    return response.text


def parse_hacker_news(
    html: str,
    limit: int = 5
) -> list[Story]:

    """Extract stories from Hacker News HTML."""
    soup = BeautifulSoup(html, "lxml")
    stories: list[Story] = []

    for story_row in soup.select(
        "tr.athing.submission"
    )[:limit]:

        title_link = story_row.select_one(
            ".titleline > a"
        )

        if title_link is None:
            continue

        href = title_link.get("href")

        story_url = urljoin(
            HN_URL,
            href if isinstance(href, str) else ""
        )

        subtext_row = story_row.find_next_sibling("tr")

        subtext = (
            subtext_row.select_one(".subtext")
            if subtext_row else None
        )

        rank_text = story_row.select_one(".rank")
        score_text = (
            subtext.select_one(".score")
            if subtext else None
        )

        author_link = (
            subtext.select_one(".hnuser")
            if subtext else None
        )

        age_link = (
            subtext.select_one(".age a")
            if subtext else None
        )

        comment_link = None

        if subtext:
            for link in subtext.select("a"):
                link_text = link.get_text(
                    " ",
                    strip=True
                ).lower()

                if (
                    "comment" in link_text
                    or link_text == "discuss"
                ):
                    comment_link = link
                    break

        stories.append(
            Story(
                rank=extract_number(
                    rank_text.get_text()
                    if rank_text else None
                ),
                title=title_link.get_text(
                    " ",
                    strip=True
                ),
                url=story_url,
                points=extract_number(
                    score_text.get_text()
                    if score_text else None
                ),
                author=(
                    author_link.get_text(
                        " ",
                        strip=True
                    )
                    if author_link else "unknown"
                ),
                age=(
                    age_link.get_text(
                        " ",
                        strip=True
                    )
                    if age_link else "unknown"
                ),
                comments=extract_number(
                    comment_link.get_text()
                    if comment_link else None
                ),
            )
        )

    return stories


def main() -> None:
    html = fetch_html(HN_URL)
    stories = parse_hacker_news(html)

    for story in stories:
        print(f"{story.rank}. {story.title}")
        print(f"URL: {story.url}")

        print(
            f"{story.points} points by "
            f"{story.author} | "
            f"{story.age} | "
            f"{story.comments} comments"
        )

        print()


if __name__ == "__main__":
    main()

Step 4: Run the Scraper

Save the file and execute it:

python hacker_news_scraper.py

The script should return structured information similar to this:

1. Example Hacker News Story
   URL: https://example.com/story
   34 points by example_user | 1 hour ago | 25 comments

The actual stories and values will change because Hacker News updates continuously.

However, the scraper should consistently return the same fields:

  • Rank.
  • Title.
  • URL.
  • Points.
  • Author.
  • Publication age.
  • Comment count.

Understanding How the Script Works

The example uses several important Python techniques.

Fetching HTML

The fetch_html() function sends an HTTP request and returns the page content.

response = requests.get(
    url,
    headers=HEADERS,
    timeout=TIMEOUT_SECONDS
)

response.raise_for_status()

The timeout prevents the request from waiting indefinitely, while raise_for_status() helps identify unsuccessful HTTP responses.

Parsing HTML

Beautiful Soup converts the downloaded HTML into a structure that Python can search.

soup = BeautifulSoup(html, "lxml")

Selecting Elements

CSS selectors identify the HTML elements containing the information we need.

soup.select("tr.athing.submission")

This selects the story rows.

Cleaning Numeric Values

Website information often contains text such as 34 points or 25 comments.

The extract_number() function converts these values into integers, making them easier to sort, compare, and analyze.

Handling Missing Information

The script checks whether an element exists before trying to extract its contents. This reduces errors when certain fields are unavailable.

Clean and Extract Structured Data

Once the scraper has collected information, the next step is organizing it.

In our example, the extracted records are stored as Story objects.

For example:

Story(
    rank=1,
    title="Example Hacker News Story",
    url="https://example.com/story",
    points=31,
    author="example_user",
    age="1 hour ago",
    comments=25,
)

Structured data is easier to filter, sort, analyze, and export than raw HTML or unorganized text.

For example, you can display only stories with at least 20 points:

popular_stories = [
    story
    for story in stories
    if story.points >= 20
]

for story in popular_stories:
    print(
        f"{story.title} "
        f"({story.points} points)"
    )

At this stage, you’ve completed the basic Python web scraping workflow:

  1. Download a real web page.
  2. Inspect its HTML structure.
  3. Select the required elements.
  4. Extract relevant information.
  5. Clean numeric values.
  6. Store the results in structured objects.

This approach works well for many static websites.

Common Beginner Mistakes in Python Web Scraping

When building your first scraper, you may encounter errors caused by incorrect requests, missing elements, unexpected HTML structures, or dynamically loaded content.

Understanding these common problems will make debugging much easier.

1. Parsing Before Checking the Response

Always verify that the website returned a successful response before passing its content to Beautiful Soup.

response = requests.get(
    HN_URL,
    headers=HEADERS,
    timeout=10
)

response.raise_for_status()

soup = BeautifulSoup(
    response.text,
    "lxml"
)

This helps prevent your script from processing an error page as if it were the expected website.

2. Forgetting to Set a Timeout

Requests does not automatically impose a timeout.

Without one, a slow or unresponsive website could cause your script to wait much longer than expected.

response = requests.get(
    HN_URL,
    headers=HEADERS,
    timeout=10
)

3. Expecting Beautiful Soup to Run JavaScript

Beautiful Soup parses HTML, but it does not execute JavaScript.

If the information appears in your browser but is missing from response.text, the website may be loading that information dynamically.

In that situation, consider using Playwright or an appropriate scraping API.

4. Using Fragile CSS Selectors

Avoid selectors that depend heavily on the exact position of elements in the page.

For example:

# More reliable
soup.select("tr.athing.submission")

A deeply nested selector can be more fragile:

# More dependent on page structure
soup.select(
    "body > center > table > tr > td > table > tr:nth-child(3)"
)

Whenever possible, use meaningful class names and elements that are directly associated with the information you need.

5. Ignoring Data Cleaning

Extracted values often contain extra spaces, symbols, or descriptive text.

Clean values before storing them if you plan to sort, filter, or calculate with them.

For example:

points = extract_number("31 points")
comments = extract_number("25 comments")

6. Assuming Website HTML Never Changes

Websites regularly update their layouts, class names, and page structures.

A scraper that works today may stop working after a website redesign.

To make your code easier to maintain:

  • Check for missing elements.
  • Keep extraction logic in separate functions.
  • Save sample HTML when debugging.
  • Test important selectors after website changes.
  • Handle unexpected responses clearly.

7. Sending Too Many Requests

Sending large numbers of requests too quickly can overload a website or trigger access restrictions.

For small projects, a simple delay may be sufficient:

import time

for page_url in page_urls:
    html = fetch_html(page_url)

    time.sleep(2)

For larger projects, use appropriate request limits, caching, and controlled concurrency.

Always check the website’s terms, access rules, and available APIs before collecting data.

When Static Web Scraping Is No Longer Enough

Requests and Beautiful Soup are excellent starting points, but they cannot handle every website.

Static scraping works when the information is included in the initial HTML response.

However, some websites rely on JavaScript to load content after the page opens.

Common signs include:

  • Information appears in your browser but not in response.text.
  • The initial HTML contains empty containers that are filled later.
  • Search results appear after clicking a button.
  • Additional products load when you scroll.
  • Content changes after selecting filters or entering information.
  • Data arrives through background network requests.

You can inspect the downloaded HTML first:

html = fetch_html("https://example.com")

print(html[:1000])

If the required information is missing, changing your Beautiful Soup selectors will not solve the problem.

You have several options:

  1. Look for an API or background request that provides the data.
  2. Use Playwright to load and interact with the page.
  3. Use a scraping API that supports JavaScript rendering.

Before moving to browser automation, open your browser’s Developer Tools and inspect the Network tab.

Some websites load their information from JSON endpoints. If an endpoint is publicly accessible and permitted for your use, requesting the structured data directly may be faster than loading the entire page.

Use Playwright for JavaScript-Rendered Websites

When a website relies on JavaScript to display its content, a browser automation tool may be necessary.

Playwright opens a real browser, waits for the page to render, and allows Python to interact with page elements.

Use Playwright when:

  • Data appears after JavaScript execution.
  • You need to click buttons or navigate through menus.
  • Additional content loads when scrolling.
  • A page requires search form interaction.
  • You need to wait for specific elements to appear.
  • The website relies on browser session information.

Playwright is more resource-intensive than Requests, so it should not be your default choice for simple static pages.

Install Playwright

Run:

pip install playwright
playwright install chromium

This installs Playwright and the Chromium browser.

Playwright also supports Firefox and WebKit.

Scrape a JavaScript-Rendered Page

For this example, we’ll use:

https://quotes.toscrape.com/js-delayed

The website displays quotes after a short JavaScript delay.

A basic Requests scraper may not retrieve the final content, but Playwright can wait for the quotes to appear.

from __future__ import annotations

from dataclasses import dataclass

from playwright.sync_api import (
    Error as PlaywrightError,
    TimeoutError as PlaywrightTimeoutError,
    sync_playwright,
)


URL = "https://quotes.toscrape.com/js-delayed/"
TIMEOUT_MS = 15_000


@dataclass
class Quote:
    text: str
    author: str
    tags: list[str]


def scrape_quotes(limit: int = 5) -> list[Quote]:
    """Scrape quotes from a JavaScript-rendered page."""

    try:
        with sync_playwright() as p:
            browser = p.chromium.launch(
                headless=True
            )

            page = browser.new_page(
                user_agent="python-web-scraping-tutorial/1.0"
            )

            try:
                page.goto(
                    URL,
                    wait_until="domcontentloaded",
                    timeout=TIMEOUT_MS
                )

                page.wait_for_selector(
                    ".quote",
                    timeout=TIMEOUT_MS
                )

                quote_cards = page.locator(".quote")

                quote_count = min(
                    quote_cards.count(),
                    limit
                )

                quotes: list[Quote] = []

                for index in range(quote_count):
                    card = quote_cards.nth(index)

                    quotes.append(
                        Quote(
                            text=card.locator(
                                ".text"
                            ).inner_text().strip(),

                            author=card.locator(
                                ".author"
                            ).inner_text().strip(),

                            tags=[
                                tag.strip()
                                for tag in card.locator(
                                    ".tag"
                                ).all_inner_texts()
                            ],
                        )
                    )

                return quotes

            finally:
                browser.close()

    except PlaywrightTimeoutError as exc:
        raise RuntimeError(
            "Timed out waiting for quote content."
        ) from exc

    except PlaywrightError as exc:
        raise RuntimeError(
            f"Playwright failed: {exc}"
        ) from exc


def main() -> None:
    quotes = scrape_quotes()

    for quote in quotes:
        print(quote.text)
        print(f"Author: {quote.author}")

        print(
            "Tags: "
            f"{', '.join(quote.tags) if quote.tags else 'none'}"
        )

        print()


if __name__ == "__main__":
    main()

Example Output

“The world as we have created it is a process
of our thinking. It cannot be changed without
changing our thinking.”

Author: Albert Einstein
Tags: change, deep-thoughts, thinking, world

“It is our choices, Harry, that show what we
truly are, far more than our abilities.”

Author: J.K. Rowling
Tags: abilities, choices

Understanding the Playwright Script

The script follows the same general extraction process as the Requests example, but uses a browser.

It performs these steps:

  1. Launches Chromium.
  2. Opens the target website.
  3. Waits for the page to load.
  4. Waits for elements matching .quote.
  5. Extracts quote text, author names, and tags.
  6. Returns structured Quote objects.

The most important line is:

page.wait_for_selector(
    ".quote",
    timeout=TIMEOUT_MS
)

This tells Playwright to wait until the expected content appears.

It is generally more reliable than using a fixed time.sleep() delay because the script waits for an actual page condition.

When Should You Use Playwright?

Playwright is particularly useful for:

  • JavaScript-rendered search results.
  • Websites with dynamically loaded product listings.
  • Pages with a “Load more” button.
  • Infinite scrolling.
  • Search forms and filters.
  • Content that requires browser interaction.

For static pages, Requests and Beautiful Soup remain simpler and faster.

Use Scrapy for Larger Web Crawling Projects

Requests and Beautiful Soup are useful for smaller scraping tasks, while Playwright handles browser-dependent pages.

But what happens when you need to collect information from hundreds or thousands of pages?

This is where Scrapy becomes useful.

Scrapy is a Python framework designed for web crawling. It provides a structured way to manage requests, follow links, handle pagination, process extracted items, and export results.

Use Scrapy when you need to:

  • Crawl many pages automatically.
  • Follow links between pages.
  • Collect information from paginated listings.
  • Export data into CSV, JSON, or JSON Lines.
  • Manage retries and request delays.
  • Build repeatable scraping projects.
  • Organize extraction logic into reusable components.

For a single static page, Scrapy may be unnecessary. It becomes valuable when your project grows into a crawler.

Install Scrapy

Install the package:

pip install scrapy

Create a new project:

scrapy startproject bookcrawler
cd bookcrawler

For this example, we’ll crawl a few pages from:

https://books.toscrape.com

This practice website contains book titles, prices, ratings, availability information, and pagination.

Create a Scrapy Spider

Generate a spider:

scrapy genspider books books.toscrape.com

This creates a file named:

bookcrawler/spiders/books.py

Replace its contents with the following code:

from __future__ import annotations

from collections.abc import Iterable
from typing import Any

import scrapy
from scrapy.http import Response


class BooksSpider(scrapy.Spider):

    name = "books"

    allowed_domains = [
        "books.toscrape.com"
    ]

    start_urls = [
        "https://books.toscrape.com/"
    ]

    max_pages = 3

    custom_settings = {
        "DOWNLOAD_DELAY": 1,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2,
    }

    def parse(
        self,
        response: Response
    ) -> Iterable[dict[str, Any] | scrapy.Request]:

        current_page = response.meta.get(
            "page",
            1
        )

        for book in response.css(
            "article.product_pod"
        ):

            title = book.css(
                "h3 a::attr(title)"
            ).get(default="").strip()

            relative_url = book.css(
                "h3 a::attr(href)"
            ).get(default="")

            price = book.css(
                ".price_color::text"
            ).get(default="").strip()

            availability = " ".join(
                book.css(
                    ".availability::text"
                ).getall()
            ).strip()

            rating_class = book.css(
                "p.star-rating::attr(class)"
            ).get(default="")

            rating = rating_class.replace(
                "star-rating",
                ""
            ).strip()

            yield {
                "title": title,
                "url": response.urljoin(
                    relative_url
                ),
                "price": price,
                "availability": availability,
                "rating": rating,
                "page": current_page,
            }

        next_page = response.css(
            "li.next a::attr(href)"
        ).get()

        if (
            next_page
            and current_page < self.max_pages
        ):

            yield response.follow(
                next_page,
                callback=self.parse,
                meta={
                    "page": current_page + 1
                },
            )

Run the Scrapy Spider

To collect the results and save them as JSON:

scrapy crawl books -O books.json

To export the results as CSV:

scrapy crawl books -O books.csv

An example record looks like this:

{
  "title": "A Light in the Attic",
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "price": "£51.77",
  "availability": "In stock",
  "rating": "Three",
  "page": 1
}

What Does This Spider Do?

The spider uses several important Scrapy features:

  • start_urls defines the initial page.
  • parse() processes the downloaded response.
  • CSS selectors extract the book information.
  • yield returns structured records.
  • response.follow() navigates to the next page.
  • max_pages limits the example to three pages.

Unlike a basic Requests script, Scrapy manages the crawling process through its request and callback system.

When Is Scrapy Worth Using?

Scrapy is a good choice for projects involving:

  • Large product catalogs.
  • Multiple categories and pagination.
  • Scheduled data collection.
  • Repeated crawling tasks.
  • Structured CSV and JSON exports.
  • Custom request handling.
  • Data processing pipelines.

For a small one-page project, Requests and Beautiful Soup may be easier. For larger crawlers, Scrapy’s structure can make the code easier to maintain.

You can also compare Scrapy with other crawling frameworks, such as Crawlee for Python.

Using a Python Web Scraping API

Local scraping tools provide flexibility, but larger projects can introduce additional infrastructure requirements.

You may need to manage:

  • Browser installations.
  • JavaScript rendering.
  • Proxy configuration.
  • Geographic targeting.
  • Request retries.
  • Screenshots.
  • Scaling and execution environments.

A scraping API can handle some of these responsibilities through a hosted service.

For example, ScrapingBee provides an API that can retrieve website content, render JavaScript, and return structured extraction results.

An API may be useful when:

  • Browser setup becomes difficult to maintain.
  • You need JavaScript-rendered HTML.
  • You need supported proxy or geographic options.
  • Your project requires repeated scraping requests.
  • You want structured extraction without maintaining every selector locally.

However, an API is not necessary for every project. If Requests and Beautiful Soup already work reliably, there is no reason to introduce another service.

Scrape a JavaScript Page With ScrapingBee

We’ll use the same delayed quotes website:

https://quotes.toscrape.com/js-delayed

First, create an account with your chosen API provider and obtain an API key.

Install Requests:

pip install requests

Set your API key as an environment variable.

On macOS or Linux:

export SCRAPINGBEE_API_KEY="your-api-key"

On Windows PowerShell:

setx SCRAPINGBEE_API_KEY "your-api-key"

Now create the request:

from __future__ import annotations

import os

import requests
from requests import RequestException


SCRAPINGBEE_API_URL = (
    "https://app.scrapingbee.com/api/v1"
)

TARGET_URL = (
    "https://quotes.toscrape.com/js-delayed/"
)

TIMEOUT_SECONDS = 30


def fetch_with_scrapingbee(
    url: str
) -> str:

    api_key = os.getenv(
        "SCRAPINGBEE_API_KEY"
    )

    if not api_key:
        raise RuntimeError(
            "Missing SCRAPINGBEE_API_KEY "
            "environment variable."
        )

    try:

        response = requests.get(
            SCRAPINGBEE_API_URL,

            params={
                "api_key": api_key,
                "url": url,
                "render_js": "True",
                "wait_for": ".quote",
            },

            timeout=TIMEOUT_SECONDS,
        )

        response.raise_for_status()

    except RequestException as exc:

        raise RuntimeError(
            f"ScrapingBee request failed: {exc}"
        ) from exc

    return response.text


def main() -> None:

    html = fetch_with_scrapingbee(
        TARGET_URL
    )

    print(html[:1000])


if __name__ == "__main__":
    main()

The API returns HTML after JavaScript rendering has completed.

You can then process the returned HTML using Beautiful Soup or use the provider’s extraction features to obtain structured data.

Extract Structured Data Without Beautiful Soup

Sometimes you already know which information you need and do not want to maintain a separate HTML parser.

In supported scraping APIs, extraction rules can return structured JSON directly.

For example, ScrapingBee provides an extract_rules parameter that allows you to define the fields you want to collect.

The following example extracts quotes, authors, and tags.

from __future__ import annotations

import json
import os

from pprint import pprint

import requests
from requests import RequestException


SCRAPINGBEE_API_URL = (
    "https://app.scrapingbee.com/api/v1"
)

TARGET_URL = (
    "https://quotes.toscrape.com/js-delayed/"
)

TIMEOUT_SECONDS = 45


EXTRACT_RULES = {

    "quotes": {

        "selector": ".quote",

        "type": "list",

        "output": {

            "text": ".text",

            "author": ".author",

            "tags": {

                "selector": ".tag",

                "type": "list",

            },

        },

    }

}


def scrape_quotes_with_extract_rules() -> dict:

    api_key = os.getenv(
        "SCRAPINGBEE_API_KEY"
    )

    if not api_key:
        raise RuntimeError(
            "Missing SCRAPINGBEE_API_KEY environment variable."
        )

    try:

        response = requests.get(

            SCRAPINGBEE_API_URL,

            params={

                "api_key": api_key,

                "url": TARGET_URL,

                "render_js": "True",

                "wait_for": ".quote",

                "extract_rules": json.dumps(
                    EXTRACT_RULES
                ),

            },

            timeout=TIMEOUT_SECONDS,

        )

        response.raise_for_status()

    except RequestException as exc:

        raise RuntimeError(
            f"ScrapingBee request failed: {exc}"
        ) from exc

    return response.json()


def main() -> None:

    data = scrape_quotes_with_extract_rules()

    pprint(data)


if __name__ == "__main__":
    main()

Example JSON Response

{
  "quotes": [
    {
      "text": "The world as we have created it is a process of our thinking.",
      "author": "Albert Einstein",
      "tags": [
        "change",
        "deep-thoughts",
        "thinking",
        "world"
      ]
    },
    {
      "text": "It is our choices that show what we truly are.",
      "author": "J.K. Rowling",
      "tags": [
        "abilities",
        "choices"
      ]
    }
  ]
}

This approach allows you to retrieve structured information without writing a separate Beautiful Soup extraction function.

For stable page layouts, CSS-based extraction rules are often predictable. For more complicated layouts, some providers also offer AI-based extraction.

When Should You Use AI Extraction?

AI extraction can be useful when:

  • The HTML structure is difficult to work with.
  • Page layouts change frequently.
  • The information is easier to describe than to locate with selectors.
  • You need to extract multiple fields from complex content.

However, AI extraction may introduce additional costs and should not automatically replace traditional selectors.

For stable pages, normal CSS extraction is often simpler and easier to maintain.

Handling Retries, Timeouts, and Reliability

A good scraper should handle temporary failures without hanging indefinitely or returning incomplete data.

You don’t need a complicated reliability system for every project, but a few safeguards can make a significant difference.

Always Set Request Timeouts

Requests does not set a default timeout.

Always define one:

import requests

response = requests.get(
    "https://news.ycombinator.com/",
    timeout=(5, 15)
)

response.raise_for_status()

The two values represent:

  • 5 seconds to establish the connection.
  • 15 seconds to wait for the server response.

For smaller scripts, a single timeout value is also acceptable:

response = requests.get(
    url,
    timeout=10
)

Retry Temporary Failures Carefully

Some HTTP errors are temporary.

Examples include:

  • 429: Too many requests.
  • 500: Internal server error.
  • 502: Bad gateway.
  • 503: Service unavailable.
  • 504: Gateway timeout.

A limited retry strategy with increasing delays can help manage temporary failures.

from __future__ import annotations

import requests

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry


def create_session() -> requests.Session:

    retry_strategy = Retry(

        total=3,

        backoff_factor=1,

        status_forcelist=[
            429,
            500,
            502,
            503,
            504
        ],

        allowed_methods=["GET"],

    )

    adapter = HTTPAdapter(
        max_retries=retry_strategy
    )

    session = requests.Session()

    session.headers.update({

        "User-Agent":
        "python-web-scraping-tutorial/1.0"

    })

    session.mount(
        "https://",
        adapter
    )

    session.mount(
        "http://",
        adapter
    )

    return session


session = create_session()

response = session.get(
    "https://news.ycombinator.com/",
    timeout=(5, 15)
)

response.raise_for_status()

This example retries a limited number of failed GET requests.

Do not retry indefinitely or use retries to force access when a website is blocking your scraper.

Make Errors Easier to Debug

When something goes wrong, you need to identify whether the problem comes from:

  • The HTTP request.
  • The website’s HTML structure.
  • A missing selector.
  • JavaScript-rendered content.
  • An unexpected response.

Saving the returned HTML can help you investigate:

from pathlib import Path

Path(
    "debug_page.html"
).write_text(
    response.text,
    encoding="utf-8"
)

You can then open the saved file and inspect the actual HTML your scraper received.

It is also important to check whether an expected element exists before accessing it.

Instead of:

title = soup.select_one(
    ".titleline > a"
).get_text(strip=True)

Use:

title_element = soup.select_one(
    ".titleline > a"
)

if title_element is None:

    raise RuntimeError(
        "Could not find the title element. "
        "The page layout may have changed."
    )

title = title_element.get_text(
    strip=True
)

Useful debugging habits include:

  • Logging failed URLs.
  • Keeping retries limited.
  • Saving sample HTML.
  • Checking selectors after website updates.
  • Recording unexpected responses.
  • Slowing down when a website returns rate-limit errors.

Store Scraped Data in the Right Format

After extracting information, you need to decide where to store it.

The best format depends on how you intend to use the data.

You do not always need a database. For many small and medium-sized projects, CSV and JSON files are sufficient.

A quick guide:

FormatBest use
Python lists and dictionariesTemporary storage during execution
CSVFlat data that will be opened in Excel
JSONStructured records and nested information
JSON LinesLarge collections of individual records
SQLiteLocal databases and structured queries
PostgreSQL or MySQLLarger applications and data pipelines

Keep Data in Memory First

For a small scraper, a list of dictionaries may be enough:

stories = [

    {

        "title": "Example story",

        "url": "https://example.com/story",

        "points": 42,

        "comments": 10,

    }

]

This structure is easy to process using Python, pandas, or a database.

Use pandas to Organize Your Results

pandas makes it easier to inspect, filter, and sort extracted information.

import pandas as pd

df = pd.DataFrame(stories)

df = df.sort_values(
    "points",
    ascending=False
)

print(df.head())

Save Data as CSV

CSV is particularly useful when you want to open the results in Excel or Google Sheets.

df.to_csv(
    "stories.csv",
    index=False
)

Save Data as JSON

JSON is a good choice when your records contain nested information.

df.to_json(
    "stories.json",
    orient="records",
    indent=2
)

Use JSON Lines for Larger Datasets

JSON Lines stores one record per line.

df.to_json(
    "stories.jsonl",
    orient="records",
    lines=True
)

This format can be convenient when processing large collections one record at a time.

Use a Database When Necessary

A database becomes useful when your project requires frequent updates, filtering, joining, deduplication, or long-term storage.

For a small local project, SQLite is often sufficient:

import sqlite3

with sqlite3.connect(
    "scraped_data.db"
) as connection:

    df.to_sql(
        "stories",
        connection,
        if_exists="replace",
        index=False
    )

For larger applications and data pipelines, you may prefer PostgreSQL, MySQL, or another server-based database.

Avoid introducing unnecessary infrastructure. Start with the simplest storage option that meets your requirements.

Store Data for AI and LLM Applications

If your scraped information will be used in an AI agent or large language model workflow, organize it so that the original source can be identified later.

Useful formats include:

  • Markdown for articles and documentation.
  • JSON for structured records.
  • JSON Lines for large collections.
  • CSV for simple tables.
  • Vector databases for semantic search.

Keep source information with each record:

{
  "title": "Example page",
  "url": "https://example.com/page",
  "scraped_at": "2026-05-21T12:00:00Z",
  "content": "Cleaned page content goes here."
}

Metadata helps identify the original source, update outdated records, remove duplicates, and trace information back to its origin.

Common Python Web Scraping Errors and How to Fix Them

Most scraping problems can be traced to a few common causes: failed requests, unexpected HTML, incorrect selectors, or JavaScript-loaded content.

The best approach is to investigate one step at a time.

Request and HTTP Errors

First, check whether the page was downloaded successfully.

response = requests.get(
    url,
    headers=HEADERS,
    timeout=10
)

print(response.status_code)
print(response.url)
print(response.text[:500])

response.raise_for_status()

Common HTTP status codes include:

Status codeMeaning
200Request successful
301 / 302Redirect
403Access forbidden
404Page not found
429Too many requests
500Internal server error
502 / 503Gateway or service error

If you receive a 403 or 429 response, don’t simply increase the number of retries.

Slow down, review the website’s rules, reduce request volume, and consider an official API if one is available.

Selector Errors

An error such as:

AttributeError: 'NoneType' object has no attribute 'get_text'

usually means your selector did not find the expected element.

Instead of accessing the result immediately:

title = soup.select_one(
    ".titleline > a"
).get_text(strip=True)

Check whether the element exists:

title_element = soup.select_one(
    ".titleline > a"
)

if title_element is None:

    raise RuntimeError(
        "The expected title element was not found."
    )

title = title_element.get_text(
    strip=True
)

If the selector still fails, save the HTML you actually received and inspect it.

JavaScript and Missing Content

Sometimes a browser displays information that is missing from response.text.

This often happens when JavaScript loads the content after the initial HTML response.

You can check for expected content:

html = response.text

if "Expected text from the page" not in html:

    print(
        "The data may be loaded by JavaScript."
    )

If the content is not included in the HTML response, Beautiful Soup cannot extract it.

Consider these alternatives:

  • Find the relevant background API request.
  • Use Playwright.
  • Use Selenium if WebDriver compatibility is required.
  • Use a scraping API that supports JavaScript rendering.

Encoding and Text Cleanup Issues

Extracted text may contain unusual spacing or non-breaking spaces.

You can normalize the text before storing it:

text = element.get_text(
    " ",
    strip=True
)

text = text.replace(
    "\xa0",
    " "
)

For numerical values, convert the text into the correct data type before performing calculations.

Check What Actually Rendered

For dynamic pages, inspecting the raw HTML may not be enough.

A screenshot can help you identify whether the browser loaded the expected content or displayed an error page instead.

For example, a scraping API may support screenshot output:

from pathlib import Path

import requests


response = requests.get(

    SCRAPINGBEE_API_URL,

    params={

        "api_key": api_key,

        "url": TARGET_URL,

        "render_js": "True",

        "screenshot": "True",

    }

)

response.raise_for_status()

Path(
    "debug_screenshot.png"
).write_bytes(
    response.content
)

You can also launch Playwright in visible browser mode:

browser = p.chromium.launch(
    headless=False,
    slow_mo=100
)

Setting headless=False opens the browser window. The slow_mo option slows down browser actions so you can observe navigation, clicks, and rendering.

A good debugging workflow is straightforward: fetch the page, inspect the response, verify the selectors, clean the extracted values, and then expand the scraper.

Legal, Ethical, and Maintenance Considerations

Web scraping involves more than writing code.

Automated data collection can affect website servers, user privacy, business rules, and intellectual property rights.

Before running a scraping project, consider whether the information can be collected and used appropriately.

Check the Website’s Rules First

Before scraping, review:

  • The website’s terms of service.
  • Its robots.txt file.
  • Available API documentation.
  • Data usage restrictions.
  • Rate limits and crawling guidance.
  • Relevant copyright and licensing requirements.

If an official API exists, it is often more stable than extracting information from HTML.

Remember that robots.txt provides crawling guidance. It does not automatically grant permission to collect, store, reuse, or republish every accessible page.

Collect Only the Information You Need

A responsible scraper should collect the minimum amount of information necessary for its intended purpose.

Pay particular attention to:

  • Personal information.
  • User profiles.
  • Contact details.
  • Private or restricted content.
  • Reviews and comments.
  • Copyrighted articles and media.
  • Information behind login pages or paywalls.

If you do not need a particular field, leave it out.

For example, a project tracking product prices may only need:

{
  "title": "Example product",
  "url": "https://example.com/product",
  "price": "29.99",
  "scraped_at": "2026-05-21T12:00:00Z"
}

Collecting only the necessary fields can simplify storage, maintenance, and compliance.

Build Your Scraper for Long-Term Maintenance

Websites change regularly. They may rename CSS classes, redesign pages, move information into JavaScript, or introduce new navigation elements.

Your scraper should be designed with these changes in mind.

Good practices include:

  • Keeping selectors readable.
  • Separating parsing logic into functions.
  • Logging failed URLs.
  • Recording collection timestamps.
  • Saving debug HTML or screenshots.
  • Testing important extraction functions.
  • Reviewing the scraper when the target website changes.

If the data is important to your business or application, treat the scraper as a maintained software integration rather than a one-time script.

Scrape Respectfully

Responsible scraping reduces unnecessary requests and avoids creating problems for the websites you access.

Follow these practices:

  • Send only the requests you need.
  • Use reasonable delays and request limits.
  • Avoid aggressive concurrency.
  • Stop after repeated blocking or CAPTCHA responses.
  • Use official APIs or exports when available.
  • Respect access restrictions.
  • Do not republish content unless you have the necessary rights.

The objective is to collect useful information responsibly, not to bypass every restriction a website introduces.

Conclusion: Choosing the Right Python Web Scraping Tool

The best Python web scraping setup is the simplest one that reliably collects the information you need.

Each tool serves a different purpose:

  • Requests and Beautiful Soup: Best for static websites where the required information is already available in the HTML.
  • Playwright: Useful for JavaScript-rendered pages and tasks involving browser interaction.
  • Scrapy: Designed for larger crawling projects involving multiple pages, pagination, and structured data exports.
  • Scraping APIs: Useful when browser rendering, proxies, and scraping infrastructure become difficult to manage locally.
  • pandas: Helps organize, clean, analyze, and export collected information.

A practical learning path is:

  1. Check whether an official API is available.
  2. Learn Requests and Beautiful Soup.
  3. Practice extracting and cleaning structured information.
  4. Explore Playwright when JavaScript rendering is required.
  5. Learn Scrapy when your projects involve larger crawls.
  6. Use an API when infrastructure becomes a significant part of the problem.

Start with small projects, build your understanding of HTML and HTTP requests, and gradually introduce more advanced tools as your requirements grow.

A reliable scraper doesn’t need to be complicated. It needs to collect the right information, handle errors properly, respect website rules, and remain easy to maintain.

Python Web Scraping FAQs

What is web scraping in Python?

Web scraping in Python means using code to retrieve information from websites and extract it into a structured format.

Common applications include:

  • Tracking product prices.
  • Collecting job listings.
  • Gathering article information.
  • Monitoring website changes.
  • Extracting tables and links.
  • Building datasets for analysis.

Instead of manually copying information, a Python script automates the collection and organization process.

Why use Python for web scraping?

Python is a popular choice because it offers a wide range of libraries for different scraping requirements.

Some of the most useful include:

  • Requests for downloading web pages.
  • Beautiful Soup and lxml for parsing HTML.
  • Playwright and Selenium for browser automation.
  • Scrapy for large crawling projects.
  • pandas for cleaning and exporting data.

You can begin with a small script and expand it into a larger data collection workflow without changing programming languages.

What is the difference between static and dynamic web scraping?

Static websites include their main content in the initial HTML response. You can usually extract this information using Requests and Beautiful Soup.

Dynamic websites may load additional content through JavaScript after the page opens.

If the information appears in your browser but is missing from response.text, you may need Playwright, Selenium, or a scraping API that supports JavaScript rendering.

Is web scraping with Python legal?

Web scraping is not automatically illegal, but its legality depends on the website, the information being collected, the method of access, and how the extracted data is used.

Before starting a project, review the website’s terms, available API options, access restrictions, and relevant data protection and copyright requirements.

Avoid collecting sensitive personal information, bypassing access controls, or republishing material without the necessary rights.

How can I avoid getting blocked while web scraping?

To reduce unnecessary blocking:

  • Use a clear User-Agent where appropriate.
  • Send requests at a reasonable rate.
  • Avoid aggressive concurrency.
  • Respect website rate limits.
  • Use limited retries with backoff.
  • Check for an official API.
  • Stop when repeated 403, 429, or CAPTCHA responses occur.

Proxies and scraping APIs may be appropriate for permitted use cases, but they should not be used to bypass access restrictions.

How can I make a Python web scraper faster?

Start by removing unnecessary work.

Useful techniques include:

  • Avoid downloading pages you do not need.
  • Reuse HTTP sessions.
  • Keep CSS selectors simple.
  • Extract only the required fields.
  • Cache pages during development.
  • Use Scrapy for larger crawling projects.
  • Consider asyncio and aiohttp when network waiting is the main bottleneck.

Asynchronous requests can help with collecting information from multiple static pages, but they do not execute JavaScript.

For dynamic pages, use an appropriate browser automation tool or scraping API.

Can Beautiful Soup scrape dynamic websites?

Beautiful Soup can parse HTML returned by a website, but it cannot execute JavaScript.

If a page loads information dynamically, you need to retrieve the rendered HTML through a browser automation tool such as Playwright or another suitable method before passing that HTML to Beautiful Soup.

What is the difference between web scraping and web crawling?

Web scraping focuses on extracting specific information from web pages.

Web crawling focuses on discovering and visiting pages, often by following links from one page to another.

A crawler can collect URLs across a website, while a scraper extracts the required information from those pages.

Many larger projects combine both techniques.

Can Python scrape data into Excel?

Yes. You can use Python libraries such as pandas to export extracted information into Excel-compatible formats.

For example:

import pandas as pd

df = pd.DataFrame(stories)

df.to_excel(
    "scraped_data.xlsx",
    index=False
)

This makes it easy to review, filter, and analyze collected data in Microsoft Excel.

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button