Web Scraping with Python: Collect Web Data Automatically, No More Manual Copying

Hong Kong SMEs doing market research, comparing supplier prices or tracking industry news usually open each page by hand and copy the data into Excel. Eight or ten entries is fine, but a hundred — updated every single day — is a huge time sink and a recipe for typos.

The technique for automatically pulling data out of web pages is called web scraping. With Python and two free libraries, a dozen lines of code extract every title, price and link from a page into a spreadsheet. Even if you've never written code before, you can follow along.

Step 1: Install two libraries

We'll use requests to download the page and BeautifulSoup to parse and extract the data. Open Command Prompt and run:

pip install requests beautifulsoup4

When you see Successfully installed, you're ready.

Step 2: What does the script do?

This script does three things for you in one go:

Download the page — requests pulls down the page's raw HTML
Find the data — BeautifulSoup pinpoints the elements you want (e.g. each headline) among thousands of lines of HTML
Save to CSV — writes the results into a file Excel can open

Step 3: Write the script

Create a new file called scrape.py (Notepad is fine) and paste in the code below:

import requests
from bs4 import BeautifulSoup
import csv

URL = "https://news.ycombinator.com/"

resp = requests.get(URL, timeout=10)
soup = BeautifulSoup(resp.text, "html.parser")

rows = []
for item in soup.select("span.titleline a"):
    title = item.get_text()
    link = item.get("href", "")
    rows.append([title, link])

with open("headlines.csv", "w", newline="", encoding="utf-8-sig") as f:
    writer = csv.writer(f)
    writer.writerow(["Title", "Link"])
    writer.writerows(rows[:20])

print("Done! Scraped", len(rows), "headlines")

Line by line

import requests / from bs4 import BeautifulSoup / import csv — loads three tools: download pages, parse pages, and write CSV.
URL = "https://news.ycombinator.com/" — the target page; change this to whichever site you want to scrape.
resp = requests.get(URL, timeout=10) — downloads the page; resp.text is the raw HTML.
soup = BeautifulSoup(resp.text, "html.parser") — turns the HTML into a searchable tree.
soup.select("span.titleline a") — uses a CSS selector to grab every headline link in one go.
item.get_text() — gets the title text; item.get("href", "") — gets the link.
writer.writerow(...) — writes the first 20 rows to CSV; encoding="utf-8-sig" keeps Chinese readable in Excel.

Step 4: Run it

In Command Prompt, navigate to where you saved scrape.py and run:

python scrape.py

Done! A headlines.csv file appears in the same folder — open it in Excel and all the titles and links are there, no manual copying needed.

Three rules of scraping etiquette

Scraping is powerful, but use it responsibly or you'll get your IP blocked:

Check robots.txt — most sites have a robots.txt file listing what you're not allowed to scrape; respect it before you start.
Don't hammer the server — leave a short delay between requests, one or two per second. Bombarding a site slows it down and gets you treated as an attack.
Only take public data — don't scrape content behind a login, or anything involving personal privacy.

Summary

Python Tutorial: With a dozen lines of code you hand the "open each page, copy into Excel" grunt work to the computer, so market research, price comparison and industry news are collected automatically. Using the same idea, just swap the target URL and selectors to grab any public data you need.