News Watch

What It Does

News Watch scrapes structured news data from 79 Indonesian and international news websites. Give it keywords and a date range, then get article data back with title, publish date, author, content, keyword, category, source, and link. Or use latest mode for continuous monitoring without keywords. Run source health checks with --health-report to verify scraper availability, or route all traffic through a residential proxy with --proxy when running on servers behind anti-bot blocks.

The v1.0 release freezes the public API under SemVer, so the documented 8-field output schema and CLI surface are now stable.

It replaces manual collection across Kompas, Detik, CNN Indonesia, AP News, Al Jazeera, and dozens of other sources with one CLI command or Python function.

How It Works

Query. Search by keyword and date, or use latest-news mode for monitoring.

Scrape. Playwright renders JavaScript-heavy pages and runs selected scrapers locally.

Output. Results can be returned as Python objects, a pandas DataFrame, CSV, XLSX, JSONL, or JSON.

Try It

pip install news-watch
playwright install chromium

CLI usage:

newswatch --keywords ihsg --start_date 2025-01-01

Python usage:

import newswatch as nw

df = nw.scrape_to_dataframe("teknologi,startup", "2025-01-01")
print(df[["source", "title"]].head())

Latest monitoring:

latest = nw.latest_to_dataframe(scrapers="antaranews,kompas,viva")
print(latest[["source", "title"]].head())

Releases

Current version is 1.2.5. See the changelog for the full history, which is generated from the repo rather than retyped here.

Sources are not only added. They get quarantined when a site starts blocking, and replaced when a publication shuts down, which is what the --health-report probes are for.

Supported Sources

Indonesian national and regional outlets plus international wires, currently 79, of which 74 support keyword search. Run newswatch --list_scrapers for the live list, or see the architecture notes for how the registry decides which scraper handles a source.

Real Use

Economic monitoring: collect news on IHSG, BI rate, inflation, and banks for market or policy tracking.

Policy monitoring: follow ministry, agency, or regulation topics across major sources.

NLP pipelines: feed article bodies into sentiment analysis, topic modeling, or LLM summaries.

Research datasets: build time-series datasets for media framing or event studies.