The Informational Diet of /pol/ — a factual audit of external sources cited on 4chan's political board.
What does the /pol/ community actually read and share? This project scrapes 4chan's Politically Incorrect board at regular intervals, extracts every external URL posted, classifies each source into a media category, and measures the sentiment of the surrounding discussion. The goal is to produce reproducible, quantitative evidence about the information ecosystem of one of the internet's most politically influential spaces.
What information does /pol/ consume?
Rather than asking what /pol/ "says", the project asks a prior, factual question: what sources does it cite? The board is treated as an informational environment, and the URLs posted are traces of that environment's media diet.
/pol/ cites alternative and social media sources more than mainstream legacy media.
The board is widely described as a space of distrust toward mainstream institutions. This hypothesis tests that claim quantitatively by comparing citation volumes across all source categories.
Sentiment toward mainstream media is more negative than toward alternative sources.
Using a RoBERTa sentiment model trained on noisy social-media text, the project scores the textual context around each cited link and compares mean sentiment across categories.
Source preferences vary systematically by thread topic.
Geopolitical threads, electoral discussions and identity debates may draw on different media diets. The project classifies threads by keyword and cross-tabulates source categories against topic.
A reproducible, fully local pipeline.
Scraping, parsing, classification and sentiment analysis all run locally, with no remote API dependency. Every result is traceable to its input data and parameters.
From thread to statistic.
The pipeline connects five modular steps, each of which can be run and inspected independently.
Scrape
4TCT collects all active threads on /pol/ via the 4chan JSON API and saves each thread as a JSON file. Data is organised by date: saves/YYYY_MM_DD/threads/pol/.
Merge
convert_4tct_to_jsonl.py flattens the per-thread JSON files into a single JSONL document (one post per line) per scrape day.
Classify
source_classifier.py extracts every external URL from the HTML com field, normalises the domain (strips www., aliases x.com to twitter, etc.), and classifies it into one of five categories.
Analyse
pipeline.py aggregates citations by domain and category, and scores the sentiment of each linking post using cardiffnlp/twitter-roberta-base-sentiment-latest.
Cross-tabulate
topic_source_matrix.py classifies each thread's OP into a topic (Geopolitics, Elections, Economy, Identity) via keyword matching and builds a topic × source-category matrix.
Five source categories.
Each cited domain is mapped to one of five categories via a transparent, dictionary-based lookup (see source_classifier.py). The classification is deterministic and editable.
Mainstream
CNN, NYT, BBC, Reuters, The Guardian, CBC, Bloomberg, Le Monde, Der Spiegel …
Alternative
Breitbart, Fox News, Epoch Times, Rebel News, Zero Hedge, Daily Wire, Newsmax …
State-funded
RT, Xinhua, CGTN, TASS, Sputnik, France 24 …
Social Media
Twitter/X, YouTube, Reddit, Telegram, TikTok, Rumble, Gab, Truth Social …
Institutional
Wikipedia, .gov, .mil, .edu domains, governmental agencies, academic institutions.
Six scripts, one pipeline.
Every script in the repository is self-contained and documented. All output is saved as JSON and CSV for inspection and reuse.
source_classifier.py
Two functions used by every downstream script: normalize_domain(url) and classify_source(domain). Contains the dictionary of known domains for all five categories.
pipeline.py
Extracts URLs, classifies them, runs RoBERTa sentiment analysis, and outputs aggregated stats (JSON) and per-post data (CSV). Accepts glob patterns to process multiple JSONL files at once.
topic_source_matrix.py
Classifies thread OPs into 5 topics (Geopolitics, Elections, Economy, Identity, Other) by keyword matching, then cross-tabulates cited source categories per topic. Outputs a table and a stacked bar chart.
viz.py
Generates bar charts, scatter plots, pie charts and comparison tables from the pipeline's stats output. Publication-ready, 200 dpi PNGs.
export_sheet.py
Produces a multi-sheet Excel file (.xlsx) with consolidated stats and per-post detail for offline analysis in Numbers or Excel.
convert_4tct_to_jsonl.py
Converts 4TCT's per-thread JSON files into flat JSONL, one line per post. Handles multiple date folders automatically.
What this version cannot do yet.
Volume
The current corpus is a single ~30-minute scrape (6 931 posts, 124 linking posts). Reliable statistics require multiple sessions. The scraper is designed to be run periodically and aggregate results.
Domain coverage
Many cited domains (file hosts, image boards, pastebins) fall into "Other" because they are not yet in the classification dictionaries. Coverage will improve as the corpus grows and the most frequent unknown domains are inspected and mapped.
Platform-specific URLs
4chan auto-links URLs as plain text, not as anchor tags, and inserts <wbr> word-break tags inside long URLs. Both are handled by the current parser, but edge cases may remain.
Canadian corpus comparison
The ultimate goal is to compare /pol/'s citation structure with a corpus of official Canadian political discourse (Hansard, strategic documents, press releases) processed through the same pipeline — making the informational fracture visible at a glance.