Why a plain request to a Toutiao link returns no text: robots.txt, share links and article addresses (Python)
Disclosure: written with AI assistance. The facts come from adding Toutiao (今日头条) to our tool, LinkDigest; all four code blocks below were re-run on 2026-11-21 and their output is pasted as printed. Standard library only
Disclosure: written with AI assistance. The facts come from adding Toutiao (今日头条) to our tool, LinkDigest; all four code blocks below were re-run on 2026-11-21 and their output is pasted as printed. Standard library only, only addresses robots.txt allows, with a pause between requests. Platforms change, so trust your own run over mine.
Short version: a plain request to a Toutiao article address returns a 72,914-byte page with 0 visible characters and no RENDER_DATA (the JSON that carries the article): the text only appears after a browser runs the page's scripts. That is why an AI assistant with web access says it "can't open" a 头条 link. Also, the two forms the app shares, /group/<id>/ and /item/<id>/, are disallowed by robots.txt; /article/<id>/ and /video/<id>/ are allowed.
The code prints Chinese labels; here is the key: 允许 = allowed, 不允许 = disallowed, 字节数 = bytes, 能看到的文字 = visible characters, 有没有 RENDER_DATA = RENDER_DATA present, 标题/作者/正文 = title/author/body.
1. Read robots.txt first
urllib.robotparser answers "may I fetch this address?". The script fetches robots.txt with its own User-Agent and hands the lines to parse():
import time
import urllib.request
from urllib import robotparser
UA = "robots-demo/1.0 (python urllib; tutorial script)"
ARTICLE_ID = "7709705250554266155"
VIDEO_ID = "7628244045329563689"
for host in ("https://www.toutiao.com", "https://m.toutiao.com"):
req = urllib.request.Request(host + "/robots.txt", headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=20) as r:
rp = robotparser.RobotFileParser()
rp.parse(r.read().decode("utf-8", "replace").splitlines())
for path in (f"/group/{ARTICLE_ID}/", f"/item/{ARTICLE_ID}/",
f"/article/{ARTICLE_ID}/", f"/video/{VIDEO_ID}/"):
verdict = "允许" if rp.can_fetch("*", host + path) else "不允许"
print(f"{host:<24} {path:<30} {verdict}")
time.sleep(2) # 两次请求之间停一下
Output:
https://www.toutiao.com /group/7709705250554266155/ 不允许
https://www.toutiao.com /item/7709705250554266155/ 不允许
https://www.toutiao.com /article/7709705250554266155/ 允许
https://www.toutiao.com /video/7628244045329563689/ 允许
https://m.toutiao.com /group/7709705250554266155/ 不允许
https://m.toutiao.com /item/7709705250554266155/ 不允许
https://m.toutiao.com /article/7709705250554266155/ 允许
https://m.toutiao.com /video/7628244045329563689/ 允许
Both hosts agree: the share forms /group/ and /item/ are disallowed; the same article under /article/<id>/ is allowed, and videos live at /video/<id>/, also allowed. So step one of reading Toutiao is taking the id out of the share address and using the /article/ address. The share address itself is never requested.
2. What one plain request to an allowed address returns
Check robots.txt, request once, and count what is left after removing scripts, styles and tags:
import html
import re
import time
import urllib.request
from urllib import robotparser
UA = "robots-demo/1.0 (python urllib; tutorial script)"
URL = "https://www.toutiao.com/article/7709705250554266155/"
req = urllib.request.Request("https://www.toutiao.com/robots.txt", headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=20) as r:
rp = robotparser.RobotFileParser()
rp.parse(r.read().decode("utf-8", "replace").splitlines())
assert rp.can_fetch("*", URL), "robots.txt 不允许的地址,不请求"
time.sleep(2)
req = urllib.request.Request(URL, headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=20) as r:
status, body = r.status, r.read()
page = body.decode("utf-8", "replace")
no_code = re.sub(r"(?is)<(script|style|noscript)\b.*?</\1\s*>", " ", page)
visible = "".join(html.unescape(re.sub(r"(?s)<[^>]+>", " ", no_code)).split())
print("HTTP", status)
print("字节数:", f"{len(body):,}")
print("能看到的文字:", len(visible), "个字符")
print("有没有 RENDER_DATA:", "RENDER_DATA" in page)
Output:
HTTP 200
字节数: 72,914
能看到的文字: 0 个字符
有没有 RENDER_DATA: False
Seventy-odd kilobytes and not one visible character, and no RENDER_DATA. The page needs a browser to run its scripts before the content shows up.
This post does not cover getting around that step. It does not touch the page's script check, signatures or cookies. That is brittle, and it goes against what the site asks for.
3. Share link → allowed address (a pure function)
A Toutiao article id is a long number (the function accepts 15 to 20 digits). The app's /group/<id>/ and /item/<id>/, the old /i<id>/ and /a<id>/, and /article/<id>/ all carry the same id. This function makes no request; it only handles strings:
import re
from urllib.parse import urlsplit
_ARTICLE = re.compile(r"/(?:group|item|article)/(\d{15,20})(?:[/?#]|$)|/[ai](\d{15,20})(?:[/?#]|$)")
_VIDEO = re.compile(r"/video/(\d{15,20})(?:[/?#]|$)")
def allowed_address(pasted: str) -> str | None:
"""粘贴进来的头条链接(可以带着分享文字)→ robots.txt 允许的地址;认不出就返回 None。"""
m = re.search(r"https?://[^\s,。!]+", pasted)
if not m:
return None
parts = urlsplit(m.group(0))
host = (parts.hostname or "").lower()
if host != "toutiao.com" and not host.endswith(".toutiao.com"):
return None
video = _VIDEO.search(parts.path)
if video:
return f"https://www.toutiao.com/video/{video.group(1)}/"
article = _ARTICLE.search(parts.path)
if article:
return f"https://www.toutiao.com/article/{article.group(1) or article.group(2)}/"
return None
CASES = [
"https://m.toutiao.com/group/7709705250554266155/?app=news_article",
"https://www.toutiao.com/item/7709705250554266155/",
"https://www.toutiao.com/article/7709705250554266155/?log_from=x",
"https://www.toutiao.com/i7709705250554266155/",
"https://www.toutiao.com/a7709705250554266155/",
"【标题】 https://m.toutiao.com/group/7709705250554266155/ 复制打开",
"https://www.toutiao.com/video/7628244045329563689/",
"https://m.toutiao.com/is/AbCd123/",
"https://www.toutiao.com/w/1834567890123456789/",
]
for case in CASES:
print(case)
print(" →", allowed_address(case))
Output:
https://m.toutiao.com/group/7709705250554266155/?app=news_article
→ https://www.toutiao.com/article/7709705250554266155/
https://www.toutiao.com/item/7709705250554266155/
→ https://www.toutiao.com/article/7709705250554266155/
https://www.toutiao.com/article/7709705250554266155/?log_from=x
→ https://www.toutiao.com/article/7709705250554266155/
https://www.toutiao.com/i7709705250554266155/
→ https://www.toutiao.com/article/7709705250554266155/
https://www.toutiao.com/a7709705250554266155/
→ https://www.toutiao.com/article/7709705250554266155/
【标题】 https://m.toutiao.com/group/7709705250554266155/ 复制打开
→ https://www.toutiao.com/article/7709705250554266155/
https://www.toutiao.com/video/7628244045329563689/
→ https://www.toutiao.com/video/7628244045329563689/
https://m.toutiao.com/is/AbCd123/
→ None
https://www.toutiao.com/w/1834567890123456789/
→ None
| Pasted | Becomes |
|---|---|
m.toutiao.com/group/<id>/?app=news_article |
www.toutiao.com/article/<id>/ |
www.toutiao.com/item/<id>/ |
www.toutiao.com/article/<id>/ |
www.toutiao.com/article/<id>/?log_from=x |
www.toutiao.com/article/<id>/ |
www.toutiao.com/i<id>/, /a<id>/ (old forms) |
www.toutiao.com/article/<id>/ |
| a whole share text with the title around the link | www.toutiao.com/article/<id>/ |
www.toutiao.com/video/<id>/ |
kept as is |
short link m.toutiao.com/is/…/
|
None: the id is only known after one redirect, out of scope here |
微头条 (micro post) www.toutiao.com/w/<id>/
|
None: not an article |
Notes:
- Only
toutiao.comand its subdomains are accepted; anything else returns None. - Query parameters (share source, tokens) are dropped; the address keeps only the id.
- Tested on 2026-11-20 with a fresh
/group/share link: after swapping it for/article/<id>/, the read returned the article that had been shared.
4. What RENDER_DATA looks like
After a browser has run an article page's scripts, the page holds a <script id="RENDER_DATA"> whose content is URL-encoded JSON with the title, author and body. Decoding takes two steps: urllib.parse.unquote, then json.loads. Here it is on a small made-up sample (a real page is far bigger, with many more fields):
import json
import re
from urllib.parse import quote, unquote
# 编出来的小样本,只为演示格式:真实页面里这段内容要浏览器把页面脚本跑完才会出现
made_up = {"data": {"title": "示例标题", "mediaInfo": {"name": "示例作者"},
"content": "<p>第一段正文。</p><p>第二段正文。</p>"}}
SAMPLE = ('<script id="RENDER_DATA" type="application/json">'
+ quote(json.dumps(made_up, ensure_ascii=False)) + "</script>")
print(SAMPLE[:96] + "…")
raw = re.search(r'<script id="RENDER_DATA"[^>]*>(.*?)</script>', SAMPLE, re.S).group(1)
data = json.loads(unquote(raw))["data"]
paragraphs = [p for p in re.split(r"</p>", data["content"]) if p.strip()]
print("标题:", data["title"])
print("作者:", data["mediaInfo"]["name"])
print("正文:", " / ".join(re.sub(r"<[^>]+>", "", p) for p in paragraphs))
Output:
<script id="RENDER_DATA" type="application/json">%7B%22data%22%3A%20%7B%22title%22%3A%20%22%E7%A…
标题: 示例标题
作者: 示例作者
正文: 第一段正文。 / 第二段正文。
To be clear: on a real page this content only exists after a browser has run the page's scripts. The plain request in section 2 does not get it. If you read Toutiao, open only the addresses robots.txt allows (/article/, /video/), one at a time, with a pause between requests.
Summary
- Read robots.txt first:
/group/and/item/are disallowed, while/article/and/video/are allowed. The rules are the same on www and m. - A plain request to an article address returns 72,914 bytes, 0 visible characters and no
RENDER_DATA. The text appears only after a browser runs the page's scripts. - The number in a share link is the article id: swap the share link for
/article/<id>/and never request the share address itself. -
RENDER_DATAis URL-encoded JSON:unquote, thenjson.loads.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.