ai newsSa., 24. Okt.
EilmeldungToken Beggar: oeffentliches Audit des Claude Code Plugins zur Kostenersparnis

Pull Public X (Twitter) Data in Python Without an X Developer Account, Then Hand It to Your AI Agent

I keep running into the same small job: "grab the last few hundred posts about a topic, plus the followers of a couple of accounts, and put them in a spreadsheet." It's not a big data project. It's an afternoon script. But X data has become awkward for exactly this size of task. Browser scrapers ne

dev.toSamstag 17:10 · 24.10.6 Min. Lesezeit
Bild: dev.to

I keep running into the same small job: "grab the last few hundred posts about a topic, plus the followers of a couple of accounts, and put them in a spreadsheet." It's not a big data project. It's an afternoon script. But X data has become awkward for exactly this size of task. Browser scrapers need logged-in accounts and proxies and break whenever the site changes, and the official API bills every post and profile it returns.

This post walks through a small Python script that does both jobs with a pay-as-you-go REST API, TwexAPI, and then shows how to give the same access to an AI agent through MCP. Everything here uses public data and endpoints that are documented in its API reference.

Disclosure and scope: TwexAPI is an independent third-party service. It is not affiliated with or endorsed by X Corp. Collect only public data, and check privacy law and X's terms for your own use case.

pip install requests export TWEXAPI_KEY = "twexapi_xxx" Enter fullscreen mode Exit fullscreen mode Authentication is a single header, Authorization: Bearer <your key>, sent to https://api.twexapi.io. No OAuth dance and no app review.

Both endpoints we'll use are POST endpoints with cursor paging. You send a JSON body, get back a page of results in data, and keep going while has_next_page is true, passing next_cursor back in.

import os import time import requests API = " https://api.twexapi.io " HEADERS = { " Authorization ": f " Bearer { os. environ [ ' TWEXAPI_KEY ' ] } " } def paginate ( path, body, max_items, delay = 5 ): """ POST a cursor endpoint until we have max_items or hit the last page. """ items, cursor = [], "" while len ( items ) < max_items: res = requests. post ( f " { API }{ path } ", headers = HEADERS, json = { ** body, " next_cursor ": cursor }, timeout = 30, ) res. raise_for_status () page = res. json () items += [ row for row in page. get ( " data ", []) if row ] cursor = page. get ( " next_cursor " ) if not page. get ( " has_next_page " ) or not cursor: break time. sleep ( delay ) return items [: max_items ] Enter fullscreen mode Exit fullscreen mode The delay=5 matters on the free tier. The Starter plan is limited to 0.2 requests per second, i.e. one request every five seconds.

Paid plans start at 5 QPS, so you can drop the delay once you upgrade.

POST /twitter/advanced_search/page takes a list of searchTerms and a sortBy of "Latest" or "Top". Each page returns up to 20 posts. The search terms accept the same operators as the X search box, so you can combine a keyword with lang:, since:, min_faves: and -filter:replies.

tweets = paginate ( " /twitter/advanced_search/page ", { " searchTerms ": [ " postgres ", " lang:en ", " min_faves:50 ", " since:2026-10-01 ", " -filter:replies " ], " sortBy ": " Latest ", }, max_items = 200, ) print ( len ( tweets ), " posts " ) Enter fullscreen mode Exit fullscreen mode A few operators I use constantly:

Filtering on the query side is the cheapest optimization you have. You pay per item returned, so a tighter query is both faster and cheaper than pulling everything and filtering in pandas.

POST /twitter/followers/page takes a screen_name and returns up to 200 followers per page. Because it uses the same cursor pattern, the helper works unchanged:

followers = paginate ( " /twitter/followers/page ", { " screen_name ": " nasa " }, max_items = 1000, ) Enter fullscreen mode Exit fullscreen mode There's a matching POST /twitter/following/page if you want who an account follows instead.

import csv with open ( " posts.csv ", " w ", newline = "", encoding = " utf-8-sig " ) as f: w = csv. writer ( f ) w. writerow ([ " tweet_id ", " created_at ", " user ", " likes ", " text " ]) for t in tweets: w. writerow ([ t. get ( " tweet_id " ), t. get ( " created_at " ), ( t. get ( " user " ) or {}). get ( " screen_name " ), t. get ( " favorite_count " ), ( t. get ( " text " ) or "" ). replace ( " \n ", " " ), ]) with open ( " followers.csv ", " w ", newline = "", encoding = " utf-8-sig " ) as f: w = csv. writer ( f ) w. writerow ([ " username ", " name ", " followers ", " location ", " bio " ]) for u in followers: w. writerow ([ u. get ( " username " ), u. get ( " name " ), u. get ( " followersCount " ), u. get ( " location " ), u. get ( " description " )]) Enter fullscreen mode Exit fullscreen mode utf-8-sig is a small trick worth remembering: it writes a byte-order mark so Excel opens emoji and non-Latin names correctly instead of showing mojibake.

200 posts plus 1,000 followers is 1,200 reads. At 5 Credits each that's 6,000 Credits, which fits comfortably inside the free 20,000. In dollar terms, 100,000 Credits is about $1, so the run is worth about six cents. Two billing details are useful to know when you're debugging: a request that comes back with no results still costs 5 Credits, and a request that fails with an error isn't billed. If you outgrow the trial, the pricing page lists one-time packs starting at $20 and monthly plans starting at $50.

Scripts are great for repeatable exports. For exploratory questions like "what are people in Japan posting about this launch?" it's often faster to let an agent figure out the calls. TwexAPI runs a hosted MCP server at https://api.twexapi.io/mcp (Streamable HTTP), authenticated with your API key in an x-api-key header.

For Cursor, add this to.cursor/mcp.json in a project (or ~/.cursor/mcp.json globally):

{ "mcpServers": { "twexapi": { "url": "https://api.twexapi.io/mcp", "headers": { "x-api-key": "twexapi_YOUR_KEY_HERE" } } } } Enter fullscreen mode Exit fullscreen mode Claude Code uses the same shape in.mcp.json with "type": "http" added. Claude Desktop only speaks stdio, so the docs bridge it with the mcp-remote npm package. The MCP setup docs have configs for Codex CLI, VS Code, Windsurf and others.

The server deliberately exposes just two tools:

So the agent first discovers the right endpoint, then calls it. I like this design because the tool list stays tiny (it doesn't flood the context window with dozens of tool definitions), and the read_only flag gives you something concrete to check before you let an agent run unattended.

A prompt that works well for a first test:

"Use twexapi to list the countries available for global trends, then show me the top trending posts for Japan."

Behind the scenes that maps to GET /twitter/global-trending/countries and GET /twitter/global-trending/tweets?country=....

For small and medium jobs, the pattern above covers most of what I need from X: a search, a follower export, and a CSV, in about 60 lines of Python. Then the same key can power an agent through MCP when the question is fuzzier. Start with a narrow query, check the Credits you used, and only then scale up.

Quelle: dev.to (dev.to). Original öffnen