Overview
Google News RSS feeds provide articles via redirect links that complicate scraping and archiving. This workflow cuts through the noise by fetching the RSS feed, extracting hidden tokens from article pages, and calling Google's batch API to recover original, direct article URLs—eliminating redirects and saving you from redundant parsing.
The Impact
- Cut Redirect Delays. Access original article URLs directly without intermediate redirects.
- Slash Parsing Workload. Automate signature extraction and link resolution to avoid repeated manual steps.
- Batch Process Efficiently. Limit the number of news items fetched and decoded as per your needs.
- Ensure Reliable Targets. Get stable URLs for scraping, archiving, or monitoring without guesswork.
Who This Is For
- News Aggregators restoring real article URLs to scrape full content and summaries uniformly.
- Archivists & Researchers batch-collecting authentic links for backup and permanent storage.
- Media Monitors converting RSS items into target URLs for detailed content analysis and indexing.
- Data Engineers automating link extraction workflows to feed downstream pipelines seamlessly.
How It Works
- Fetch and Convert RSS Feed
- Retrieve Google News RSS XML and convert it to JSON for structured processing.
- Extract Link and GUID
- Loop over news items to read each RSS entry’s redirect link and its GUID identifier.
- Parse Hidden Signature & Timestamp
- Fetch the intermediate article page and extract hidden signature and timestamp tokens from HTML.
- Resolve Direct Article URL
- Use the extracted tokens with GUID to call Google’s batch API and retrieve the original article URL.
- Aggregate Results
- Append each resolved original URL into an array and output this collection after processing completes.
What You'll Need
Before using this template, make sure you have:
- Access to Google News RSS feeds (no special credentials needed).
- Internet connectivity to fetch RSS feeds, article pages, and call Google's batch interface.
- Basic setup to run HTTP requests and execute Python code within the workflow environment.
How to Use
- Step 1. Set NewsLimit Parameter
- Step 2. Trigger Workflow Start
- Step 3. Process Each News Item
- Step 4. Collect Original URLs
- Step 5. Verify Output
Define how many news items to process from the RSS feed, default is 3 for quick tests or increase for batch jobs.
Initiate the workflow to fetch and convert the RSS feed into a workable JSON structure.
The workflow loops through each news entry, extracting links, fetching pages, parsing tokens, and resolving original URLs.
Each resolved direct URL is appended into a result array for consolidated output.
Check the output array named NewsOriginalUrl to ensure all original article links have been successfully restored.
