Overview
Sifting through sitemap.xml files manually to gather all page URLs is tedious and error-prone. This workflow automates the extraction process by detecting sitemap indexes or direct URL lists, fetching sub-sitemaps when needed, and deduplicating URLs. It outputs a clean, comprehensive URL list ready for SEO auditing, crawler seeding, or content migration prep.
The Impact
- Extract Complete URL Sets. Automatically retrieve every page URL from complex sitemap structures.
- Eliminate Duplicates. Deliver a clean list by removing redundant URLs while preserving order.
- Save Manual Hours. Remove the need for hand-maintaining or crawling sitemap indexes and sub-sitemaps.
- Provide Actionable Stats. Output total URLs, duplicates removed, and sitemap counts for analysis.
Who This Is For
- SEO Specialists preparing comprehensive URL lists for site audits and crawler inputs.
- Developers automating seed URL generation for web crawlers and bots.
- Content Teams compiling exhaustive page inventories for migration or backup checklists.
- Data Researchers scraping full site URL datasets from sitemaps efficiently.
How It Works
- Fetch Main Sitemap
- Request sitemap.xml from the domain root to get the main sitemap content.
- Parse and Identify Sitemap Type
- Analyze the XML to determine if it's a sitemap index or a direct URL list, extracting sub-sitemaps or URLs accordingly.
- Expand Sub-Sitemaps or Process URLs
- If an index, loop through each sub-sitemap, fetch and parse URLs; if direct URLs, proceed to processing them.
- Aggregate and Deduplicate
- Combine all collected URLs, remove duplicates preserving order, and compile statistical data.
What You'll Need
Before using this template, make sure you have:
- The root domain URL of the target website, including protocol and trailing slash (e.g., https://example.com/).
- Network access to fetch the sitemap.xml files from the target domain.
How to Use
- Step 1. Enter Domain
- Step 2. Run Workflow
- Step 3. Process Sitemap Type
- Step 4. Expand Sub-Sitemaps if Needed
- Step 5. Verify Results
Input the full root domain URL of the website, including protocol and a trailing slash.
Start the automation to fetch and parse the main sitemap.xml automatically.
The workflow detects if it’s a sitemap index or direct URL list and branches accordingly.
If a sitemap index, sub-sitemaps are fetched and parsed for URLs; otherwise URLs are processed directly.
Check the final aggregated URL list and statistics for completeness and duplicates removed.
