Overview
Manually extracting and archiving website content from sitemaps wastes time and invites errors. This workflow automates sitemap scraping, filters URLs by rules, crawls page content, and saves each page as a text file in Google Drive—streamlining backup, audit, and content retrieval.
The Impact
- Slash manual scraping hours. Automate batch extraction and storage of filtered sitemap pages.
- Ensure consistent backups. Archive relevant pages automatically for audit and offline access.
- Track competitor content. Grab pages with "agent" or "tool" keywords for monitoring changes.
- Build text corpora fast. Export large volumes of filtered page content for analysis or model training.
Who This Is For
- Website Administrators needing periodic content backups and audits.
- Product & Operations Staff monitoring competitor pages with specific keywords.
- Data Engineers building text datasets from website content.
- SEO Specialists extracting targeted URLs for analysis.
How It Works
- Fetch Sitemap XML
- Send an HTTP GET request to the configured sitemap URL to retrieve the raw XML content.
- Parse and Filter URLs
- Convert the XML to URL objects and apply filters to keep URLs from the root domain or containing "agent" or "tool".
- Loop Over Filtered URLs
- Iterate through each filtered URL to process individually.
- Crawl Web Page Content
- Extract the page's Title and Content using the crawl tool.
- Save Text to Google Drive
- Save the scraped content as a text file named by combining the URL and page title.
What You'll Need
Before using this template, make sure you have:
- Google Drive credentials authorized for file creation (required for saving scraped content).
- A valid sitemap XML URL pointing to the website's top-level sitemap or index sitemap if multiple levels exist.
How to Use
- Step 1. Configure Credentials
- Step 2. Set Sitemap URL
- Step 3. Run the Workflow
- Step 4. Monitor Progress
- Step 5. Verify Results
Input your Google Drive credentials in the "Save Text To Google Drive" node to enable file saving.
Enter the full sitemap XML URL in the "WebSiteMapURL" parameter, replacing the default if scraping a different site.
Start the automation to fetch, filter, crawl, and save page contents in batch.
Observe the loop processing each URL, crawling pages, and creating text files in Google Drive.
Check your Google Drive for the newly created text files matching the filtered pages to confirm success.
