Overview
This automation eliminates the tedious manual parsing of Screaming Frog exports by filtering and converting site crawl CSVs into a ready-to-use Markdown text file. It isolates only HTTP 200, indexable HTML pages and compiles their URLs, titles, and meta descriptions into a clean summary for SEO review or large language model input.
The Impact
- Filter out noise. Automatically exclude non-200 status, non-indexable, and non-HTML pages.
- Save hours. Replace manual CSV parsing with instant Markdown list generation.
- Standardize data. Harmonize multilingual Screaming Frog headers into uniform outputs.
- Fuel LLM workflows. Deliver structured, clean text for batch processing and model training.
Who This Is For
- SEO Analysts auditing title and meta description coverage at scale.
- Content Managers archiving and quality-checking website page summaries.
- Data Engineers preparing clean, structured text data for language model training.
- Digital Marketers streamlining site audit documentation and reporting.
How It Works
- Download and Parse CSV
- Retrieve the Screaming Frog internal_html.csv file from the provided URL and parse its multilingual headers.
- Filter Valid Pages
- Keep only rows with HTTP status 200, marked as indexable, and content type containing text/html.
- Extract Key Fields
- Pull URL, page title (fallback to URL if missing), and meta description for each valid page.
- Format as Markdown List
- Generate markdown lines with format '- [title](url): description' and compile them into a single text block.
- Add Header and Output
- Prefix the content with the website name and description as markdown header and blockquote, then output as llms_txt_content.
What You'll Need
Before using this template, make sure you have:
- An exported Screaming Frog internal_html.csv file containing key columns like Address, Title, Meta Description, Status Code, Indexability, and Content Type.The CSV file hosted at a publicly accessible URL for the workflow to download.Basic information about the website: its name and a short descriptive summary.
How to Use
- Step 1. Prepare CSV Export
- Step 2. Upload and Obtain URL
- Step 3. Configure Template Parameters
- Step 4. Run the Workflow
- Step 5. Verify Output
Export the internal_html.csv from Screaming Frog with standard crawl data including URLs, titles, and metadata.
Host the CSV file on a server or cloud storage where it can be accessed via a direct link.
Fill in WebsiteName, WebsiteDescription, and CsvFileUrl fields in the workflow start node.
Trigger execution to download, filter, and transform the CSV into the llms.txt output.
Check the generated llms_txt_content for a clean Markdown summary reflecting only valid, indexable HTML pages.
