Overview
Manually fetching complete launch datasets from paginated APIs is tedious and error-prone. This workflow automates requests to the SpaceX API, reads pagination metadata, iterates through pages up to a set limit, extracts structured launch records, and consolidates them into a single array—streamlining data collection for archiving or analysis.
The Impact
- Eliminate manual pagination. Automatically gather multi-page launch data without user intervention.
- Consolidate scattered data. Aggregate all launch records into one structured array ready for analysis.
- Control data volume. Limit requests by page count and item size to optimize performance.
- Accelerate workflows. Cut down hours of manual API querying to seconds of automated execution.
Who This Is For
- Researchers Needing periodic archives of SpaceX launches for trend studies.
- Data Engineers Who require clean, paginated launch data for ETL pipelines or warehouses.
- Product Teams Building timelines or news summaries based on launch events.
- Analysts Preparing datasets for visualization or detailed exploration.
How It Works
- Build Initial Request and Query Pagination
- Construct the first request with page=1 and user-defined limit, then fetch total page count from the API.
- Extract and Convert Page Numbers
- Use an LLM node to parse current and total pages, then convert these strings into numbers for control flow.
- Initialize Result Array and Loop
- Create an empty array to hold results, then loop through pages up to the smaller of total pages or PageCount.
- Fetch and Extract Page Data
- Build request for each page, send to API, extract structured launch entries using LLM, and append to results.
- Output Aggregated Results
- After completing all pages, return the compiled array of launch data as the workflow's final output.
What You'll Need
Before using this template, make sure you have:
- Access to the official SpaceX API endpoint at https://api.spacexdata.com/v4/launches/query
- Ability to configure numeric parameters for pagination: Limit (items per page) and PageCount (max pages)
- An environment supporting HTTP POST requests with JSON payloads and handling JSON responses
- LLM integration for parsing JSON responses and extracting needed fields
How to Use
- Step 1. Set Parameters
- Step 2. Run Initial Query
- Step 3. Automatic Pagination Loop
- Step 4. Aggregate Data
- Step 5. Verify Results
Define the number of items per page with Limit and the maximum pages to fetch with PageCount.
Trigger the workflow to send the first request and retrieve pagination info from the SpaceX API.
The workflow loops through each page, fetching and extracting launch data until the page limit or PageCount is reached.
Each page's extracted launch documents append to a growing result array automatically.
Check the final output array at the End node to ensure all requested pages' data is consolidated correctly.
