Elasticsearch

1. Overview

Elasticsearch is a highly scalable open-source full-text search and analytics engine that allows you to store, search, and analyze big volumes of data quickly and in near real-time.

Through the Elasticsearch node in GoInsight, you can seamlessly integrate powerful search and data management capabilities into your automated workflows. You can achieve full lifecycle management of your indices and documents, including:

• Creating, retrieving, and deleting indices to manage your data structures.
• Adding, updating, and deleting individual documents within your indices.
• Fetching single or multiple documents and indices efficiently for downstream processing.

2. Prerequisites

Before using the Elasticsearch node, you need to ensure the following:

• You have a running and accessible Elasticsearch cluster (either self-hosted or Elastic Cloud).
• You have a valid account or API key with sufficient privileges to perform the desired operations (e.g., read/write access to specific indices).
• Your GoInsight environment has network access to your Elasticsearch instance.

3. Credentials

For detailed guidelines on how to obtain and configure credentials, please refer to our official documentation: Credentials Configuration Guide.

4. Supported Operations

Summary

This node primarily operates on two core resources within Elasticsearch: Document and Index. The table below summarizes all supported operations grouped by these resources.

Operation Details

Create a Document

Creates a new document in an Elasticsearch index

Input Parameters:

• IndexName: The name of the target index where the document will be created. Must be lowercase and can contain letters, numbers, and hyphens. Example: "user-data" or "products-2025"
• Document: The document content to be indexed. Provide as a JSON object with your custom fields. Common Fields: - name (string): User name - age (number): User age - email (string): Email address - created_at (string): Date in YYYY-MM-DD format Example: {"name": "John Doe", "age": 30, "email": "john@example.com"}

Options:

• DocumentId: The ID for the document. Behavior: - If empty: Elasticsearch auto-generates a unique ID (POST /_doc). This path is not idempotent; retrying may create duplicate documents. - If specified: Uses PUT /_create/{id}; request fails with 409 if the ID already exists. Prefer this path when you need safe retries. Example: "user-12345" or leave empty for auto-generation.
• Routing: ⚠️ Advanced Option: Custom routing value to control data distribution. Elasticsearch stores data across multiple servers (shards). By default, it auto-distributes based on document ID. Use this only if you need to: - Co-locate related documents on the same shard for faster queries - Implement custom partitioning logic Most users should leave this empty. Example: "user-123" (to group all documents for user-123 together)
• Refresh: Controls when the document becomes searchable after creation: - false (default): Document will be searchable within 1 second (recommended for most cases) - true: Document is immediately searchable (slower, use only if you need instant search) - wait_for: Wait until the document is searchable before returning (balanced option) Recommendation: Leave empty to use default behavior (false). Example: "false"
• Timeout: Request timeout for the Elasticsearch operation. If not specified or empty, defaults to 1m (1 minute). Format examples: - "30s" = 30 seconds - "1m" = 1 minute - "2m30s" = 2 minutes 30 seconds Example: "1m"
• Pipeline: ⚠️ Advanced Option: The name of a pre-configured ingest pipeline to transform/enrich the document before indexing. Use cases: - Add timestamps automatically - Extract fields from text - Validate data format Prerequisites: The pipeline must be created in Elasticsearch first (via Kibana Dev Tools or API). Most users should leave this empty. Example: "user-data-enrichment"

Output:

• DocumentId (string): The unique identifier of the created document. Use this ID to retrieve, update, or delete the document in subsequent operations. Example: "AXyZ1234567890"
• Index (string): The name of the index where the document was created.
• Version (number): The version number of the document. Starts at 1 for new documents and increments with each update.
• Result (string): The result of the operation. Value: "created" (New document was successfully created). If DocumentId is specified and already exists, the operation will fail.
• PrimaryTerm (number): The primary term of the document for optimistic concurrency control. Used with SeqNo to prevent conflicting updates.
• SeqNo (number): The sequence number of the document for optimistic concurrency control. Used with PrimaryTerm to prevent conflicting updates.
• Shards (object): Information about the shards involved in the operation. Contains: - total (number): Total number of shards targeted - successful (number): Number of shards that successfully processed the operation - failed (number): Number of shards that failed (0 means all succeeded) Example: {"total": 2, "successful": 1, "failed": 0}
• OriginalStatusCode (number): The original HTTP status code returned by the upstream API. Default 0 means the request did not reach upstream (e.g., timeout). Use for debugging.
• StatusCode (number): Status code semantics: -1 = local parameter validation failure; 200 = request reached Elasticsearch and business result is described by output fields/ErrorMessage (including upstream 4xx); 500 = network/timeout/system error or upstream 5xx.
• ErrorMessage (string): The detailed error information returned when the operation fails. If the Status Code is 200 and this field is not empty, it indicates a failure at the business logic level (such as document ID already exists, index does not exist, etc.). When successful, this field is empty.

Delete a Document

WARNING: This operation permanently deletes the document from Elasticsearch. The deletion cannot be undone. If you need to preserve history, consider using a soft delete approach (adding a 'deleted: true' field instead).

Input Parameters:

• IndexName: The name of the index containing the document. You can view all available indices using the List Indices action. Example: "products-2024"
• DocumentId: The unique identifier of the document to delete. You can obtain document IDs by using the Search Documents or Get Document action. Example: "prod_abc123def456"

Options:

• Routing: Custom routing value to ensure the delete operation is routed to the correct shard
• Refresh: Refresh policy controls when the deletion becomes visible to search: - true: Immediate refresh (use when you need to query immediately after deletion) - false (default): Async refresh (best performance) - wait_for: Wait for refresh to complete before returning (use in testing) Example: "wait_for"
• Timeout: Request timeout (default: 1m)
• IfSeqNo: Only perform the operation if the document has this sequence number (0 to disable)
• IfPrimaryTerm: Only perform the operation if the document has this primary term (0 to disable)
• Version: Explicit version number for concurrency control (0 to disable)
• VersionType: Version type for concurrency control: - internal: Use Elasticsearch's internal versioning (default) - external: Use external version numbers (e.g., from your database) - external_gte: Accept versions >= specified version Leave empty for most use cases.

Output:

• DocumentId (string): The ID of the deleted document.
• Index (string): The name of the index where the document was deleted from.
• Version (number): The version number of the document after deletion.
• Result (string): The result of the operation (deleted or not_found).
• PrimaryTerm (number): The primary term of the document for optimistic concurrency control.
• SeqNo (number): The sequence number of the document for optimistic concurrency control.
• Shards (object): Information about the shards involved in the operation. Contains: - total (number): Total number of shards - successful (number): Number of successful shard operations - failed (number): Number of failed shard operations
• OriginalStatusCode (number): The original HTTP status code returned by the upstream API. Default 0 means the request did not reach upstream (e.g., timeout). Use for debugging.
• StatusCode (number): Status code semantics: -1 = local parameter validation failure; 200 = request reached Elasticsearch and business result is described by output fields/ErrorMessage (including upstream 4xx); 500 = network/timeout/system error or upstream 5xx.
• ErrorMessage (string): Error description if the operation fails, empty string on success.

Get a Document

Retrieves a specified document from an Elasticsearch index by its ID. Use this action when you need to fetch a single document's full content. For searching multiple documents by criteria, use Search Documents instead. Common use cases: fetching user profile by user_id, retrieving order details by order_id.

Input Parameters:

• IndexName: The name of the index containing the document to retrieve (required)
• DocumentId: The ID of the document to retrieve (required). This ID is returned when you index a document using Index a Document action, or can be obtained via Search Documents action. Format can be a custom string (e.g., 'doc_12345') or UUID (e.g., '550e8400-e29b-41d4-a716-446655440000'). Example: "doc_12345"

Options:

• SourceIncludes: Comma-separated list of source fields to include in the response
• SourceExcludes: Comma-separated list of source fields to exclude from the response
• StoredFields: Comma-separated list of stored fields to return
• Routing: Custom routing value to ensure the get operation is routed to the correct shard. ⚠️ Advanced option: Only needed if you used custom routing when indexing the document. Most users should leave this empty. Example: "user_123"
• Preference: Preference value to control which shard replica executes the operation. ⚠️ Advanced option: Leave empty to use default load balancing. Common values: '_local' (prefer local shard), '_primary' (only primary shard). Example: "_local"
• Realtime: Whether to perform the operation in realtime (default: true)
• Refresh: Whether to refresh the shard before retrieving the document (default: false)
• Version: Only return the document if it has this version number (0 to disable)
• VersionType: Version type for optimistic concurrency control. Options: 'internal' (use Elasticsearch's internal version, default), 'external' (use external system's version number), 'external_gte' (external version >= current version). ⚠️ Advanced option: Leave empty unless you need version control. Example: "internal"

Output:

• DocumentId (string): The ID of the retrieved document.
• Index (string): The name of the index where the document is stored.
• Found (bool): Whether the document was found.
• Version (number): The version number of the document.
• PrimaryTerm (number): The primary term of the document for optimistic concurrency control.
• SeqNo (number): The sequence number of the document for optimistic concurrency control.
• Source (object): The document content (structure depends on the index mapping). This object contains all fields stored in the document, such as 'title', 'content', 'timestamp', etc. The actual fields vary based on your index schema. Example: {"title": "Sample", "content": "...", "timestamp": "2024-01-15T10:30:00Z"}
• OriginalStatusCode (number): The original HTTP status code returned by the upstream API. Default 0 means the request did not reach upstream (e.g., timeout). Use for debugging.
• StatusCode (number): Status code semantics: -1 = local parameter validation failure; 200 = request reached Elasticsearch and business result is described by output fields/ErrorMessage (including upstream 4xx); 500 = network/timeout/system error or upstream 5xx.
• ErrorMessage (string): Error description if the operation fails, empty string on success.

Get Many Documents

Retrieves multiple documents from an Elasticsearch index using the Multi Get API

Input Parameters:

• IndexName: The name of the index containing the documents to retrieve (required)
• DocumentIds: Array of document IDs to retrieve. Example: ["doc-001", "doc-002", "doc-003"]

Options:

• SourceIncludes: Comma-separated list of source fields to include in the response
• SourceExcludes: Comma-separated list of source fields to exclude from the response
• StoredFields: Advanced Option: Comma-separated list of stored fields to return. Note: Stored fields are different from _source fields and must be explicitly configured in your index mapping. Most users should use SourceIncludes instead.
• Routing: Advanced Option: Custom routing value to ensure the get operation is routed to the correct shard. Only needed if you use custom routing in your index. Most users should leave this empty.
• Preference: Advanced Option: Preference value to control which shard replica executes the operation (e.g., '_local' to prefer local shards). Most users should leave this empty.
• Realtime: Whether to perform the operation in realtime. Set to false to improve performance if you can tolerate reading slightly stale data (data not yet refreshed to disk). Default: true.
• Refresh: Whether to refresh the shard before retrieving documents. Set to true to ensure you read the absolute latest data, but this will impact performance. Only use when immediate consistency is critical. Default: false.

Output:

• Documents (object-array): Array of document objects returned by _mget. Each object contains DocumentId, Index, Found, Version, Source, and optional Error fields.
• TotalRequested (number): The total number of documents requested.
• TotalFound (number): The number of documents that were found.
• TotalNotFound (number): The number of documents that were not found.
• OriginalStatusCode (number): The original HTTP status code returned by the upstream API. Default 0 means the request did not reach upstream (e.g., timeout). Use for debugging.
• StatusCode (number): Status code semantics: -1 = local parameter validation failure; 200 = request reached Elasticsearch and business result is described by output fields/ErrorMessage (including upstream 4xx); 500 = network/timeout/system error or upstream 5xx.
• ErrorMessage (string): Error description if the operation fails, empty string on success.

Update a Document

Updates an existing document in an Elasticsearch index

Input Parameters:

• IndexName: The name of the index containing the document to update (required)
• DocumentId: The ID of the document to update (required)

Options:

• Document: Document fields to update. Optional when Script is provided. At least one of Document or Script must be provided.
• Upsert: The document to insert if it does not exist, in JSON format
• DocAsUpsert: If true, uses the Document as the upsert document when the document does not exist
• Script: Painless script to execute for the update
• ScriptParams: Parameters for the script in JSON format
• AdditionalOptions: Advanced options (object). Supported keys: routing, refresh, timeout, retry_on_conflict, if_seq_no, if_primary_term. Example: {"refresh": "wait_for", "retry_on_conflict": 3, "timeout": "30s"}

Output:

• DocumentId (string): The ID of the updated document.
• Index (string): The name of the index where the document is stored.
• Version (number): The new version number of the document after the update.
• Result (string): The result of the operation (updated, noop, or created).
• PrimaryTerm (number): The primary term of the document for optimistic concurrency control.
• SeqNo (number): The sequence number of the document for optimistic concurrency control.
• Shards (object): Information about the shards involved in the operation.
• OriginalStatusCode (number): The original HTTP status code returned by Elasticsearch API. Default 0 means the request did not reach upstream (e.g., timeout). Use for debugging.
• StatusCode (number): Status code semantics: -1 = local parameter validation failure; 200 = request reached Elasticsearch and business result is described by output fields/ErrorMessage (including upstream 4xx); 500 = network/timeout/system error or upstream 5xx.
• ErrorMessage (string): Error description if the operation fails, empty string on success.

Create an Index

Creates a new index in Elasticsearch with optional settings and mappings

Input Parameters:

• IndexName: The name of the index to create. Must be lowercase, cannot start with underscore (_) or hyphen (-). Can contain letters, numbers, and hyphens. Example: "user-data" or "products-2025"

Options:

• Settings: Index settings configuration (JSON object). Controls shard allocation, refresh intervals, and other index-level behaviors. Common Fields: - number_of_shards (number): Number of primary shards (default 1). Higher = better write throughput. Recommended: 1-5 for most use cases. - number_of_replicas (number): Number of replica copies (default 1). Higher = better read availability and fault tolerance. Recommended: 1-2. - refresh_interval (string): How often to make new data searchable, e.g., "30s", "1m" (default "1s"). Increase for better indexing performance. Example: ``json { "number_of_shards": 3, "number_of_replicas": 2, "refresh_interval": "30s" } `` Note: Leave empty to use Elasticsearch defaults (1 shard, 1 replica).
• Mappings: Field mappings configuration (JSON object). Defines the data types and indexing behavior for each field in your documents. Common Field Types: - text: Full-text search (analyzed, for search boxes). Example: product descriptions - keyword: Exact match (not analyzed, for filters/aggregations). Example: status codes, tags - long/integer/float: Numeric values. Example: prices, quantities - date: Date/time values (ISO 8601 format). Example: created_at, updated_at - boolean: true/false values - object: Nested JSON object Example: ``json { "properties": { "title": {"type": "text"}, "status": {"type": "keyword"}, "created_at": {"type": "date"}, "price": {"type": "float"} } } `` Note: If not specified, Elasticsearch will auto-detect field types (dynamic mapping).
• Aliases: Index aliases configuration (JSON object). Aliases are alternate names for your index, enabling flexible querying and zero-downtime reindexing. Use Cases: 1. Multiple names: Point "products" and "items" to the same index 2. Zero-downtime reindexing: Create new index, swap alias when ready 3. Simplified queries: Use "current-logs" alias instead of "logs-2025-01-15" Example: ``json { "my_search_alias": {}, "current_data": {} } ` Note: Each key is an alias name. The empty object {}` means no filtering (alias points to all documents).
• Timeout: Request timeout for the index creation operation. If not specified, defaults to 1m (1 minute). Supported time units: - ms: milliseconds (e.g., "500ms") - s: seconds (e.g., "30s") - m: minutes (e.g., "1m", "2m") - h: hours (e.g., "1h") Example: "1m"
• MasterTimeout: Timeout for connecting to the master node (the coordinator node in the Elasticsearch cluster). If not specified, defaults to 30s. What is master node? The master node manages cluster-wide settings and index creation. If the master is busy or unreachable, this timeout determines how long to wait. Supported time units: ms, s, m, h Example: "30s"
• WaitForActiveShards: Wait for specified number of active shard copies before returning success. What are active shards? Elasticsearch stores data across multiple copies (shards) for redundancy. This setting ensures a minimum number of copies are ready before the operation completes. Common values: - "" (empty): Don't wait, return immediately (default) - "1": Wait for primary shard only - "all": Wait for all shards (primary + all replicas) - safest but slowest - "2", "3", etc.: Wait for specific number of shard copies Recommendation: Use "1" for most cases, "all" for critical data.

Output:

• Acknowledged (bool): Whether the index creation request was acknowledged by the cluster. - true: Index creation was accepted by the cluster - false: Cluster did not acknowledge within timeout. Possible causes: cluster is overloaded, master node unreachable, or invalid configuration. Check ErrorMessage for details.
• ShardsAcknowledged (bool): Whether the required number of shard copies were started before the timeout. - true: All required shards are active and ready - false: Shards did not start in time. The index may still be created, but data operations might be delayed. This often happens when WaitForActiveShards is set to a high value or the cluster is under heavy load.
• Index (string): The name of the created index. Use this to reference the index in subsequent operations (e.g., Create Document, Search).
• OriginalStatusCode (number): The original HTTP status code returned by the upstream API. Default 0 means the request did not reach upstream (e.g., timeout). Use for debugging.
• StatusCode (number): Status code semantics: -1 = local parameter validation failure; 200 = request reached Elasticsearch and business result is described by output fields/ErrorMessage (including upstream 4xx); 500 = network/timeout/system error or upstream 5xx.
• ErrorMessage (string): Detailed error message if the operation fails. Empty string on success. Common errors include: "Index already exists", "Invalid JSON format", "Unauthorized", "Connection timeout".

Delete an Index

Deletes a single, specific index from Elasticsearch.

Input Parameters:

• IndexName: The name of the index to delete (required). For safety, wildcard patterns and comma-separated multiple indices are not allowed.

Options:

• Timeout: Request timeout (default: 1m)
• MasterTimeout: Timeout for connecting to the master node (default: 30s)
• IgnoreUnavailable: If true, missing indices will not cause an error (default: false)

Output:

• Acknowledged (bool): Whether the index deletion request was acknowledged by the cluster.
• OriginalStatusCode (number): The original HTTP status code returned by the Elasticsearch API. Default 0 means the request did not reach upstream (e.g., timeout). Use for debugging.
• StatusCode (number): Status code semantics: -1 = local parameter validation failure; 200 = request reached Elasticsearch and business result is described by output fields/ErrorMessage (including upstream 4xx); 500 = network/timeout/system error or upstream 5xx.
• ErrorMessage (string): Error description if the operation fails, empty string on success.

Get an Index

Retrieves detailed information about an Elasticsearch index including settings, mappings, and aliases

Input Parameters:

• IndexName: The name of the index to retrieve. Supports wildcards and comma-separated names (required)

Options:

• IncludeDefaults: Whether to include default settings in the response (default: false)
• FlatSettings: Whether to return settings in a flat format (default: false)
• Local: Whether to retrieve information from the local node only (default: false)
• MasterTimeout: Timeout for connecting to the master node (default: 30s)
• Features: Comma-separated list of features to return: aliases, mappings, settings

Output:

• IndexName (string): The name of the index.
• Aliases (object): The aliases associated with the index.
• Mappings (object): The field mappings for the index.
• Settings (object): The index settings including shards, replicas, and other configurations.
• Exists (bool): Whether the index exists.
• OriginalStatusCode (number): The original HTTP status code returned by the upstream API. Default 0 means the request did not reach upstream (e.g., timeout). Use for debugging.
• StatusCode (number): Status code semantics: -1 = local parameter validation failure; 200 = request reached Elasticsearch and business result is described by output fields/ErrorMessage (including upstream 4xx); 500 = network/timeout/system error or upstream 5xx.
• ErrorMessage (string): Error description if the operation fails, empty string on success.

Get Many Indices

Retrieve information from multiple indexes in Elasticsearch. Note: The paging function of this tool is completed locally, not on the Elasticsearch server side. This means that when querying a large number of indexes, all matching index data will still be obtained from Elasticsearch, and then sliced internally in this tool to return the specified page. Therefore, in large-scale clusters, please use the '*' indexing mode with caution to avoid potential performance issues and memory consumption.

Input Parameters:

• IndexPattern: Pattern to match index names. Supports wildcards. Required for safety (do not use in large clusters). Example: "logs-"

Options:

• Health: Filter by health status: green, yellow, red
• ExpandWildcards: How to expand wildcard patterns. Most users should use default 'open'. - open: Match open indices - closed: Match closed indices - hidden: Match hidden indices (starting with dot) - all: Match all indices - none: Do not expand wildcards
• IncludeHidden: Whether to include hidden indices starting with dot (default: false)
• MasterTimeout: Timeout for connecting to the master node (default: 30s)
• Page: The page number to be retrieved. Note: This page is implemented internally within the tool and is not a server-side page for Elasticsearch. The default value is 1.
• PageSize: The number of indexes returned per page. The maximum value is 200. Note: This page is implemented internally within the tool and is not a server-side page for Elasticsearch. The default value is 50,max: 200.

Output:

• Indices (object-array): Array of index objects. Each object contains: - IndexName (string): Index name - Health (string): Cluster health status (green/yellow/red) - Status (string): Index status (open/close) - Uuid (string): Unique identifier for the index - PrimaryShards (number): Number of primary shards - ReplicaShards (number): Number of replica shards - DocsCount (number): Total number of documents - DocsDeleted (number): Number of deleted documents (not yet purged) - StoreSize (string): Total disk space used (e.g., "2.5gb") - PrimaryStoreSize (string): Disk space used by primary shards (e.g., "1.3gb") - CreationDate (string): Index creation timestamp (ISO 8601 format)
• TotalCount (number): Total number of indices returned.
• HasMore (bool): Whether more pages are available.
• NextPage (number): Next page number when HasMore is true; otherwise 0.
• OriginalStatusCode (number): The original HTTP status code returned by Elasticsearch API. Default 0 means the request did not reach upstream (e.g., timeout). Use for debugging.
• StatusCode (number): Status code semantics: -1 = local parameter validation failure; 200 = request reached Elasticsearch and business result is described by output fields/ErrorMessage (including upstream 4xx); 500 = network/timeout/system error or upstream 5xx.
• ErrorMessage (string): Error description if the operation fails, empty string on success.

5. Example Usage

This section will guide you through creating a simple workflow to create a new document in your Elasticsearch index.

Workflow Overview: Start -> Elasticsearch -> Answer

Step-by-Step Guide:

1. Add the Tool Node:
   • On the workflow canvas, click the "+" icon to add a new node.
   • Select the "Tools" tab in the pop-up panel.
   • Find and select Elasticsearch from the list of tools.
   • In the supported operations list for Elasticsearch, click on Create a Document. This will add the corresponding node to your canvas.
2. Configure the Node:
   • Click on the newly added Create a Document node to open its configuration panel on the right.
   • Credentials: At the top of the panel, locate the credentials field. Click the dropdown menu and select your pre-configured Elasticsearch credentials.
   • Parameter Configuration:
   • IndexName: Enter the name of the target index where you want to store the document, for example, "customer-data".
   • Document: Provide the JSON object containing the data you want to index. For example, enter {"name": "Alice", "email": "alice@example.com", "status": "active"}.
3. Run and Verify:
   • Once all required parameters are correctly filled, the error indicator on the top right of the node will disappear.
   • Click the "Test Run" button at the top right of the canvas to execute the workflow.
   • After a successful execution, you can click the log icon to view the detailed input and output of the node, verifying that your document was successfully created and noting its generated DocumentId.

6. FAQs

Q: Why am I getting a 401 Unauthorized or 403 Forbidden error?

A: Please check the following points:

• Ensure your API key or username/password credentials are correct.
• Verify that the user associated with the credentials has the necessary privileges (e.g., write or create_index permissions) for the target index.

Q: Why does the "Create a Document" operation fail with a 409 Conflict error?

A: This happens when you specify a DocumentId that already exists in the target index.

• If you want to update the existing document, use the Update a Document action instead.
• If you want Elasticsearch to auto-generate a unique ID, leave the DocumentId field empty.

Q: Why am I getting an "Invalid JSON format" error when providing the Document or Settings?

A: Please check the following points:

• Ensure that your input is a valid JSON object.
• Make sure all keys and string values are enclosed in double quotes ("), not single quotes (').

7. Official Documentation

Elasticsearch Official API Documentation