Fetching run logs periodically for analysis

Use the monitoring endpoints to fetch run logs of a given period.

The /executions/{runId}/logs endpoint returns a response based on log size:

• 200 (OK): Complete log retrieved in a single response.
• 206 (Partial Content): Log exceeds size limits; only the last 100,000 records are returned. The response includes the X-Log-Next-Request-End-Time header for pagination.

For each response, use the POST method at /executions/{runId}/logs/status with the returned token to retrieve a presigned download URL. Each token is valid for one hour.

Before you begin

• You must have the Author, the Viewer and the Executor permissions for the workspace to which the task to be monitored belongs.

• You must know the ID of the task run you need to monitor.

  This ID is available on the Task execution log page, reading as Task execution ID. Or you can obtain this ID via a GET call from the /executables/tasks/{taskId}/executions endpoint.

About this task

In this example, Talend API Tester is used to issue API requests. For further information about Talend API Tester, see Talend Cloud API Tester User Guide.

Procedure

Scenario 1: Generating and downloading small logs

Use this scenario when you expect a complete log to be returned in a single response (HTTP 200).

1. Open Talend API Tester in your browser and select POST from the Method list.

2. Enter the endpoint: https://api.<your_environment>.cloud.talend.com/monitoring/executions/<runId>/logs

3. Add the startTime and endTime query parameters to define your log’s time range.

   Date format: yyyy-mm-ddThh:mm:ssZ (for example, 2021-09-01T07:01:45Z).

4. Add an Authorization header with the value Bearer <your_token> (separate Bearer and the token with a single space).

5. Verify that the run has completed, then wait 30 seconds before sending the API call to ensure log completeness.

6. Click Send to issue your request.

   The response contains the log size and access token:

   {
     "size": "log_size_in_bytes",
     "token": "generated_token"
   }
   

   If the response status is 206 (Partial Content), follow Scenario 2 instead.

7. Create a new POST request.

8. Enter the endpoint: https://api.<your_environment>.cloud.talend.com/monitoring/executions/<runId>/logs/status

9. In the Body field, enter the token from step 6.

   Optionally, use the expression builder to dynamically retrieve the token from your saved request.

   

   Example expression builder:

10. Click Send to issue your request.

    The response contains a presigned download URL and status:

    {
      "presignedURL": "https://s3_download_url",
      "status": "READY"
    }
    

    If the status is not READY, retry this request until it shows READY.

11. Use the presignedURL to download the complete log. The presigned URL expires after one hour.

---

Scenario 2: Downloading large logs with pagination

Use this scenario when the initial request returns HTTP 206 (Partial Content). You must iterate through multiple requests, calling the status endpoint for each chunk to generate a presigned URL and download it before requesting the next chunk.

1. Follow Scenario 1, steps 1–6 to issue the initial /logs request.

   The response status is 206 and includes:

   • token: Use this token to retrieve the download URL for this log chunk.
   • X-Log-Next-Request-End-Time header: Use this timestamp as the endTime for the next request.

   {
     "size": "100000",
     "token": "token_for_chunk_1"
   }
   

2. Create a POST request to retrieve the presigned download URL for this log chunk.

   Endpoint: https://api.<your_environment>.cloud.talend.com/monitoring/executions/<runId>/logs/status

   Body: Include the token from the 206 response.

3. Click Send. The response returns:

   {
     "presignedURL": "https://s3_temporary_download_url",
     "status": "READY"
   }
   

4. Download the log chunk using the presignedURL. The presigned URL expires after one hour.

5. To retrieve the next log chunk, issue a new /logs request:

   Endpoint: https://api.<your_environment>.cloud.talend.com/monitoring/executions/<runId>/logs

   Update the endTime query parameter to the value from the X-Log-Next-Request-End-Time header of the previous response.

6. Repeat steps 2–5:

   • Call /logs/status with the new token to get the presigned URL
   • Download the log chunk
   • If the /logs response status is 206: Continue to the next iteration with the new X-Log-Next-Request-End-Time
   • If the /logs response status is 200: Download the final chunk and stop

7. Combine all downloaded log chunks in the order received to reconstruct the complete log.

Integrating into your monitoring system

You can integrate these API requests into your monitoring system to periodically fetch and analyze logs.

To analyze all runs of a task, retrieve the run IDs using this endpoint:

https://api.<your_environment>.cloud.talend.com/processing/executables/tasks/{taskId}/executions

Then pass each run ID to the /executions/{runId}/logs and /executions/{runId}/logs/status endpoints. For large logs, ensure you handle the 206 response pagination by following Scenario 2.

Hints: Automating the process

You can use Qlik Talend products to automate this workflow.

With API Tester: Design and validate a pagination cycle:

1. Creating a scenario that sequences /logs requests and one /logs/status requests.
2. Use Using expressions to extract the token from the response and pass it to the status endpoint.
3. Use Assertions values to validate that the response status code is either 200 or 206.
4. Use expressions to extract the X-Log-Next-Request-End-Time header for the next pagination cycle.

With Talend Jobs: Use the validated pagination cycle in a Qlik Talend Job:

• Use tLoop to construct loops to repeat requests while status is 206
• Use tRESTClient and tJava to call the /logs and /logs/status endpoints, get the header fields, and seed the loop.
• Extract response data (tokens, download link, log status) using tExtractJSONFields
• Download and store log chunks sequentially, then combine them after the loop exits