S3 Sink¶

Save workflow data directly to an AWS S3 bucket, supporting CSV, JSON, and text file formats with configurable output modes for aggregating multiple entries into single objects or saving each entry as a separate S3 object.

How This Block Works¶

This block uploads string content from workflow steps to S3 objects. The block:

1. Takes string content (from formatters, predictions, or other string-producing blocks) and S3 configuration as input
2. Connects to AWS S3 using the provided credentials (or the default AWS credential chain if none are supplied)
3. Selects the appropriate upload strategy based on output_mode:
4. Separate Files Mode: Creates a new S3 object for each input, generating unique keys with timestamps
5. Append Log Mode: Buffers content in memory, uploading a complete object when max_entries_per_file is reached or when the block is destroyed
6. For separate files mode: Generates a unique S3 key from the prefix, file name prefix, file type, and a timestamp, then uploads the content directly
7. For append log mode:
8. Buffers content entries in memory under a single S3 key
9. Applies format-specific handling for appending:
   • CSV: Removes the header row from subsequent appends (CSV content must include headers on first write)
   • JSON: Converts to JSONL (JSON Lines) format, parsing and re-serializing each JSON document to fit on a single line
   • TXT: Appends content directly with newlines
10. Tracks entry count and uploads the full buffer as a complete S3 object when max_entries_per_file is reached, then starts a fresh buffer with a new key
11. Uploads any remaining buffered data when the block is destroyed
12. Returns error status and messages indicating save success or failure

The block supports two storage strategies: separate files mode creates individual timestamped S3 objects per input (useful for organizing outputs by execution), while append log mode accumulates entries in memory and writes them as complete S3 objects on rotation (useful for time-series logging with controlled upload frequency). S3 key names include timestamps (format: YYYY_MM_DD_HH_MM_SS_microseconds) for unique keys and chronological ordering.

AWS Credentials¶

Credentials can be supplied in two ways: 1. Workflow inputs — declare aws_access_key_id and aws_secret_access_key as workflow inputs of kind parameter and connect them to the corresponding fields. This keeps credentials out of the workflow definition and allows them to be supplied at runtime. 2. Secrets provider block — connect the credential fields to the output of an Environment Secrets Store block, which reads values from server-side environment variables without embedding them in the workflow. Note: this is only available on self-hosted inference servers and cannot be used on the Roboflow hosted platform.

S3 Key Structure¶

The final S3 key is composed of:

{s3_prefix}/{file_name_prefix}_{timestamp}.{extension}

logs/detections/run_2024_10_18_14_09_57_622297.csv

Note on Append Log Mode¶

In append log mode, data is buffered in memory and only uploaded to S3 when: - The max_entries_per_file limit is reached (object rotation), or - The block instance is destroyed at workflow teardown

This means data may not be immediately visible in S3 after each step execution. Use separate_files mode if immediate S3 visibility is required.

Common Use Cases¶

• Cloud Data Logging: Upload detection results, metrics, or workflow outputs directly to S3 for durable cloud storage and downstream processing
• Data Pipeline Integration: Export formatted CSV or JSONL files to S3 for consumption by data pipelines, analytics tools, or ML training jobs
• Batch Result Archival: Store individual inference results as separate S3 objects organized by timestamp and prefix
• Time-Series Collection: Aggregate workflow outputs into batched JSONL or CSV files in S3 for cost-efficient log storage
• Cross-Service Integration: Write data to S3 to trigger Lambda functions, feed SQS queues, or integrate with other AWS services

Type identifier¶

Use the following identifier in step "type" field: roboflow_core/s3_sink@v1to add the block as as step in your workflow.

Properties¶

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections¶

Input and Output Bindings¶

The available connections depend on its binding kinds. Check what binding kinds S3 Sink in version v1 has.

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/s3_sink@v1",
    "content": "$steps.csv_formatter.csv_content",
    "file_type": "csv",
    "output_mode": "append_log",
    "bucket_name": "my-inference-results",
    "s3_prefix": "logs/detections",
    "file_name_prefix": "my_output",
    "max_entries_per_file": 1024,
    "aws_access_key_id": "$steps.secrets.aws_access_key_id",
    "aws_secret_access_key": "$steps.secrets.aws_secret_access_key",
    "aws_region": "us-east-1"
}