Roboflow Dataset Upload¶

v2¶

Upload images and model predictions to a Roboflow dataset for active learning, model improvement, and data collection, with configurable usage quotas, probabilistic sampling, batch organization, image compression, and optional annotation persistence.

How This Block Works¶

This block uploads workflow images and predictions to your Roboflow dataset for storage, labeling, and model training. The block:

1. Takes images and optional model predictions (object detection, instance segmentation, keypoint detection, or classification) as input
2. Validates the Roboflow API key is available (required for uploading)
3. Applies probabilistic sampling based on data_percentage setting, randomly selecting a percentage of inputs to upload (e.g., 50% uploads half the data, 100% uploads everything)
4. Checks usage quotas (minutely, hourly, daily limits) to ensure uploads stay within configured rate limits for active learning strategies
5. Prepares images by resizing if they exceed maximum size (maintaining aspect ratio) and compressing to specified quality level
6. Generates labeling batch names based on the prefix and batch creation frequency (never, daily, weekly, or monthly), organizing uploaded data into batches
7. Optionally persists model predictions as annotations if persist_predictions is enabled, allowing predictions to serve as pre-labels for review and correction
8. Attaches registration tags to images for organization and filtering in the Roboflow platform
9. Registers the image (and annotations if enabled) to the specified Roboflow project via the Roboflow API
10. Executes synchronously or asynchronously based on fire_and_forget setting, allowing non-blocking uploads for faster workflow execution
11. Returns error status and messages indicating upload success, failure, or sampling skip

The block supports active learning workflows by implementing usage quotas that prevent excessive data collection, helping focus on collecting valuable training data within rate limits. The probabilistic sampling feature (new in v2) allows you to randomly sample a percentage of data for upload, enabling cost-effective data collection strategies where you want to collect representative samples rather than all data. Images are organized into labeling batches that can be automatically recreated on a schedule (daily, weekly, monthly), making it easier to manage and review collected data over time. The block can operate in fire-and-forget mode for asynchronous execution, allowing workflows to continue processing without waiting for uploads to complete, or synchronously for debugging and error handling.

Version Differences (v2 vs v1)¶

New Features in v2:

• Probabilistic Data Sampling: Added data_percentage parameter (0-100%) that enables random sampling of data for upload. This allows you to upload only a percentage of workflow inputs (e.g., 25% samples one in four images), reducing storage and annotation costs while still collecting representative data. When sampling skips an upload, the block returns a message indicating the skip.

• Improved Default Settings:

• max_image_size default increased from (512, 512) to (1920, 1080) for higher resolution data collection
• compression_level default increased from 75 to 95 for better image quality preservation

Behavior Changes:

• By default, data_percentage is set to 100, so v2 behaves identically to v1 unless sampling is explicitly configured
• The block now uses probabilistic sampling before quota checking and image preparation, allowing efficient filtering before resource-intensive operations

Requirements¶

API Key Required: This block requires a valid Roboflow API key to upload data. The API key must be configured in your environment or workflow configuration. Visit https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key to learn how to retrieve an API key.

Common Use Cases¶

• Active Learning Data Collection: Collect images and predictions from production environments where models struggle or are uncertain (e.g., low-confidence detections, edge cases), enabling iterative model improvement by gathering challenging examples for retraining
• Probabilistic Data Sampling: Use data_percentage to randomly sample a subset of data for upload (e.g., upload 20% of all detections, 50% of low-confidence cases), enabling cost-effective data collection strategies that reduce storage and annotation overhead while maintaining dataset diversity
• Production Data Logging: Continuously upload production inference data to Roboflow datasets for monitoring, analysis, and future model training, creating a growing dataset from real-world deployments
• Pre-Labeled Data Collection: Upload images with model predictions as pre-labels (when persist_predictions is enabled), accelerating annotation workflows by providing initial labels that can be reviewed and corrected rather than starting from scratch
• Stratified Data Sampling: Combine probabilistic sampling with rate limiting and quotas to selectively collect data based on specific criteria (e.g., sample 30% of detections that pass filters), ensuring diverse and balanced dataset collection without overwhelming storage or annotation resources
• Batch-Based Labeling Workflows: Organize uploaded data into batches with automatic recreation schedules (daily, weekly, monthly), making it easier to manage labeling tasks, track progress, and organize data collection efforts over time

Connecting to Other Blocks¶

This block receives data from workflow steps and uploads it to Roboflow:

• After detection or analysis blocks (e.g., Object Detection Model, Instance Segmentation Model, Classification Model, Keypoint Detection Model) to upload images along with their predictions, enabling active learning by collecting inference data with model outputs for annotation and retraining
• After filtering or analytics blocks (e.g., Detections Filter, Continue If, Overlap Filter) to selectively upload only specific types of data (e.g., low-confidence detections, overlapping objects, specific classes), focusing data collection on valuable edge cases or interesting scenarios
• After rate limiter blocks (e.g., Rate Limiter) to throttle upload frequency and stay within usage quotas, ensuring controlled data collection that respects rate limits and prevents excessive storage usage
• Image inputs or preprocessing blocks to upload raw images or processed images (e.g., crops, transformed images) without predictions, enabling collection of image data for future labeling or analysis
• Conditional workflows using flow control blocks (e.g., Continue If) to upload data only when certain conditions are met (e.g., upload only when detection count exceeds threshold, upload only errors or failures), enabling selective data collection based on workflow state
• Batch processing workflows where multiple images or predictions are generated, allowing bulk upload of workflow outputs to Roboflow datasets with probabilistic sampling for organized and cost-effective data collection

Type identifier¶

Use the following identifier in step "type" field: roboflow_core/roboflow_dataset_upload@v2to add the block as as step in your workflow.

Properties¶

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections¶

Input and Output Bindings¶

The available connections depend on its binding kinds. Check what binding kinds Roboflow Dataset Upload in version v2 has.

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/roboflow_dataset_upload@v2",
    "images": "$inputs.image",
    "target_project": "my_dataset",
    "predictions": "$steps.object_detection_model.predictions",
    "data_percentage": 100,
    "minutely_usage_limit": 10,
    "hourly_usage_limit": 10,
    "daily_usage_limit": 10,
    "usage_quota_name": "quota-for-data-sampling-1",
    "max_image_size": [
        1920,
        1080
    ],
    "compression_level": 95,
    "registration_tags": [
        "location-florida",
        "factory-name",
        "$inputs.dynamic_tag"
    ],
    "persist_predictions": true,
    "disable_sink": true,
    "fire_and_forget": "<block_does_not_provide_example>",
    "labeling_batch_prefix": "my_labeling_batch_name",
    "labeling_batches_recreation_frequency": "never",
    "image_name": "serial_12345",
    "metadata": {
        "camera_id": "cam_01",
        "location": "$inputs.location"
    }
}

v1¶

Upload images and model predictions to a Roboflow dataset for active learning, model improvement, and data collection, with configurable usage quotas, batch organization, image compression, and optional annotation persistence.

How This Block Works¶

This block uploads workflow images and predictions to your Roboflow dataset for storage, labeling, and model training. The block:

1. Takes images and optional model predictions (object detection, instance segmentation, keypoint detection, or classification) as input
2. Validates the Roboflow API key is available (required for uploading)
3. Checks usage quotas (minutely, hourly, daily limits) to ensure uploads stay within configured rate limits for active learning strategies
4. Prepares images by resizing if they exceed maximum size (maintaining aspect ratio) and compressing to specified quality level
5. Generates labeling batch names based on the prefix and batch creation frequency (never, daily, weekly, or monthly), organizing uploaded data into batches
6. Optionally persists model predictions as annotations if persist_predictions is enabled, allowing predictions to serve as pre-labels for review and correction
7. Attaches registration tags to images for organization and filtering in the Roboflow platform
8. Registers the image (and annotations if enabled) to the specified Roboflow project via the Roboflow API
9. Executes synchronously or asynchronously based on fire_and_forget setting, allowing non-blocking uploads for faster workflow execution
10. Returns error status and messages indicating upload success or failure

The block supports active learning workflows by implementing usage quotas that prevent excessive data collection, helping focus on collecting valuable training data within rate limits. Images are organized into labeling batches that can be automatically recreated on a schedule (daily, weekly, monthly), making it easier to manage and review collected data over time. The block can operate in fire-and-forget mode for asynchronous execution, allowing workflows to continue processing without waiting for uploads to complete, or synchronously for debugging and error handling.

Requirements¶

API Key Required: This block requires a valid Roboflow API key to upload data. The API key must be configured in your environment or workflow configuration. Visit https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key to learn how to retrieve an API key.

Common Use Cases¶

• Active Learning Data Collection: Collect images and predictions from production environments where models struggle or are uncertain (e.g., low-confidence detections, edge cases), enabling iterative model improvement by gathering challenging examples for retraining
• Production Data Logging: Continuously upload production inference data to Roboflow datasets for monitoring, analysis, and future model training, creating a growing dataset from real-world deployments
• Pre-Labeled Data Collection: Upload images with model predictions as pre-labels (when persist_predictions is enabled), accelerating annotation workflows by providing initial labels that can be reviewed and corrected rather than starting from scratch
• Stratified Data Sampling: Use rate limiting and quotas to selectively collect data based on specific criteria (e.g., combine with Rate Limiter or Continue If blocks), ensuring diverse and balanced dataset collection without overwhelming storage or annotation resources
• Batch-Based Labeling Workflows: Organize uploaded data into batches with automatic recreation schedules (daily, weekly, monthly), making it easier to manage labeling tasks, track progress, and organize data collection efforts over time
• Tagged Data Organization: Attach metadata tags to uploaded images (e.g., location, camera ID, time period, model version), enabling filtering and organization of collected data in Roboflow for better dataset management and analysis

Connecting to Other Blocks¶

This block receives data from workflow steps and uploads it to Roboflow:

• After detection or analysis blocks (e.g., Object Detection Model, Instance Segmentation Model, Classification Model, Keypoint Detection Model) to upload images along with their predictions, enabling active learning by collecting inference data with model outputs for annotation and retraining
• After filtering or analytics blocks (e.g., Detections Filter, Continue If, Overlap Filter) to selectively upload only specific types of data (e.g., low-confidence detections, overlapping objects, specific classes), focusing data collection on valuable edge cases or interesting scenarios
• After rate limiter blocks (e.g., Rate Limiter) to throttle upload frequency and stay within usage quotas, ensuring controlled data collection that respects rate limits and prevents excessive storage usage
• Image inputs or preprocessing blocks to upload raw images or processed images (e.g., crops, transformed images) without predictions, enabling collection of image data for future labeling or analysis
• Conditional workflows using flow control blocks (e.g., Continue If) to upload data only when certain conditions are met (e.g., upload only when detection count exceeds threshold, upload only errors or failures), enabling selective data collection based on workflow state
• Batch processing workflows where multiple images or predictions are generated, allowing bulk upload of workflow outputs to Roboflow datasets for organized data collection and management

Type identifier¶

Use the following identifier in step "type" field: roboflow_core/roboflow_dataset_upload@v1to add the block as as step in your workflow.

Properties¶

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections¶

Input and Output Bindings¶

The available connections depend on its binding kinds. Check what binding kinds Roboflow Dataset Upload in version v1 has.

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/roboflow_dataset_upload@v1",
    "image": "$inputs.image",
    "predictions": "$steps.object_detection_model.predictions",
    "target_project": "my_project",
    "minutely_usage_limit": 10,
    "hourly_usage_limit": 10,
    "daily_usage_limit": 10,
    "usage_quota_name": "quota-for-data-sampling-1",
    "max_image_size": [
        512,
        512
    ],
    "compression_level": 75,
    "registration_tags": [
        "location-florida",
        "factory-name",
        "$inputs.dynamic_tag"
    ],
    "persist_predictions": true,
    "disable_sink": true,
    "fire_and_forget": true,
    "labeling_batch_prefix": "my_labeling_batch_name",
    "labeling_batches_recreation_frequency": "never",
    "image_name": "serial_12345"
}