Skip to content

Model Monitoring Inference Aggregator

Class: ModelMonitoringInferenceAggregatorBlockV1

Source: inference.core.workflows.core_steps.sinks.roboflow.model_monitoring_inference_aggregator.v1.ModelMonitoringInferenceAggregatorBlockV1

Periodically aggregate and report a curated sample of inference predictions to Roboflow Model Monitoring by collecting predictions in memory, grouping by class, selecting the most confident prediction per class, and sending aggregated results at configurable intervals to enable efficient video processing monitoring, production analytics, and model performance tracking workflows with minimal performance overhead.

How This Block Works

This block aggregates predictions over time and sends representative samples to Roboflow Model Monitoring at regular intervals, reducing API calls and maintaining video processing performance. The block:

  1. Receives predictions and configuration:
  2. Takes predictions from any supported model type (object detection, instance segmentation, keypoint detection, or classification)
  3. Receives model ID for identification in Model Monitoring
  4. Accepts frequency parameter specifying reporting interval in seconds
  5. Receives execution mode flag (fire-and-forget)
  6. Validates Roboflow API key:
  7. Checks that a valid Roboflow API key is available (required for API access)
  8. Raises an error if API key is missing with instructions on how to retrieve one
  9. Collects predictions in memory:
  10. Stores predictions in an in-memory aggregator organized by model ID
  11. Accumulates predictions between reporting intervals
  12. Maintains state for the duration of the workflow execution session
  13. Checks reporting interval:
  14. Uses cache to track last report time based on unique aggregator key
  15. Calculates time elapsed since last report
  16. Compares elapsed time to configured frequency threshold
  17. Skips reporting if interval has not been reached (returns status message)
  18. Consolidates predictions when reporting:
  19. Formats all collected predictions for Model Monitoring
  20. Groups predictions by class name across all collected data
  21. For each class, sorts predictions by confidence (highest first)
  22. Selects the most confident prediction per class as representative sample
  23. Creates a curated set of predictions (one per class with highest confidence)
  24. Retrieves workspace information:
  25. Gets workspace ID from Roboflow API using the provided API key
  26. Uses caching (15-minute expiration) to avoid repeated API calls
  27. Caches workspace name using MD5 hash of API key as cache key
  28. Sends aggregated data to Model Monitoring:
  29. Constructs inference data payload with timestamp, source info, device ID, and server version
  30. Includes system information (if available) for monitoring context
  31. Sends aggregated predictions (one per class) to Roboflow Model Monitoring API
  32. Flushes in-memory aggregator after sending (starts fresh collection)
  33. Updates last report time in cache
  34. Executes synchronously or asynchronously:
  35. Asynchronous mode (fire_and_forget=True): Submits task to background thread pool or FastAPI background tasks, allowing workflow to continue without waiting for API call to complete
  36. Synchronous mode (fire_and_forget=False): Waits for API call to complete and returns immediate status, useful for debugging and error handling
  37. Returns status information:
  38. Outputs error_status indicating success (False) or failure (True)
  39. Outputs message with reporting status or error details
  40. Provides feedback on whether aggregation was sent or skipped

The block is optimized for video processing workflows where sending every prediction would create excessive API calls and impact performance. By aggregating predictions and selecting representative samples (most confident per class), the block provides meaningful monitoring data while minimizing overhead. The interval-based reporting ensures regular updates to Model Monitoring without constant API calls.

Common Use Cases

🔍 Why Use This Block?

This block is a game-changer for projects relying on video processing in Workflows. With its aggregation process, it identifies the most confident predictions across classes and sends them at regular intervals in small messages to Roboflow backend - ensuring that video processing performance is impacted to the least extent.

Perfect for:

  • Monitoring production line performance in real-time 🏭.

  • Debugging and validating your model’s performance over time ⏱️.

  • Providing actionable insights from inference workflows with minimal overhead 🔧.

🚨 Limitations

  • The block is should not be relied on when running Workflow in inference server or via HTTP request to Roboflow hosted platform, as the internal state is not persisted in a memory that would be accessible for all requests to the server, causing aggregation to only have a scope of single request. We will solve that problem in future releases if proven to be serious limitation for clients.

Connecting to Other Blocks

This block receives predictions and outputs status information:

  • After model blocks (Object Detection Model, Instance Segmentation Model, Classification Model, Keypoint Detection Model) to aggregate and report predictions to Model Monitoring (e.g., aggregate detection results, report classification outputs, monitor model predictions), enabling model-to-monitoring workflows
  • After filtering or analytics blocks (DetectionsFilter, ContinueIf, OverlapFilter) to aggregate filtered or analyzed results for monitoring (e.g., aggregate filtered detections, report analytics results, monitor processed predictions), enabling analysis-to-monitoring workflows
  • In video processing workflows to efficiently monitor video analysis with minimal performance impact (e.g., aggregate video frame detections, report video processing results, monitor video analysis performance), enabling video monitoring workflows
  • After preprocessing or transformation blocks to monitor transformed predictions (e.g., aggregate transformed detections, report processed results, monitor transformation outputs), enabling transformation-to-monitoring workflows
  • In production deployment workflows to track model performance in production environments (e.g., monitor production inference, track deployment performance, report production metrics), enabling production monitoring workflows
  • As a sink block to send aggregated monitoring data without blocking workflow execution (e.g., background monitoring reporting, non-blocking analytics, efficient data collection), enabling sink-to-monitoring workflows

Requirements

This block requires a valid Roboflow API key configured in the environment or workflow configuration. The API key is required to authenticate with Roboflow API and access Model Monitoring features. Visit https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key to learn how to retrieve an API key. The block maintains in-memory state for aggregation, which means it works best for long-running workflows (like video processing with InferencePipeline). The block should not be relied upon when running workflows in inference server or via HTTP requests to Roboflow hosted platform, as the internal state is only accessible for single requests and aggregation scope is limited to single request execution. The block aggregates data for all video feeds connected to a single InferencePipeline process (cannot separate aggregations per video feed). The frequency parameter must be at least 1 second. For more information on Model Monitoring at Roboflow, see https://docs.roboflow.com/deploy/model-monitoring.

Type identifier

Use the following identifier in step "type" field: roboflow_core/model_monitoring_inference_aggregator@v1to add the block as as step in your workflow.

Properties

Name Type Description Refs
name str Enter a unique identifier for this step.. ❌
frequency int Reporting frequency in seconds. Specifies how often aggregated predictions are sent to Roboflow Model Monitoring. For example, if set to 5, the block collects predictions for 5 seconds, then sends the aggregated sample (one most confident prediction per class) to Model Monitoring. Must be at least 1 second. Lower values provide more frequent updates but increase API calls. Higher values reduce API calls but provide less frequent updates. Default: 5 seconds. Works well for video processing where you want regular but not excessive reporting.. ✅
unique_aggregator_key str Unique key used internally to track the aggregation session and cache last report time. This key must be unique for each instance of this block in your workflow. The key is used to create cache entries that track when the last report was sent, enabling interval-based reporting. This field is automatically generated and hidden in the UI.. ❌
fire_and_forget bool Execution mode flag. When True (default), the block runs asynchronously in the background, allowing the workflow to continue processing without waiting for the API call to complete. This provides faster workflow execution but errors are not immediately available. When False, the block runs synchronously and waits for the API call to complete, returning immediate status and error information. Use False for debugging and error handling, True for production workflows where performance is prioritized.. ✅

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to Model Monitoring Inference Aggregator in version v1.

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds Model Monitoring Inference Aggregator in version v1 has.

Bindings

  • input

    • predictions (Union[classification_prediction, object_detection_prediction, keypoint_detection_prediction, instance_segmentation_prediction]): Model predictions (object detection, instance segmentation, keypoint detection, or classification) to aggregate and report to Roboflow Model Monitoring. Predictions are collected in memory, grouped by class name, and the most confident prediction per class is selected as a representative sample. Predictions accumulate between reporting intervals based on the frequency setting. Supported prediction types: supervision Detections objects or classification prediction dictionaries..
    • model_id (roboflow_model_id): Roboflow model ID (format: 'project/version') to associate with the predictions in Model Monitoring. This identifies which model generated the predictions being reported. The model ID is included in the monitoring data sent to Roboflow, allowing you to track performance per model in the Model Monitoring dashboard..
    • frequency (string): Reporting frequency in seconds. Specifies how often aggregated predictions are sent to Roboflow Model Monitoring. For example, if set to 5, the block collects predictions for 5 seconds, then sends the aggregated sample (one most confident prediction per class) to Model Monitoring. Must be at least 1 second. Lower values provide more frequent updates but increase API calls. Higher values reduce API calls but provide less frequent updates. Default: 5 seconds. Works well for video processing where you want regular but not excessive reporting..
    • fire_and_forget (boolean): Execution mode flag. When True (default), the block runs asynchronously in the background, allowing the workflow to continue processing without waiting for the API call to complete. This provides faster workflow execution but errors are not immediately available. When False, the block runs synchronously and waits for the API call to complete, returning immediate status and error information. Use False for debugging and error handling, True for production workflows where performance is prioritized..

  • output

    • error_status (boolean): Boolean flag.
    • message (string): String value.
Example JSON definition of step Model Monitoring Inference Aggregator in version v1 
{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/model_monitoring_inference_aggregator@v1",
    "predictions": "$steps.object_detection.predictions",
    "model_id": "my_project/3",
    "frequency": 3,
    "unique_aggregator_key": "session-1v73kdhfse",
    "fire_and_forget": true
}