Model Monitoring Inference Aggregator¶

Periodically aggregate and report a curated sample of inference predictions to Roboflow Model Monitoring by collecting predictions in memory, grouping by class, selecting the most confident prediction per class, and sending aggregated results at configurable intervals to enable efficient video processing monitoring, production analytics, and model performance tracking workflows with minimal performance overhead.

How This Block Works¶

This block aggregates predictions over time and sends representative samples to Roboflow Model Monitoring at regular intervals, reducing API calls and maintaining video processing performance. The block:

1. Receives predictions and configuration:
2. Takes predictions from any supported model type (object detection, instance segmentation, keypoint detection, or classification)
3. Receives model ID for identification in Model Monitoring
4. Accepts frequency parameter specifying reporting interval in seconds
5. Receives execution mode flag (fire-and-forget)
6. Validates Roboflow API key:
7. Checks that a valid Roboflow API key is available (required for API access)
8. Raises an error if API key is missing with instructions on how to retrieve one
9. Collects predictions in memory:
10. Stores predictions in an in-memory aggregator organized by model ID
11. Accumulates predictions between reporting intervals
12. Maintains state for the duration of the workflow execution session
13. Checks reporting interval:
14. Uses cache to track last report time based on unique aggregator key
15. Calculates time elapsed since last report
16. Compares elapsed time to configured frequency threshold
17. Skips reporting if interval has not been reached (returns status message)
18. Consolidates predictions when reporting:
19. Formats all collected predictions for Model Monitoring
20. Groups predictions by class name across all collected data
21. For each class, sorts predictions by confidence (highest first)
22. Selects the most confident prediction per class as representative sample
23. Creates a curated set of predictions (one per class with highest confidence)
24. Retrieves workspace information:
25. Gets workspace ID from Roboflow API using the provided API key
26. Uses caching (15-minute expiration) to avoid repeated API calls
27. Caches workspace name using MD5 hash of API key as cache key
28. Sends aggregated data to Model Monitoring:
29. Constructs inference data payload with timestamp, source info, device ID, and server version
30. Includes system information (if available) for monitoring context
31. Sends aggregated predictions (one per class) to Roboflow Model Monitoring API
32. Flushes in-memory aggregator after sending (starts fresh collection)
33. Updates last report time in cache
34. Executes synchronously or asynchronously:
35. Asynchronous mode (fire_and_forget=True): Submits task to background thread pool or FastAPI background tasks, allowing workflow to continue without waiting for API call to complete
36. Synchronous mode (fire_and_forget=False): Waits for API call to complete and returns immediate status, useful for debugging and error handling
37. Returns status information:
38. Outputs error_status indicating success (False) or failure (True)
39. Outputs message with reporting status or error details
40. Provides feedback on whether aggregation was sent or skipped

The block is optimized for video processing workflows where sending every prediction would create excessive API calls and impact performance. By aggregating predictions and selecting representative samples (most confident per class), the block provides meaningful monitoring data while minimizing overhead. The interval-based reporting ensures regular updates to Model Monitoring without constant API calls.

Common Use Cases¶

🔍 Why Use This Block?¶

This block is a game-changer for projects relying on video processing in Workflows. With its aggregation process, it identifies the most confident predictions across classes and sends them at regular intervals in small messages to Roboflow backend - ensuring that video processing performance is impacted to the least extent.

Perfect for:

• Monitoring production line performance in real-time 🏭.

• Debugging and validating your model’s performance over time ⏱️.

• Providing actionable insights from inference workflows with minimal overhead 🔧.

🚨 Limitations¶

• The block is should not be relied on when running Workflow in inference server or via HTTP request to Roboflow hosted platform, as the internal state is not persisted in a memory that would be accessible for all requests to the server, causing aggregation to only have a scope of single request. We will solve that problem in future releases if proven to be serious limitation for clients.

Connecting to Other Blocks¶

This block receives predictions and outputs status information:

• After model blocks (Object Detection Model, Instance Segmentation Model, Classification Model, Keypoint Detection Model) to aggregate and report predictions to Model Monitoring (e.g., aggregate detection results, report classification outputs, monitor model predictions), enabling model-to-monitoring workflows
• After filtering or analytics blocks (DetectionsFilter, ContinueIf, OverlapFilter) to aggregate filtered or analyzed results for monitoring (e.g., aggregate filtered detections, report analytics results, monitor processed predictions), enabling analysis-to-monitoring workflows
• In video processing workflows to efficiently monitor video analysis with minimal performance impact (e.g., aggregate video frame detections, report video processing results, monitor video analysis performance), enabling video monitoring workflows
• After preprocessing or transformation blocks to monitor transformed predictions (e.g., aggregate transformed detections, report processed results, monitor transformation outputs), enabling transformation-to-monitoring workflows
• In production deployment workflows to track model performance in production environments (e.g., monitor production inference, track deployment performance, report production metrics), enabling production monitoring workflows
• As a sink block to send aggregated monitoring data without blocking workflow execution (e.g., background monitoring reporting, non-blocking analytics, efficient data collection), enabling sink-to-monitoring workflows

Requirements¶

This block requires a valid Roboflow API key configured in the environment or workflow configuration. The API key is required to authenticate with Roboflow API and access Model Monitoring features. Visit https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key to learn how to retrieve an API key. The block maintains in-memory state for aggregation, which means it works best for long-running workflows (like video processing with InferencePipeline). The block should not be relied upon when running workflows in inference server or via HTTP requests to Roboflow hosted platform, as the internal state is only accessible for single requests and aggregation scope is limited to single request execution. The block aggregates data for all video feeds connected to a single InferencePipeline process (cannot separate aggregations per video feed). The frequency parameter must be at least 1 second. For more information on Model Monitoring at Roboflow, see https://docs.roboflow.com/deploy/model-monitoring.

Type identifier¶

Use the following identifier in step "type" field: roboflow_core/model_monitoring_inference_aggregator@v1to add the block as as step in your workflow.

Properties¶

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections¶

Input and Output Bindings¶

The available connections depend on its binding kinds. Check what binding kinds Model Monitoring Inference Aggregator in version v1 has.

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/model_monitoring_inference_aggregator@v1",
    "predictions": "$steps.object_detection.predictions",
    "model_id": "my_project/3",
    "frequency": 3,
    "unique_aggregator_key": "session-1v73kdhfse",
    "fire_and_forget": true
}