Detections Classes Replacement¶

Replace class labels of detection bounding boxes with classes predicted by a classification model applied to cropped regions, combining generic detection results with specialized classification predictions to enable two-stage detection workflows, fine-grained classification, and class refinement workflows where generic detections are refined with specific class labels from specialized classifiers.

How This Block Works¶

This block combines results from a detection model (with bounding boxes and generic classes) with classification predictions (from a specialized classifier applied to cropped regions) to replace generic class labels with specific ones. The block:

1. Receives two inputs with different dimensionality levels:
2. object_detection_predictions: Detection results (dimensionality level 1) containing bounding boxes with generic classes (e.g., "dog", "person", "vehicle")
3. classification_predictions: Classification results (dimensionality level 2) from a classifier applied to cropped regions of each detection (e.g., "Golden Retriever", "Labrador" for dog detections). Can also be a list of strings (e.g. from OCR).
4. Matches classifications to detections:
5. Uses PARENT_ID_KEY (detection_id) in classification predictions to link each classification result to its source detection, OR
6. Uses positional mapping (order-based) if predictions are raw strings/lists without parent IDs.
7. Extracts leading class from each classification prediction:

For single-label classifications: - Uses the "top" class (predicted class) from the classification result - Extracts class name, class ID, and confidence from the classification prediction

For multi-label classifications: - Finds the class with the highest confidence score - Uses the most confident label as the replacement class - Extracts class name, class ID, and confidence from the highest-confidence prediction

For string predictions: - Uses the string as the class name - Assigns a default confidence of 1.0 and class ID of 0

1. Handles missing classifications:
2. Detections without corresponding classification predictions are discarded by default
3. If fallback_class_name is provided, detections without classifications use the fallback class instead of being discarded
4. Fallback class ID is set to the provided value, or sys.maxsize if not specified or negative
5. Filters detections:
6. Keeps only detections that have classification results (or fallback if specified)
7. Removes detections that cannot be matched to classification predictions
8. Replaces class information:
9. Replaces class names in detections with classification class names
10. Replaces class IDs in detections with classification class IDs
11. Replaces confidence scores in detections with classification confidence scores
12. Updates all detection metadata to reflect the new class information
13. Generates new detection IDs:
14. Creates new unique detection IDs for updated detections (prevents ID conflicts)
15. Ensures detection IDs are unique after class replacement
16. Returns updated detections:
17. Outputs detections with replaced classes, maintaining bounding box coordinates and other properties
18. Output dimensionality matches input detection predictions (dimensionality level 1)

The block enables two-stage detection workflows where a generic detection model locates objects and a specialized classification model provides fine-grained labels. This is useful when you need generic localization (e.g., "dog") combined with specific classification (e.g., "Golden Retriever", "German Shepherd") without losing spatial information.

Common Use Cases¶

• Two-Stage Detection and Classification: Combine generic detection with specialized classification for fine-grained labeling (e.g., detect "dog" then classify breed, detect "vehicle" then classify type, detect "person" then classify age group), enabling two-stage detection workflows
• Class Refinement: Refine generic class labels with specific classifications from specialized models (e.g., refine "animal" to specific species, refine "vehicle" to specific models, refine "food" to specific dishes), enabling class refinement workflows
• Multi-Model Workflows: Combine detection and classification models to leverage the strengths of both (e.g., use generic detector for localization and specialist classifier for identification, combine coarse and fine-grained models, leverage specialized classifiers with general detectors), enabling multi-model workflows
• Hierarchical Classification: Apply hierarchical classification where detection provides high-level classes and classification provides detailed sub-classes (e.g., detect "mammal" then classify species, detect "plant" then classify variety, detect "structure" then classify type), enabling hierarchical classification workflows
• Crop-Based Classification: Use classification results from cropped regions to enhance detection results (e.g., classify crops to improve detection labels, apply specialized classifiers to detected regions, refine detections with crop classifications), enabling crop-based classification workflows
• Fine-Grained Object Recognition: Enable fine-grained recognition by combining localization and detailed classification (e.g., recognize specific product models, identify specific animal breeds, classify specific vehicle types), enabling fine-grained recognition workflows

Connecting to Other Blocks¶

This block receives detection and classification predictions and produces detections with replaced classes:

• After detection and classification model blocks to combine generic detection with specialized classification (e.g., object detection + classification to refined detections, detection model + classifier to labeled detections), enabling detection-classification fusion workflows
• After crop blocks that create crops from detections for classification (e.g., crop detections then classify crops, create crops for classification then replace classes), enabling crop-classification workflows
• Before visualization blocks to display detections with refined classes (e.g., visualize refined detections, display detections with specific labels, show classification-enhanced detections), enabling refined detection visualization workflows
• Before filtering blocks to filter detections with refined classes (e.g., filter by specific classes, filter refined detections, apply filters to classified detections), enabling refined detection filtering workflows
• Before analytics blocks to perform analytics on refined detections (e.g., analyze specific classes, perform analytics on classified detections, track refined detection metrics), enabling refined detection analytics workflows
• In workflow outputs to provide refined detections as final output (e.g., two-stage detection outputs, classification-enhanced detection outputs, refined detection results), enabling refined detection output workflows

Requirements¶

This block requires object detection predictions (with bounding boxes) and classification predictions from crops of those bounding boxes. The classification predictions must have PARENT_ID_KEY (detection_id) to link classifications to their source detections. The block accepts different dimensionality levels: detection predictions at level 1 and classification predictions at level 2 (from crops). For single-label classifications, the "top" class is used. For multi-label classifications, the most confident class is selected. Detections without classification results are discarded unless fallback_class_name is provided. The block outputs detections with replaced classes, class IDs, and confidences, with new detection IDs generated. Output dimensionality matches input detection predictions (level 1).

Type identifier¶

Use the following identifier in step "type" field: roboflow_core/detections_classes_replacement@v1to add the block as as step in your workflow.

Properties¶

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections¶

Input and Output Bindings¶

The available connections depend on its binding kinds. Check what binding kinds Detections Classes Replacement in version v1 has.

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/detections_classes_replacement@v1",
    "object_detection_predictions": "$steps.object_detection_model.predictions",
    "classification_predictions": "$steps.classification_model.predictions",
    "fallback_class_name": null,
    "fallback_class_id": null
}