VLM as Detector¶
v2¶
Class: VLMAsDetectorBlockV2
(there are multiple versions of this block)
Source: inference.core.workflows.core_steps.formatters.vlm_as_detector.v2.VLMAsDetectorBlockV2
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
The block expects string input that would be produced by blocks exposing Large Language Models (LLMs) and Visual Language Models (VLMs). Input is parsed to object-detection prediction and returned as block output.
Accepted formats:
-
valid JSON strings
-
JSON documents wrapped with Markdown tags
Example
{"my": "json"}
Details regarding block behavior:
-
error_status
is setTrue
whenever parsing cannot be completed -
in case of multiple markdown blocks with raw JSON content - only first will be parsed
Type identifier¶
Use the following identifier in step "type"
field: roboflow_core/vlm_as_detector@v2
to add the block as
as step in your workflow.
Properties¶
Name | Type | Description | Refs |
---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
classes |
List[str] |
List of all classes used by the model, required to generate mapping between class name and class id.. | ✅ |
model_type |
str |
Type of the model that generated prediction. | ❌ |
task_type |
str |
Task type to performed by model.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow
runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to VLM as Detector
in version v2
.
- inputs:
Image Preprocessing
,Relative Static Crop
,Background Color Visualization
,Line Counter Visualization
,SIFT
,Dynamic Zone
,OpenAI
,Reference Path Visualization
,Grid Visualization
,Blur Visualization
,Image Contours
,Pixelate Visualization
,Llama 3.2 Vision
,Camera Focus
,Color Visualization
,Ellipse Visualization
,Label Visualization
,Image Slicer
,Crop Visualization
,Stitch Images
,Polygon Zone Visualization
,Dot Visualization
,Bounding Box Visualization
,Florence-2 Model
,Image Slicer
,Perspective Correction
,Size Measurement
,Dimension Collapse
,Image Blur
,Stability AI Image Generation
,Dynamic Crop
,Image Convert Grayscale
,Stability AI Inpainting
,Clip Comparison
,Clip Comparison
,Model Comparison Visualization
,Triangle Visualization
,Classification Label Visualization
,Trace Visualization
,Anthropic Claude
,Buffer
,Depth Estimation
,Corner Visualization
,SIFT Comparison
,Camera Calibration
,Absolute Static Crop
,Keypoint Visualization
,Mask Visualization
,Image Threshold
,Florence-2 Model
,Halo Visualization
,Circle Visualization
,Google Gemini
,Polygon Visualization
- outputs:
Line Counter Visualization
,Background Color Visualization
,Roboflow Dataset Upload
,Gaze Detection
,Detections Stabilizer
,Detections Transformation
,Object Detection Model
,Roboflow Dataset Upload
,Instance Segmentation Model
,Color Visualization
,Label Visualization
,Detections Filter
,Dot Visualization
,Overlap Filter
,Byte Tracker
,Object Detection Model
,Bounding Box Visualization
,Detections Consensus
,Template Matching
,Single-Label Classification Model
,Path Deviation
,Dynamic Crop
,Model Comparison Visualization
,Corner Visualization
,SIFT Comparison
,Multi-Label Classification Model
,Byte Tracker
,Line Counter
,Detection Offset
,Keypoint Visualization
,Mask Visualization
,Florence-2 Model
,Halo Visualization
,Polygon Zone Visualization
,Polygon Visualization
,Instance Segmentation Model
,Twilio SMS Notification
,Dynamic Zone
,Roboflow Custom Metadata
,Reference Path Visualization
,Blur Visualization
,Detections Classes Replacement
,Pixelate Visualization
,Keypoint Detection Model
,Detections Stitch
,Ellipse Visualization
,Byte Tracker
,Crop Visualization
,Line Counter
,Florence-2 Model
,Stitch OCR Detections
,Webhook Sink
,Email Notification
,Perspective Correction
,Multi-Label Classification Model
,Size Measurement
,Velocity
,Time in Zone
,Model Monitoring Inference Aggregator
,Triangle Visualization
,Trace Visualization
,Classification Label Visualization
,Keypoint Detection Model
,Single-Label Classification Model
,Distance Measurement
,Segment Anything 2 Model
,Path Deviation
,Time in Zone
,Circle Visualization
,Slack Notification
,Detections Merge
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
VLM as Detector
in version v2
has.
Bindings
-
input
image
(image
): The image which was the base to generate VLM prediction.vlm_output
(language_model_output
): The string with raw classification prediction to parse..classes
(list_of_values
): List of all classes used by the model, required to generate mapping between class name and class id..
-
output
error_status
(boolean
): Boolean flag.predictions
(object_detection_prediction
): Prediction with detected bounding boxes in form of sv.Detections(...) object.inference_id
(inference_id
): Inference identifier.
Example JSON definition of step VLM as Detector
in version v2
{
"name": "<your_step_name_here>",
"type": "roboflow_core/vlm_as_detector@v2",
"image": "$inputs.image",
"vlm_output": [
"$steps.lmm.output"
],
"classes": [
"$steps.lmm.classes",
"$inputs.classes",
[
"class_a",
"class_b"
]
],
"model_type": [
"google-gemini",
"anthropic-claude",
"florence-2"
],
"task_type": "<block_does_not_provide_example>"
}
v1¶
Class: VLMAsDetectorBlockV1
(there are multiple versions of this block)
Source: inference.core.workflows.core_steps.formatters.vlm_as_detector.v1.VLMAsDetectorBlockV1
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
The block expects string input that would be produced by blocks exposing Large Language Models (LLMs) and Visual Language Models (VLMs). Input is parsed to object-detection prediction and returned as block output.
Accepted formats:
-
valid JSON strings
-
JSON documents wrapped with Markdown tags
Example
{"my": "json"}
Details regarding block behavior:
-
error_status
is setTrue
whenever parsing cannot be completed -
in case of multiple markdown blocks with raw JSON content - only first will be parsed
Type identifier¶
Use the following identifier in step "type"
field: roboflow_core/vlm_as_detector@v1
to add the block as
as step in your workflow.
Properties¶
Name | Type | Description | Refs |
---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
classes |
List[str] |
List of all classes used by the model, required to generate mapping between class name and class id.. | ✅ |
model_type |
str |
Type of the model that generated prediction. | ❌ |
task_type |
str |
Task type to performed by model.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow
runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to VLM as Detector
in version v1
.
- inputs:
Image Preprocessing
,Relative Static Crop
,Background Color Visualization
,Line Counter Visualization
,SIFT
,Dynamic Zone
,OpenAI
,Reference Path Visualization
,Grid Visualization
,Blur Visualization
,Image Contours
,Pixelate Visualization
,Llama 3.2 Vision
,Camera Focus
,Color Visualization
,Ellipse Visualization
,Label Visualization
,Image Slicer
,Crop Visualization
,Stitch Images
,Polygon Zone Visualization
,Dot Visualization
,Bounding Box Visualization
,Florence-2 Model
,Image Slicer
,Perspective Correction
,Size Measurement
,Dimension Collapse
,Image Blur
,Stability AI Image Generation
,Dynamic Crop
,Image Convert Grayscale
,Stability AI Inpainting
,Clip Comparison
,Clip Comparison
,Model Comparison Visualization
,Triangle Visualization
,Classification Label Visualization
,Trace Visualization
,Anthropic Claude
,Buffer
,Depth Estimation
,Corner Visualization
,SIFT Comparison
,Camera Calibration
,Absolute Static Crop
,Keypoint Visualization
,Mask Visualization
,Image Threshold
,Florence-2 Model
,Halo Visualization
,Circle Visualization
,Google Gemini
,Polygon Visualization
- outputs:
Line Counter Visualization
,Background Color Visualization
,OpenAI
,Roboflow Dataset Upload
,Gaze Detection
,Detections Stabilizer
,Detections Transformation
,Object Detection Model
,LMM For Classification
,Cache Set
,Llama 3.2 Vision
,Roboflow Dataset Upload
,Instance Segmentation Model
,Color Visualization
,Pixel Color Count
,Label Visualization
,YOLO-World Model
,Detections Filter
,Dot Visualization
,Overlap Filter
,Byte Tracker
,Object Detection Model
,Bounding Box Visualization
,Detections Consensus
,Template Matching
,Single-Label Classification Model
,Google Vision OCR
,Path Deviation
,Dynamic Crop
,Clip Comparison
,Model Comparison Visualization
,Anthropic Claude
,Corner Visualization
,SIFT Comparison
,Multi-Label Classification Model
,Byte Tracker
,Line Counter
,Detection Offset
,Keypoint Visualization
,Mask Visualization
,Image Threshold
,Florence-2 Model
,Halo Visualization
,Polygon Zone Visualization
,Google Gemini
,Polygon Visualization
,Image Preprocessing
,Instance Segmentation Model
,Twilio SMS Notification
,Dynamic Zone
,Roboflow Custom Metadata
,Reference Path Visualization
,CLIP Embedding Model
,Blur Visualization
,Detections Classes Replacement
,Pixelate Visualization
,Keypoint Detection Model
,Detections Stitch
,OpenAI
,Ellipse Visualization
,Byte Tracker
,Crop Visualization
,Line Counter
,Florence-2 Model
,Stitch OCR Detections
,CogVLM
,LMM
,Webhook Sink
,Cache Get
,Email Notification
,Perspective Correction
,Multi-Label Classification Model
,Size Measurement
,Velocity
,Time in Zone
,Stability AI Image Generation
,Image Blur
,Stability AI Inpainting
,Model Monitoring Inference Aggregator
,Triangle Visualization
,Trace Visualization
,Classification Label Visualization
,Keypoint Detection Model
,Single-Label Classification Model
,Distance Measurement
,Segment Anything 2 Model
,Path Deviation
,Time in Zone
,Circle Visualization
,Slack Notification
,Detections Merge
,Local File Sink
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
VLM as Detector
in version v1
has.
Bindings
-
input
image
(image
): The image which was the base to generate VLM prediction.vlm_output
(language_model_output
): The string with raw classification prediction to parse..classes
(list_of_values
): List of all classes used by the model, required to generate mapping between class name and class id..
-
output
error_status
(boolean
): Boolean flag.predictions
(object_detection_prediction
): Prediction with detected bounding boxes in form of sv.Detections(...) object.inference_id
(string
): String value.
Example JSON definition of step VLM as Detector
in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/vlm_as_detector@v1",
"image": "$inputs.image",
"vlm_output": [
"$steps.lmm.output"
],
"classes": [
"$steps.lmm.classes",
"$inputs.classes",
[
"class_a",
"class_b"
]
],
"model_type": [
"google-gemini",
"anthropic-claude",
"florence-2"
],
"task_type": "<block_does_not_provide_example>"
}