VLM as Detector¶
v2¶
Class: VLMAsDetectorBlockV2 (there are multiple versions of this block)
Source: inference.core.workflows.core_steps.formatters.vlm_as_detector.v2.VLMAsDetectorBlockV2
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
The block expects string input that would be produced by blocks exposing Large Language Models (LLMs) and Visual Language Models (VLMs). Input is parsed to object-detection prediction and returned as block output.
Accepted formats:
-
valid JSON strings
-
JSON documents wrapped with Markdown tags
Example
{"my": "json"}
Details regarding block behavior:
-
error_statusis setTruewhenever parsing cannot be completed -
in case of multiple markdown blocks with raw JSON content - only first will be parsed
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/vlm_as_detector@v2to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
classes |
List[str] |
List of all classes used by the model, required to generate mapping between class name and class id.. | ✅ |
model_type |
str |
Type of the model that generated prediction. | ❌ |
task_type |
str |
Task type to performed by model.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to VLM as Detector in version v2.
- inputs:
Florence-2 Model,Stitch Images,Pixelate Visualization,SIFT Comparison,Corner Visualization,Blur Visualization,Mask Visualization,Perspective Correction,SIFT,Polygon Zone Visualization,Polygon Visualization,Halo Visualization,Grid Visualization,Dimension Collapse,Image Slicer,Model Comparison Visualization,Trace Visualization,Camera Focus,OpenAI,Image Threshold,Size Measurement,Keypoint Visualization,Crop Visualization,Image Preprocessing,Buffer,Clip Comparison,Anthropic Claude,Dynamic Zone,Image Blur,Relative Static Crop,Circle Visualization,Image Convert Grayscale,Dot Visualization,Google Gemini,Background Color Visualization,Clip Comparison,Bounding Box Visualization,Ellipse Visualization,Image Contours,Label Visualization,Classification Label Visualization,Line Counter Visualization,Florence-2 Model,Stability AI Inpainting,Reference Path Visualization,Dynamic Crop,Color Visualization,Triangle Visualization,Llama 3.2 Vision,Absolute Static Crop - outputs:
Multi-Label Classification Model,Pixelate Visualization,Path Deviation,Keypoint Detection Model,Gaze Detection,Line Counter,Instance Segmentation Model,Single-Label Classification Model,Blur Visualization,Mask Visualization,Object Detection Model,Line Counter,Detections Filter,Model Monitoring Inference Aggregator,Polygon Visualization,Halo Visualization,Email Notification,Model Comparison Visualization,Byte Tracker,Keypoint Visualization,Detections Classes Replacement,Template Matching,Detection Offset,Slack Notification,Roboflow Dataset Upload,Stitch OCR Detections,Background Color Visualization,Bounding Box Visualization,Label Visualization,Classification Label Visualization,Ellipse Visualization,Line Counter Visualization,Byte Tracker,Reference Path Visualization,Dynamic Crop,Byte Tracker,Triangle Visualization,Object Detection Model,Distance Measurement,Time in Zone,Florence-2 Model,Detections Stitch,SIFT Comparison,Keypoint Detection Model,Corner Visualization,Perspective Correction,Polygon Zone Visualization,Twilio SMS Notification,Trace Visualization,Webhook Sink,Detections Consensus,Size Measurement,Roboflow Custom Metadata,Instance Segmentation Model,Crop Visualization,Roboflow Dataset Upload,Dot Visualization,Circle Visualization,Segment Anything 2 Model,Single-Label Classification Model,Time in Zone,Florence-2 Model,Detections Stabilizer,Path Deviation,Color Visualization,Multi-Label Classification Model,Detections Transformation
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
VLM as Detector in version v2 has.
Bindings
-
input
image(image): The image which was the base to generate VLM prediction.vlm_output(language_model_output): The string with raw classification prediction to parse..classes(list_of_values): List of all classes used by the model, required to generate mapping between class name and class id..
-
output
error_status(boolean): Boolean flag.predictions(object_detection_prediction): Prediction with detected bounding boxes in form of sv.Detections(...) object.inference_id(inference_id): Inference identifier.
Example JSON definition of step VLM as Detector in version v2
{
"name": "<your_step_name_here>",
"type": "roboflow_core/vlm_as_detector@v2",
"image": "$inputs.image",
"vlm_output": [
"$steps.lmm.output"
],
"classes": [
"$steps.lmm.classes",
"$inputs.classes",
[
"class_a",
"class_b"
]
],
"model_type": [
"google-gemini",
"anthropic-claude",
"florence-2"
],
"task_type": "<block_does_not_provide_example>"
}
v1¶
Class: VLMAsDetectorBlockV1 (there are multiple versions of this block)
Source: inference.core.workflows.core_steps.formatters.vlm_as_detector.v1.VLMAsDetectorBlockV1
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
The block expects string input that would be produced by blocks exposing Large Language Models (LLMs) and Visual Language Models (VLMs). Input is parsed to object-detection prediction and returned as block output.
Accepted formats:
-
valid JSON strings
-
JSON documents wrapped with Markdown tags
Example
{"my": "json"}
Details regarding block behavior:
-
error_statusis setTruewhenever parsing cannot be completed -
in case of multiple markdown blocks with raw JSON content - only first will be parsed
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/vlm_as_detector@v1to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
classes |
List[str] |
List of all classes used by the model, required to generate mapping between class name and class id.. | ✅ |
model_type |
str |
Type of the model that generated prediction. | ❌ |
task_type |
str |
Task type to performed by model.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to VLM as Detector in version v1.
- inputs:
Florence-2 Model,Stitch Images,Pixelate Visualization,SIFT Comparison,Corner Visualization,Blur Visualization,Mask Visualization,Perspective Correction,SIFT,Polygon Zone Visualization,Polygon Visualization,Halo Visualization,Grid Visualization,Dimension Collapse,Image Slicer,Model Comparison Visualization,Trace Visualization,Camera Focus,OpenAI,Image Threshold,Size Measurement,Keypoint Visualization,Crop Visualization,Image Preprocessing,Buffer,Clip Comparison,Anthropic Claude,Dynamic Zone,Image Blur,Relative Static Crop,Circle Visualization,Image Convert Grayscale,Dot Visualization,Google Gemini,Background Color Visualization,Clip Comparison,Bounding Box Visualization,Ellipse Visualization,Image Contours,Label Visualization,Classification Label Visualization,Line Counter Visualization,Florence-2 Model,Stability AI Inpainting,Reference Path Visualization,Dynamic Crop,Color Visualization,Triangle Visualization,Llama 3.2 Vision,Absolute Static Crop - outputs:
Multi-Label Classification Model,Pixelate Visualization,Path Deviation,LMM For Classification,Keypoint Detection Model,Gaze Detection,Line Counter,Instance Segmentation Model,CLIP Embedding Model,Single-Label Classification Model,Blur Visualization,Mask Visualization,Object Detection Model,Line Counter,Detections Filter,YOLO-World Model,Model Monitoring Inference Aggregator,Cache Get,Polygon Visualization,Halo Visualization,Google Vision OCR,Email Notification,Model Comparison Visualization,CogVLM,Byte Tracker,Image Threshold,Keypoint Visualization,Detections Classes Replacement,Template Matching,Image Preprocessing,Detection Offset,Slack Notification,Roboflow Dataset Upload,Stitch OCR Detections,Background Color Visualization,Bounding Box Visualization,Label Visualization,Classification Label Visualization,Ellipse Visualization,Line Counter Visualization,Byte Tracker,LMM,Reference Path Visualization,Stability AI Inpainting,Dynamic Crop,Byte Tracker,Triangle Visualization,Object Detection Model,Distance Measurement,Time in Zone,Florence-2 Model,Detections Stitch,SIFT Comparison,Keypoint Detection Model,Corner Visualization,Perspective Correction,Local File Sink,Polygon Zone Visualization,Twilio SMS Notification,Trace Visualization,Webhook Sink,Detections Consensus,Size Measurement,Roboflow Custom Metadata,OpenAI,Cache Set,Instance Segmentation Model,Crop Visualization,Roboflow Dataset Upload,Clip Comparison,Anthropic Claude,Image Blur,Dot Visualization,Circle Visualization,Google Gemini,Segment Anything 2 Model,Single-Label Classification Model,Time in Zone,Florence-2 Model,Detections Stabilizer,Path Deviation,OpenAI,Color Visualization,Multi-Label Classification Model,Pixel Color Count,Llama 3.2 Vision,Detections Transformation
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
VLM as Detector in version v1 has.
Bindings
-
input
image(image): The image which was the base to generate VLM prediction.vlm_output(language_model_output): The string with raw classification prediction to parse..classes(list_of_values): List of all classes used by the model, required to generate mapping between class name and class id..
-
output
error_status(boolean): Boolean flag.predictions(object_detection_prediction): Prediction with detected bounding boxes in form of sv.Detections(...) object.inference_id(string): String value.
Example JSON definition of step VLM as Detector in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/vlm_as_detector@v1",
"image": "$inputs.image",
"vlm_output": [
"$steps.lmm.output"
],
"classes": [
"$steps.lmm.classes",
"$inputs.classes",
[
"class_a",
"class_b"
]
],
"model_type": [
"google-gemini",
"anthropic-claude",
"florence-2"
],
"task_type": "<block_does_not_provide_example>"
}