VLM as Detector¶
v2¶
Class: VLMAsDetectorBlockV2 (there are multiple versions of this block)
Source: inference.core.workflows.core_steps.formatters.vlm_as_detector.v2.VLMAsDetectorBlockV2
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
The block expects string input that would be produced by blocks exposing Large Language Models (LLMs) and Visual Language Models (VLMs). Input is parsed to object-detection prediction and returned as block output.
Accepted formats:
-
valid JSON strings
-
JSON documents wrapped with Markdown tags
Example
{"my": "json"}
Details regarding block behavior:
-
error_statusis setTruewhenever parsing cannot be completed -
in case of multiple markdown blocks with raw JSON content - only first will be parsed
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/vlm_as_detector@v2to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
classes |
List[str] |
List of all classes used by the model, required to generate mapping between class name and class id.. | ✅ |
model_type |
str |
Type of the model that generated prediction. | ❌ |
task_type |
str |
Task type to performed by model.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to VLM as Detector in version v2.
- inputs:
Image Slicer,Stability AI Inpainting,Pixelate Visualization,Perspective Correction,Clip Comparison,OpenAI,Relative Static Crop,Dimension Collapse,SIFT Comparison,Google Gemini,Grid Visualization,Ellipse Visualization,SIFT,Model Comparison Visualization,Halo Visualization,Image Contours,Crop Visualization,Absolute Static Crop,Camera Focus,Image Blur,Trace Visualization,Circle Visualization,Buffer,Image Preprocessing,Background Color Visualization,Dot Visualization,Clip Comparison,Polygon Zone Visualization,Size Measurement,Florence-2 Model,Classification Label Visualization,Bounding Box Visualization,Corner Visualization,Llama 3.2 Vision,Image Slicer,Florence-2 Model,Dynamic Crop,Reference Path Visualization,Label Visualization,Mask Visualization,Stitch Images,Triangle Visualization,Stability AI Image Generation,Image Threshold,Line Counter Visualization,Dynamic Zone,Keypoint Visualization,Color Visualization,Blur Visualization,Anthropic Claude,Image Convert Grayscale,Polygon Visualization - outputs:
Segment Anything 2 Model,Perspective Correction,Object Detection Model,Roboflow Custom Metadata,Object Detection Model,SIFT Comparison,Detection Offset,Ellipse Visualization,Multi-Label Classification Model,Trace Visualization,Multi-Label Classification Model,Dot Visualization,Roboflow Dataset Upload,Polygon Zone Visualization,Gaze Detection,Classification Label Visualization,Corner Visualization,Byte Tracker,Dynamic Crop,Reference Path Visualization,Line Counter,Label Visualization,Detections Stabilizer,Mask Visualization,Triangle Visualization,Template Matching,Line Counter Visualization,Detections Transformation,Model Monitoring Inference Aggregator,Time in Zone,Blur Visualization,Line Counter,Instance Segmentation Model,Webhook Sink,Time in Zone,Instance Segmentation Model,Slack Notification,Detections Filter,Stitch OCR Detections,Pixelate Visualization,Path Deviation,Detections Consensus,Twilio SMS Notification,Keypoint Detection Model,Roboflow Dataset Upload,Model Comparison Visualization,Halo Visualization,Crop Visualization,Byte Tracker,Distance Measurement,Circle Visualization,Velocity,Keypoint Detection Model,Background Color Visualization,Size Measurement,Florence-2 Model,Single-Label Classification Model,Bounding Box Visualization,Florence-2 Model,Byte Tracker,Detections Stitch,Keypoint Visualization,Email Notification,Color Visualization,Path Deviation,Single-Label Classification Model,Detections Classes Replacement,Polygon Visualization
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
VLM as Detector in version v2 has.
Bindings
-
input
image(image): The image which was the base to generate VLM prediction.vlm_output(language_model_output): The string with raw classification prediction to parse..classes(list_of_values): List of all classes used by the model, required to generate mapping between class name and class id..
-
output
error_status(boolean): Boolean flag.predictions(object_detection_prediction): Prediction with detected bounding boxes in form of sv.Detections(...) object.inference_id(inference_id): Inference identifier.
Example JSON definition of step VLM as Detector in version v2
{
"name": "<your_step_name_here>",
"type": "roboflow_core/vlm_as_detector@v2",
"image": "$inputs.image",
"vlm_output": [
"$steps.lmm.output"
],
"classes": [
"$steps.lmm.classes",
"$inputs.classes",
[
"class_a",
"class_b"
]
],
"model_type": [
"google-gemini",
"anthropic-claude",
"florence-2"
],
"task_type": "<block_does_not_provide_example>"
}
v1¶
Class: VLMAsDetectorBlockV1 (there are multiple versions of this block)
Source: inference.core.workflows.core_steps.formatters.vlm_as_detector.v1.VLMAsDetectorBlockV1
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
The block expects string input that would be produced by blocks exposing Large Language Models (LLMs) and Visual Language Models (VLMs). Input is parsed to object-detection prediction and returned as block output.
Accepted formats:
-
valid JSON strings
-
JSON documents wrapped with Markdown tags
Example
{"my": "json"}
Details regarding block behavior:
-
error_statusis setTruewhenever parsing cannot be completed -
in case of multiple markdown blocks with raw JSON content - only first will be parsed
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/vlm_as_detector@v1to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
classes |
List[str] |
List of all classes used by the model, required to generate mapping between class name and class id.. | ✅ |
model_type |
str |
Type of the model that generated prediction. | ❌ |
task_type |
str |
Task type to performed by model.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to VLM as Detector in version v1.
- inputs:
Image Slicer,Stability AI Inpainting,Pixelate Visualization,Perspective Correction,Clip Comparison,OpenAI,Relative Static Crop,Dimension Collapse,SIFT Comparison,Google Gemini,Grid Visualization,Ellipse Visualization,SIFT,Model Comparison Visualization,Halo Visualization,Image Contours,Crop Visualization,Absolute Static Crop,Camera Focus,Image Blur,Trace Visualization,Circle Visualization,Buffer,Image Preprocessing,Background Color Visualization,Dot Visualization,Clip Comparison,Polygon Zone Visualization,Size Measurement,Florence-2 Model,Classification Label Visualization,Bounding Box Visualization,Corner Visualization,Llama 3.2 Vision,Image Slicer,Florence-2 Model,Dynamic Crop,Reference Path Visualization,Label Visualization,Mask Visualization,Stitch Images,Triangle Visualization,Stability AI Image Generation,Image Threshold,Line Counter Visualization,Dynamic Zone,Keypoint Visualization,Color Visualization,Blur Visualization,Anthropic Claude,Image Convert Grayscale,Polygon Visualization - outputs:
Segment Anything 2 Model,Cache Get,Stability AI Inpainting,Clip Comparison,Perspective Correction,Cache Set,Object Detection Model,Roboflow Custom Metadata,Object Detection Model,SIFT Comparison,Detection Offset,CogVLM,Ellipse Visualization,Multi-Label Classification Model,OpenAI,Trace Visualization,CLIP Embedding Model,Multi-Label Classification Model,Dot Visualization,Google Vision OCR,Roboflow Dataset Upload,Polygon Zone Visualization,Gaze Detection,Classification Label Visualization,Corner Visualization,Byte Tracker,Llama 3.2 Vision,Dynamic Crop,Reference Path Visualization,Line Counter,Label Visualization,Detections Stabilizer,Mask Visualization,Triangle Visualization,Template Matching,Line Counter Visualization,Detections Transformation,Model Monitoring Inference Aggregator,Time in Zone,Blur Visualization,Line Counter,Anthropic Claude,Instance Segmentation Model,Webhook Sink,Time in Zone,Instance Segmentation Model,Slack Notification,Detections Filter,Stitch OCR Detections,Pixelate Visualization,Path Deviation,OpenAI,Detections Consensus,Twilio SMS Notification,Keypoint Detection Model,Roboflow Dataset Upload,Google Gemini,Model Comparison Visualization,Halo Visualization,Crop Visualization,Byte Tracker,Image Blur,Distance Measurement,Circle Visualization,Velocity,Keypoint Detection Model,Image Preprocessing,Background Color Visualization,Pixel Color Count,Size Measurement,Florence-2 Model,Single-Label Classification Model,Bounding Box Visualization,Florence-2 Model,Byte Tracker,Local File Sink,LMM For Classification,Stability AI Image Generation,Image Threshold,Detections Stitch,LMM,Keypoint Visualization,Email Notification,Color Visualization,Path Deviation,Single-Label Classification Model,YOLO-World Model,Detections Classes Replacement,Polygon Visualization
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
VLM as Detector in version v1 has.
Bindings
-
input
image(image): The image which was the base to generate VLM prediction.vlm_output(language_model_output): The string with raw classification prediction to parse..classes(list_of_values): List of all classes used by the model, required to generate mapping between class name and class id..
-
output
error_status(boolean): Boolean flag.predictions(object_detection_prediction): Prediction with detected bounding boxes in form of sv.Detections(...) object.inference_id(string): String value.
Example JSON definition of step VLM as Detector in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/vlm_as_detector@v1",
"image": "$inputs.image",
"vlm_output": [
"$steps.lmm.output"
],
"classes": [
"$steps.lmm.classes",
"$inputs.classes",
[
"class_a",
"class_b"
]
],
"model_type": [
"google-gemini",
"anthropic-claude",
"florence-2"
],
"task_type": "<block_does_not_provide_example>"
}