VLM as Detector

v2

Class: VLMAsDetectorBlockV2 (there are multiple versions of this block)

Source: inference.core.workflows.core_steps.formatters.vlm_as_detector.v2.VLMAsDetectorBlockV2

Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning

The block expects string input that would be produced by blocks exposing Large Language Models (LLMs) and Visual Language Models (VLMs). Input is parsed to object-detection prediction and returned as block output.

Accepted formats:

• valid JSON strings

• JSON documents wrapped with Markdown tags

Example

{"my": "json"}

Details regarding block behavior:

• error_status is set True whenever parsing cannot be completed

• in case of multiple markdown blocks with raw JSON content - only first will be parsed

Type identifier

Use the following identifier in step "type" field: roboflow_core/vlm_as_detector@v2to add the block as as step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
classes	List[str]	List of all classes used by the model, required to generate mapping between class name and class id..	✅
model_type	str	Type of the model that generated prediction.	❌
task_type	str	Task type to performed by model..	❌

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to VLM as Detector in version v2.

• inputs: OpenAI, Size Measurement, Buffer, Image Convert Grayscale, Florence-2 Model, Absolute Static Crop, Relative Static Crop, Label Visualization, Line Counter Visualization, Background Color Visualization, Stitch Images, Camera Focus, Image Contours, Image Preprocessing, Image Slicer, Reference Path Visualization, SIFT Comparison, Triangle Visualization, Grid Visualization, Polygon Zone Visualization, Keypoint Visualization, Depth Estimation, Llama 3.2 Vision, Dynamic Zone, Bounding Box Visualization, Image Blur, Perspective Correction, Halo Visualization, Google Gemini, Ellipse Visualization, Color Visualization, Crop Visualization, Clip Comparison, Dot Visualization, Pixelate Visualization, Model Comparison Visualization, Classification Label Visualization, Camera Calibration, Dimension Collapse, Stability AI Image Generation, Polygon Visualization, Trace Visualization, Corner Visualization, Image Threshold, Blur Visualization, Image Slicer, Clip Comparison, Stability AI Inpainting, Mask Visualization, SIFT, Anthropic Claude, Florence-2 Model, Circle Visualization, Dynamic Crop
• outputs: Size Measurement, Roboflow Custom Metadata, Detection Offset, Multi-Label Classification Model, Distance Measurement, Line Counter Visualization, Gaze Detection, Detections Classes Replacement, Background Color Visualization, Keypoint Detection Model, Reference Path Visualization, Instance Segmentation Model, SIFT Comparison, Object Detection Model, Triangle Visualization, Path Deviation, Line Counter, Detections Stabilizer, Detections Transformation, Multi-Label Classification Model, Byte Tracker, Roboflow Dataset Upload, Perspective Correction, Object Detection Model, Crop Visualization, Webhook Sink, Dot Visualization, Detections Filter, Model Comparison Visualization, Email Notification, Classification Label Visualization, Instance Segmentation Model, Slack Notification, Detections Merge, Time in Zone, Time in Zone, Trace Visualization, Corner Visualization, Line Counter, Blur Visualization, Keypoint Detection Model, Circle Visualization, Overlap Filter, Path Deviation, Florence-2 Model, Twilio SMS Notification, Label Visualization, Detections Stitch, Template Matching, Byte Tracker, Polygon Zone Visualization, Keypoint Visualization, Stitch OCR Detections, Bounding Box Visualization, Halo Visualization, Ellipse Visualization, Color Visualization, Pixelate Visualization, Single-Label Classification Model, Velocity, Roboflow Dataset Upload, Polygon Visualization, Segment Anything 2 Model, Single-Label Classification Model, Model Monitoring Inference Aggregator, Mask Visualization, Florence-2 Model, Byte Tracker, Dynamic Crop, Detections Consensus

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds VLM as Detector in version v2 has.

Bindings

• input

  • image (image): The image which was the base to generate VLM prediction.
  • vlm_output (language_model_output): The string with raw classification prediction to parse..
  • classes (list_of_values): List of all classes used by the model, required to generate mapping between class name and class id..

• output

  • error_status (boolean): Boolean flag.
  • predictions (object_detection_prediction): Prediction with detected bounding boxes in form of sv.Detections(...) object.
  • inference_id (inference_id): Inference identifier.

Example JSON definition of step VLM as Detector in version v2

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/vlm_as_detector@v2",
    "image": "$inputs.image",
    "vlm_output": [
        "$steps.lmm.output"
    ],
    "classes": [
        "$steps.lmm.classes",
        "$inputs.classes",
        [
            "class_a",
            "class_b"
        ]
    ],
    "model_type": [
        "google-gemini",
        "anthropic-claude",
        "florence-2"
    ],
    "task_type": "<block_does_not_provide_example>"
}

v1

Class: VLMAsDetectorBlockV1 (there are multiple versions of this block)

Source: inference.core.workflows.core_steps.formatters.vlm_as_detector.v1.VLMAsDetectorBlockV1

Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning

The block expects string input that would be produced by blocks exposing Large Language Models (LLMs) and Visual Language Models (VLMs). Input is parsed to object-detection prediction and returned as block output.

Accepted formats:

• valid JSON strings

• JSON documents wrapped with Markdown tags

Example

{"my": "json"}

Details regarding block behavior:

• error_status is set True whenever parsing cannot be completed

• in case of multiple markdown blocks with raw JSON content - only first will be parsed

Type identifier

Use the following identifier in step "type" field: roboflow_core/vlm_as_detector@v1to add the block as as step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
classes	List[str]	List of all classes used by the model, required to generate mapping between class name and class id..	✅
model_type	str	Type of the model that generated prediction.	❌
task_type	str	Task type to performed by model..	❌

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to VLM as Detector in version v1.

• inputs: OpenAI, Size Measurement, Buffer, Image Convert Grayscale, Florence-2 Model, Absolute Static Crop, Relative Static Crop, Label Visualization, Line Counter Visualization, Background Color Visualization, Stitch Images, Camera Focus, Image Contours, Image Preprocessing, Image Slicer, Reference Path Visualization, SIFT Comparison, Triangle Visualization, Grid Visualization, Polygon Zone Visualization, Keypoint Visualization, Depth Estimation, Llama 3.2 Vision, Dynamic Zone, Bounding Box Visualization, Image Blur, Perspective Correction, Halo Visualization, Google Gemini, Ellipse Visualization, Color Visualization, Crop Visualization, Clip Comparison, Dot Visualization, Pixelate Visualization, Model Comparison Visualization, Classification Label Visualization, Camera Calibration, Dimension Collapse, Stability AI Image Generation, Polygon Visualization, Trace Visualization, Corner Visualization, Image Threshold, Blur Visualization, Image Slicer, Clip Comparison, Stability AI Inpainting, Mask Visualization, SIFT, Anthropic Claude, Florence-2 Model, Circle Visualization, Dynamic Crop
• outputs: Size Measurement, Roboflow Custom Metadata, Detection Offset, LMM, Multi-Label Classification Model, Distance Measurement, Line Counter Visualization, Gaze Detection, Detections Classes Replacement, Background Color Visualization, Keypoint Detection Model, Reference Path Visualization, Instance Segmentation Model, SIFT Comparison, Object Detection Model, Triangle Visualization, Path Deviation, Line Counter, Detections Stabilizer, Detections Transformation, Multi-Label Classification Model, Byte Tracker, Llama 3.2 Vision, Roboflow Dataset Upload, Google Vision OCR, Clip Comparison, Perspective Correction, Object Detection Model, Crop Visualization, Webhook Sink, Dot Visualization, Detections Filter, Model Comparison Visualization, Email Notification, Classification Label Visualization, Instance Segmentation Model, Slack Notification, Detections Merge, Stability AI Image Generation, Cache Set, Time in Zone, Time in Zone, Trace Visualization, Corner Visualization, Line Counter, Image Threshold, Blur Visualization, Local File Sink, CogVLM, Stability AI Inpainting, Keypoint Detection Model, Circle Visualization, Overlap Filter, OpenAI, Path Deviation, Florence-2 Model, Twilio SMS Notification, Label Visualization, Image Preprocessing, Detections Stitch, Template Matching, Byte Tracker, Polygon Zone Visualization, Keypoint Visualization, LMM For Classification, Stitch OCR Detections, Bounding Box Visualization, Image Blur, OpenAI, Halo Visualization, CLIP Embedding Model, Ellipse Visualization, Color Visualization, Google Gemini, Pixelate Visualization, Single-Label Classification Model, Pixel Color Count, YOLO-World Model, Velocity, Roboflow Dataset Upload, Polygon Visualization, Segment Anything 2 Model, Single-Label Classification Model, Cache Get, Model Monitoring Inference Aggregator, Mask Visualization, Anthropic Claude, Florence-2 Model, Byte Tracker, Dynamic Crop, Detections Consensus

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds VLM as Detector in version v1 has.

Bindings

• input

  • image (image): The image which was the base to generate VLM prediction.
  • vlm_output (language_model_output): The string with raw classification prediction to parse..
  • classes (list_of_values): List of all classes used by the model, required to generate mapping between class name and class id..

• output

  • error_status (boolean): Boolean flag.
  • predictions (object_detection_prediction): Prediction with detected bounding boxes in form of sv.Detections(...) object.
  • inference_id (string): String value.

Example JSON definition of step VLM as Detector in version v1

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/vlm_as_detector@v1",
    "image": "$inputs.image",
    "vlm_output": [
        "$steps.lmm.output"
    ],
    "classes": [
        "$steps.lmm.classes",
        "$inputs.classes",
        [
            "class_a",
            "class_b"
        ]
    ],
    "model_type": [
        "google-gemini",
        "anthropic-claude",
        "florence-2"
    ],
    "task_type": "<block_does_not_provide_example>"
}