CogVLM¶
Class: CogVLMBlockV1
Source: inference.core.workflows.core_steps.models.foundation.cog_vlm.v1.CogVLMBlockV1
Ask a question to CogVLM, an open source vision-language model.
This model requires a GPU and can only be run on self-hosted devices, and is not available on the Roboflow Hosted API.
This model was previously part of the LMM block.
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/cog_vlm@v1to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
prompt |
str |
Text prompt to the CogVLM model. | ✅ |
json_output_format |
Dict[str, str] |
Holds dictionary that maps name of requested output field into its description. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to CogVLM in version v1.
- inputs:
Florence-2 Model,Stitch Images,Pixelate Visualization,Multi-Label Classification Model,LMM For Classification,SIFT Comparison,Keypoint Detection Model,Corner Visualization,Blur Visualization,Single-Label Classification Model,Mask Visualization,OCR Model,Perspective Correction,Object Detection Model,SIFT,Local File Sink,Model Monitoring Inference Aggregator,Polygon Zone Visualization,Polygon Visualization,VLM as Classifier,Halo Visualization,Grid Visualization,Image Slicer,Model Comparison Visualization,Trace Visualization,Camera Focus,CogVLM,Image Threshold,Google Vision OCR,Keypoint Visualization,Email Notification,Crop Visualization,Webhook Sink,Image Preprocessing,OpenAI,Roboflow Custom Metadata,Instance Segmentation Model,Roboflow Dataset Upload,Clip Comparison,Roboflow Dataset Upload,Stitch OCR Detections,Slack Notification,Anthropic Claude,Image Blur,Relative Static Crop,Circle Visualization,Image Convert Grayscale,Dot Visualization,Google Gemini,Background Color Visualization,Bounding Box Visualization,Ellipse Visualization,Image Contours,Label Visualization,Classification Label Visualization,Line Counter Visualization,Florence-2 Model,LMM,Stability AI Inpainting,Reference Path Visualization,Twilio SMS Notification,OpenAI,Dynamic Crop,VLM as Detector,Color Visualization,Triangle Visualization,CSV Formatter,Llama 3.2 Vision,Absolute Static Crop - outputs:
Pixelate Visualization,Gaze Detection,CLIP Embedding Model,Blur Visualization,OCR Model,Mask Visualization,Object Detection Model,SIFT,Line Counter,YOLO-World Model,Cache Get,Halo Visualization,Grid Visualization,Google Vision OCR,Email Notification,Camera Focus,Image Threshold,Byte Tracker,Template Matching,Image Preprocessing,Roboflow Dataset Upload,Relative Static Crop,Background Color Visualization,Bounding Box Visualization,Image Contours,Triangle Visualization,Bounding Rectangle,Absolute Static Crop,Distance Measurement,Time in Zone,Detections Stitch,Florence-2 Model,SIFT Comparison,Keypoint Detection Model,Local File Sink,Expression,Roboflow Custom Metadata,Cache Set,Crop Visualization,Clip Comparison,Dynamic Zone,SIFT Comparison,Image Convert Grayscale,Single-Label Classification Model,Identify Outliers,Florence-2 Model,Time in Zone,Path Deviation,OpenAI,Color Visualization,Pixel Color Count,Multi-Label Classification Model,Property Definition,Path Deviation,Multi-Label Classification Model,Stitch Images,LMM For Classification,First Non Empty Or Default,Keypoint Detection Model,Line Counter,Instance Segmentation Model,Rate Limiter,Single-Label Classification Model,Detections Filter,Model Monitoring Inference Aggregator,Polygon Visualization,VLM as Detector,Model Comparison Visualization,CogVLM,Keypoint Visualization,Detections Classes Replacement,Data Aggregator,Detection Offset,Slack Notification,Stitch OCR Detections,Identify Changes,Clip Comparison,Ellipse Visualization,Classification Label Visualization,Label Visualization,Line Counter Visualization,Byte Tracker,LMM,Stability AI Inpainting,Reference Path Visualization,VLM as Detector,Dynamic Crop,Dominant Color,Byte Tracker,Object Detection Model,Barcode Detection,Corner Visualization,Perspective Correction,Cosine Similarity,Polygon Zone Visualization,VLM as Classifier,Dimension Collapse,Continue If,Twilio SMS Notification,Trace Visualization,Detections Consensus,Webhook Sink,Size Measurement,OpenAI,Image Slicer,Instance Segmentation Model,Buffer,Roboflow Dataset Upload,VLM as Classifier,Anthropic Claude,Image Blur,Circle Visualization,Dot Visualization,Google Gemini,QR Code Detection,Segment Anything 2 Model,JSON Parser,Detections Stabilizer,Delta Filter,CSV Formatter,Llama 3.2 Vision,Detections Transformation
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
CogVLM in version v1 has.
Bindings
-
input
-
output
parent_id(parent_id): Identifier of parent for step output.root_parent_id(parent_id): Identifier of parent for step output.image(image_metadata): Dictionary with image metadata required by supervision.structured_output(dictionary): Dictionary.raw_output(string): String value.*(*): Equivalent of any element.
Example JSON definition of step CogVLM in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/cog_vlm@v1",
"images": "$inputs.image",
"prompt": "my prompt",
"json_output_format": {
"count": "number of cats in the picture"
}
}