LMM¶
Class: LMMBlockV1
Source: inference.core.workflows.core_steps.models.foundation.lmm.v1.LMMBlockV1
Ask a question to a Large Multimodal Model (LMM) with an image and text.
You can specify arbitrary text prompts to an LMMBlock.
The LLMBlock supports two LMMs:
- OpenAI's GPT-4 with Vision, and;
- CogVLM.
You need to provide your OpenAI API key to use the GPT-4 with Vision model. You do not need to provide an API key to use CogVLM.
If you want to classify an image into one or more categories, we recommend using the dedicated LMMForClassificationBlock.
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/lmm@v1to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
prompt |
str |
Holds unconstrained text prompt to LMM mode. | ✅ |
lmm_type |
str |
Type of LMM to be used. | ✅ |
lmm_config |
LMMConfig |
Configuration of LMM. | ❌ |
remote_api_key |
str |
Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v and do not require additional API key for CogVLM calls.. |
✅ |
json_output |
Dict[str, str] |
Holds dictionary that maps name of requested output field into its description. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to LMM in version v1.
- inputs:
Stitch Images,Pixelate Visualization,Multi-Label Classification Model,LMM For Classification,Blur Visualization,Single-Label Classification Model,Mask Visualization,OCR Model,Object Detection Model,SIFT,Model Monitoring Inference Aggregator,Polygon Visualization,Halo Visualization,Grid Visualization,Google Vision OCR,Model Comparison Visualization,Email Notification,Camera Focus,CogVLM,Image Threshold,Keypoint Visualization,Image Preprocessing,Roboflow Dataset Upload,Slack Notification,Stitch OCR Detections,Relative Static Crop,Background Color Visualization,Bounding Box Visualization,Ellipse Visualization,Image Contours,Label Visualization,Classification Label Visualization,Line Counter Visualization,LMM,Stability AI Inpainting,Reference Path Visualization,VLM as Detector,Dynamic Crop,Triangle Visualization,Absolute Static Crop,Florence-2 Model,SIFT Comparison,Keypoint Detection Model,Corner Visualization,Perspective Correction,Local File Sink,Polygon Zone Visualization,VLM as Classifier,Image Slicer,Trace Visualization,Webhook Sink,OpenAI,Twilio SMS Notification,Roboflow Custom Metadata,Crop Visualization,Instance Segmentation Model,Roboflow Dataset Upload,Clip Comparison,Anthropic Claude,Image Blur,Circle Visualization,Image Convert Grayscale,Dot Visualization,Google Gemini,Florence-2 Model,OpenAI,Color Visualization,CSV Formatter,Llama 3.2 Vision - outputs:
Pixelate Visualization,Gaze Detection,CLIP Embedding Model,Blur Visualization,OCR Model,Mask Visualization,Object Detection Model,SIFT,Line Counter,YOLO-World Model,Cache Get,Halo Visualization,Grid Visualization,Google Vision OCR,Email Notification,Camera Focus,Image Threshold,Byte Tracker,Template Matching,Image Preprocessing,Roboflow Dataset Upload,Relative Static Crop,Background Color Visualization,Bounding Box Visualization,Image Contours,Triangle Visualization,Bounding Rectangle,Absolute Static Crop,Distance Measurement,Time in Zone,Detections Stitch,Florence-2 Model,SIFT Comparison,Keypoint Detection Model,Local File Sink,Expression,Roboflow Custom Metadata,Cache Set,Crop Visualization,Clip Comparison,Dynamic Zone,SIFT Comparison,Image Convert Grayscale,Single-Label Classification Model,Identify Outliers,Florence-2 Model,Time in Zone,Path Deviation,OpenAI,Color Visualization,Pixel Color Count,Multi-Label Classification Model,Property Definition,Path Deviation,Multi-Label Classification Model,Stitch Images,LMM For Classification,First Non Empty Or Default,Keypoint Detection Model,Line Counter,Instance Segmentation Model,Rate Limiter,Single-Label Classification Model,Detections Filter,Model Monitoring Inference Aggregator,Polygon Visualization,VLM as Detector,Model Comparison Visualization,CogVLM,Keypoint Visualization,Detections Classes Replacement,Data Aggregator,Detection Offset,Slack Notification,Stitch OCR Detections,Identify Changes,Clip Comparison,Ellipse Visualization,Classification Label Visualization,Label Visualization,Line Counter Visualization,Byte Tracker,LMM,Stability AI Inpainting,Reference Path Visualization,VLM as Detector,Dynamic Crop,Dominant Color,Byte Tracker,Object Detection Model,Barcode Detection,Corner Visualization,Perspective Correction,Cosine Similarity,Polygon Zone Visualization,VLM as Classifier,Dimension Collapse,Continue If,Twilio SMS Notification,Trace Visualization,Detections Consensus,Webhook Sink,Size Measurement,OpenAI,Image Slicer,Instance Segmentation Model,Buffer,Roboflow Dataset Upload,VLM as Classifier,Anthropic Claude,Image Blur,Circle Visualization,Dot Visualization,Google Gemini,QR Code Detection,Segment Anything 2 Model,JSON Parser,Detections Stabilizer,Delta Filter,CSV Formatter,Llama 3.2 Vision,Detections Transformation
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
LMM in version v1 has.
Bindings
-
input
images(image): The image to infer on.prompt(string): Holds unconstrained text prompt to LMM mode.lmm_type(string): Type of LMM to be used.remote_api_key(Union[secret,string]): Holds API key required to call LMM model - in current state of development, we require OpenAI key whenlmm_type=gpt_4vand do not require additional API key for CogVLM calls..
-
output
parent_id(parent_id): Identifier of parent for step output.root_parent_id(parent_id): Identifier of parent for step output.image(image_metadata): Dictionary with image metadata required by supervision.structured_output(dictionary): Dictionary.raw_output(string): String value.*(*): Equivalent of any element.
Example JSON definition of step LMM in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/lmm@v1",
"images": "$inputs.image",
"prompt": "my prompt",
"lmm_type": "gpt_4v",
"lmm_config": {
"gpt_image_detail": "low",
"gpt_model_version": "gpt-4o",
"max_tokens": 200
},
"remote_api_key": "xxx-xxx",
"json_output": {
"count": "number of cats in the picture"
}
}