LMM¶
Class: LMMBlockV1
Source: inference.core.workflows.core_steps.models.foundation.lmm.v1.LMMBlockV1
Ask a question to a Large Multimodal Model (LMM) with an image and text.
You can specify arbitrary text prompts to an LMMBlock.
The LLMBlock supports two LMMs:
- OpenAI's GPT-4 with Vision;
You need to provide your OpenAI API key to use the GPT-4 with Vision model.
If you want to classify an image into one or more categories, we recommend using the dedicated LMMForClassificationBlock.
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/lmm@v1to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
prompt |
str |
Holds unconstrained text prompt to LMM mode. | ✅ |
lmm_type |
str |
Type of LMM to be used. | ✅ |
lmm_config |
LMMConfig |
Configuration of LMM. | ❌ |
remote_api_key |
str |
Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v.. |
✅ |
json_output |
Dict[str, str] |
Holds dictionary that maps name of requested output field into its description. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to LMM in version v1.
- inputs:
Circle Visualization,Background Color Visualization,Corner Visualization,Twilio SMS Notification,Slack Notification,VLM as Detector,LMM,Polygon Zone Visualization,Camera Focus,Image Slicer,Image Blur,Dot Visualization,Google Gemini,Roboflow Dataset Upload,Stability AI Inpainting,Pixelate Visualization,OpenAI,Image Convert Grayscale,Absolute Static Crop,Stability AI Image Generation,Webhook Sink,Color Visualization,Image Threshold,Halo Visualization,Polygon Visualization,VLM as Classifier,CogVLM,Camera Calibration,Email Notification,Classification Label Visualization,Single-Label Classification Model,Llama 3.2 Vision,Google Vision OCR,Roboflow Dataset Upload,Ellipse Visualization,Bounding Box Visualization,Object Detection Model,Line Counter Visualization,Image Preprocessing,Trace Visualization,Label Visualization,Local File Sink,Image Slicer,Anthropic Claude,Crop Visualization,OCR Model,Relative Static Crop,Model Comparison Visualization,Stitch OCR Detections,Perspective Correction,OpenAI,Mask Visualization,Clip Comparison,Dynamic Crop,CSV Formatter,Florence-2 Model,Instance Segmentation Model,Keypoint Detection Model,Image Contours,SIFT,Reference Path Visualization,Florence-2 Model,Triangle Visualization,Model Monitoring Inference Aggregator,SIFT Comparison,Keypoint Visualization,Multi-Label Classification Model,LMM For Classification,Roboflow Custom Metadata,Grid Visualization,Stitch Images,Blur Visualization - outputs:
Corner Visualization,Slack Notification,VLM as Detector,VLM as Classifier,Polygon Zone Visualization,Image Slicer,Cache Set,Dot Visualization,Path Deviation,Roboflow Dataset Upload,Single-Label Classification Model,Dominant Color,Stability AI Image Generation,Image Convert Grayscale,Halo Visualization,Data Aggregator,Detections Classes Replacement,Dynamic Zone,CogVLM,Email Notification,Camera Calibration,Object Detection Model,Cosine Similarity,Single-Label Classification Model,Llama 3.2 Vision,Byte Tracker,Ellipse Visualization,Size Measurement,Pixel Color Count,Bounding Box Visualization,Line Counter Visualization,Image Preprocessing,Label Visualization,Local File Sink,Image Slicer,Detections Transformation,Anthropic Claude,Crop Visualization,Detections Stitch,OCR Model,Model Comparison Visualization,Relative Static Crop,QR Code Detection,Delta Filter,Path Deviation,Detections Filter,Clip Comparison,Time in Zone,CSV Formatter,Florence-2 Model,Keypoint Detection Model,Buffer,SIFT Comparison,Florence-2 Model,Multi-Label Classification Model,Rate Limiter,Keypoint Visualization,Identify Changes,Multi-Label Classification Model,Roboflow Custom Metadata,Line Counter,Stitch Images,Circle Visualization,Background Color Visualization,Twilio SMS Notification,Property Definition,LMM,Camera Focus,Image Blur,First Non Empty Or Default,Detections Merge,Google Gemini,Detection Offset,Stability AI Inpainting,Pixelate Visualization,Line Counter,OpenAI,Detections Consensus,Distance Measurement,Gaze Detection,Absolute Static Crop,Webhook Sink,Color Visualization,Image Threshold,Polygon Visualization,VLM as Classifier,Continue If,Instance Segmentation Model,Classification Label Visualization,Google Vision OCR,Roboflow Dataset Upload,Expression,Cache Get,JSON Parser,Object Detection Model,Keypoint Detection Model,Trace Visualization,Clip Comparison,Identify Outliers,YOLO-World Model,Stitch OCR Detections,Perspective Correction,OpenAI,Byte Tracker,Qwen2.5-VL,Mask Visualization,Time in Zone,Dynamic Crop,Template Matching,Byte Tracker,Barcode Detection,Instance Segmentation Model,Image Contours,SIFT,Reference Path Visualization,CLIP Embedding Model,Triangle Visualization,Model Monitoring Inference Aggregator,Bounding Rectangle,Velocity,SIFT Comparison,VLM as Detector,LMM For Classification,Grid Visualization,Segment Anything 2 Model,Detections Stabilizer,Dimension Collapse,Blur Visualization
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
LMM in version v1 has.
Bindings
-
input
images(image): The image to infer on..prompt(string): Holds unconstrained text prompt to LMM mode.lmm_type(string): Type of LMM to be used.remote_api_key(Union[string,secret]): Holds API key required to call LMM model - in current state of development, we require OpenAI key whenlmm_type=gpt_4v..
-
output
parent_id(parent_id): Identifier of parent for step output.root_parent_id(parent_id): Identifier of parent for step output.image(image_metadata): Dictionary with image metadata required by supervision.structured_output(dictionary): Dictionary.raw_output(string): String value.*(*): Equivalent of any element.
Example JSON definition of step LMM in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/lmm@v1",
"images": "$inputs.image",
"prompt": "my prompt",
"lmm_type": "gpt_4v",
"lmm_config": {
"gpt_image_detail": "low",
"gpt_model_version": "gpt-4o",
"max_tokens": 200
},
"remote_api_key": "xxx-xxx",
"json_output": {
"count": "number of cats in the picture"
}
}