LMM For Classification¶
Class: LMMForClassificationBlockV1
Source: inference.core.workflows.core_steps.models.foundation.lmm_classifier.v1.LMMForClassificationBlockV1
Classify an image into one or more categories using a Large Multimodal Model (LMM).
You can specify arbitrary classes to an LMMBlock.
The LLMBlock supports two LMMs:
- OpenAI's GPT-4 with Vision, and;
- CogVLM.
You need to provide your OpenAI API key to use the GPT-4 with Vision model. You do not need to provide an API key to use CogVLM.
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/lmm_for_classification@v1to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
lmm_type |
str |
Type of LMM to be used. | ✅ |
classes |
List[str] |
List of classes that LMM shall classify against. | ✅ |
lmm_config |
LMMConfig |
Configuration of LMM. | ❌ |
remote_api_key |
str |
Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v and do not require additional API key for CogVLM calls.. |
✅ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to LMM For Classification in version v1.
- inputs:
Stitch Images,Pixelate Visualization,Multi-Label Classification Model,LMM For Classification,Blur Visualization,Single-Label Classification Model,Mask Visualization,OCR Model,Object Detection Model,SIFT,Model Monitoring Inference Aggregator,Polygon Visualization,Halo Visualization,Grid Visualization,Google Vision OCR,Model Comparison Visualization,Email Notification,Camera Focus,CogVLM,Image Threshold,Keypoint Visualization,Image Preprocessing,Roboflow Dataset Upload,Slack Notification,Stitch OCR Detections,Relative Static Crop,Background Color Visualization,Clip Comparison,Bounding Box Visualization,Ellipse Visualization,Image Contours,Label Visualization,Classification Label Visualization,Line Counter Visualization,LMM,Stability AI Inpainting,Reference Path Visualization,VLM as Detector,Dynamic Crop,Triangle Visualization,Absolute Static Crop,Florence-2 Model,SIFT Comparison,Keypoint Detection Model,Corner Visualization,Perspective Correction,Local File Sink,Polygon Zone Visualization,VLM as Classifier,Dimension Collapse,Image Slicer,Trace Visualization,Webhook Sink,OpenAI,Twilio SMS Notification,Roboflow Custom Metadata,Size Measurement,Crop Visualization,Instance Segmentation Model,Buffer,Roboflow Dataset Upload,Clip Comparison,Anthropic Claude,Dynamic Zone,Image Blur,Circle Visualization,Image Convert Grayscale,Dot Visualization,Google Gemini,Florence-2 Model,OpenAI,Color Visualization,CSV Formatter,Llama 3.2 Vision - outputs:
Path Deviation,LMM For Classification,Line Counter,Instance Segmentation Model,CLIP Embedding Model,Mask Visualization,Line Counter,YOLO-World Model,Model Monitoring Inference Aggregator,Cache Get,Polygon Visualization,Halo Visualization,Google Vision OCR,Model Comparison Visualization,Email Notification,CogVLM,Image Threshold,Keypoint Visualization,Image Preprocessing,Roboflow Dataset Upload,Slack Notification,Background Color Visualization,Bounding Box Visualization,Ellipse Visualization,Classification Label Visualization,Label Visualization,Line Counter Visualization,LMM,Stability AI Inpainting,Reference Path Visualization,Dynamic Crop,Triangle Visualization,Distance Measurement,Time in Zone,Detections Stitch,Florence-2 Model,SIFT Comparison,Corner Visualization,Perspective Correction,Local File Sink,Polygon Zone Visualization,Twilio SMS Notification,Trace Visualization,Webhook Sink,Size Measurement,OpenAI,Roboflow Custom Metadata,Cache Set,Crop Visualization,Instance Segmentation Model,Roboflow Dataset Upload,Clip Comparison,Anthropic Claude,Image Blur,Circle Visualization,Dot Visualization,Google Gemini,Segment Anything 2 Model,Florence-2 Model,Time in Zone,Path Deviation,OpenAI,Color Visualization,Pixel Color Count,Llama 3.2 Vision
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
LMM For Classification in version v1 has.
Bindings
-
input
images(image): The image to infer on.lmm_type(string): Type of LMM to be used.classes(list_of_values): List of classes that LMM shall classify against.remote_api_key(Union[secret,string]): Holds API key required to call LMM model - in current state of development, we require OpenAI key whenlmm_type=gpt_4vand do not require additional API key for CogVLM calls..
-
output
raw_output(string): String value.top(top_class): String value representing top class predicted by classification model.parent_id(parent_id): Identifier of parent for step output.root_parent_id(parent_id): Identifier of parent for step output.image(image_metadata): Dictionary with image metadata required by supervision.prediction_type(prediction_type): String value with type of prediction.
Example JSON definition of step LMM For Classification in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/lmm_for_classification@v1",
"images": "$inputs.image",
"lmm_type": "gpt_4v",
"classes": [
"a",
"b"
],
"lmm_config": {
"gpt_image_detail": "low",
"gpt_model_version": "gpt-4o",
"max_tokens": 200
},
"remote_api_key": "xxx-xxx"
}