LMM For Classification¶
Class: LMMForClassificationBlockV1
Source: inference.core.workflows.core_steps.models.foundation.lmm_classifier.v1.LMMForClassificationBlockV1
Classify an image into one or more categories using a Large Multimodal Model (LMM).
You can specify arbitrary classes to an LMMBlock.
The LLMBlock supports two LMMs:
- OpenAI's GPT-4 with Vision.
You need to provide your OpenAI API key to use the GPT-4 with Vision model.
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/lmm_for_classification@v1to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
lmm_type |
str |
Type of LMM to be used. | ✅ |
classes |
List[str] |
List of classes that LMM shall classify against. | ✅ |
lmm_config |
LMMConfig |
Configuration of LMM. | ❌ |
remote_api_key |
str |
Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v.. |
✅ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to LMM For Classification in version v1.
- inputs:
Circle Visualization,Background Color Visualization,Corner Visualization,Twilio SMS Notification,Slack Notification,VLM as Detector,LMM,Polygon Zone Visualization,Camera Focus,Image Slicer,Image Blur,Dot Visualization,Google Gemini,Roboflow Dataset Upload,Stability AI Inpainting,Pixelate Visualization,OpenAI,Image Convert Grayscale,Absolute Static Crop,Stability AI Image Generation,Webhook Sink,Color Visualization,Image Threshold,Halo Visualization,Polygon Visualization,VLM as Classifier,Dynamic Zone,CogVLM,Camera Calibration,Email Notification,Classification Label Visualization,Single-Label Classification Model,Llama 3.2 Vision,Google Vision OCR,Roboflow Dataset Upload,Ellipse Visualization,Size Measurement,Bounding Box Visualization,Object Detection Model,Line Counter Visualization,Image Preprocessing,Trace Visualization,Label Visualization,Clip Comparison,Local File Sink,Image Slicer,Anthropic Claude,Crop Visualization,OCR Model,Relative Static Crop,Model Comparison Visualization,Stitch OCR Detections,Perspective Correction,OpenAI,Mask Visualization,Clip Comparison,Dynamic Crop,CSV Formatter,Florence-2 Model,Instance Segmentation Model,Keypoint Detection Model,Image Contours,Buffer,SIFT,Reference Path Visualization,Florence-2 Model,Triangle Visualization,Model Monitoring Inference Aggregator,SIFT Comparison,Keypoint Visualization,Multi-Label Classification Model,LMM For Classification,Roboflow Custom Metadata,Grid Visualization,Stitch Images,Dimension Collapse,Blur Visualization - outputs:
Circle Visualization,Background Color Visualization,Corner Visualization,Twilio SMS Notification,Slack Notification,LMM,Polygon Zone Visualization,Image Blur,Cache Set,Dot Visualization,Path Deviation,Google Gemini,Roboflow Dataset Upload,Stability AI Inpainting,Line Counter,OpenAI,Distance Measurement,Stability AI Image Generation,Webhook Sink,Color Visualization,Image Threshold,Halo Visualization,Polygon Visualization,CogVLM,Instance Segmentation Model,Email Notification,Classification Label Visualization,Llama 3.2 Vision,Google Vision OCR,Roboflow Dataset Upload,Ellipse Visualization,Size Measurement,Pixel Color Count,Cache Get,Bounding Box Visualization,Line Counter Visualization,Image Preprocessing,Trace Visualization,Label Visualization,Local File Sink,Anthropic Claude,Crop Visualization,Detections Stitch,YOLO-World Model,Model Comparison Visualization,Perspective Correction,OpenAI,Path Deviation,Mask Visualization,Time in Zone,Clip Comparison,Time in Zone,Dynamic Crop,Florence-2 Model,Instance Segmentation Model,Florence-2 Model,Reference Path Visualization,CLIP Embedding Model,Triangle Visualization,Model Monitoring Inference Aggregator,SIFT Comparison,Keypoint Visualization,LMM For Classification,Roboflow Custom Metadata,Line Counter,Segment Anything 2 Model
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
LMM For Classification in version v1 has.
Bindings
-
input
images(image): The image to infer on..lmm_type(string): Type of LMM to be used.classes(list_of_values): List of classes that LMM shall classify against.remote_api_key(Union[string,secret]): Holds API key required to call LMM model - in current state of development, we require OpenAI key whenlmm_type=gpt_4v..
-
output
raw_output(string): String value.top(top_class): String value representing top class predicted by classification model.parent_id(parent_id): Identifier of parent for step output.root_parent_id(parent_id): Identifier of parent for step output.image(image_metadata): Dictionary with image metadata required by supervision.prediction_type(prediction_type): String value with type of prediction.
Example JSON definition of step LMM For Classification in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/lmm_for_classification@v1",
"images": "$inputs.image",
"lmm_type": "gpt_4v",
"classes": [
"a",
"b"
],
"lmm_config": {
"gpt_image_detail": "low",
"gpt_model_version": "gpt-4o",
"max_tokens": 200
},
"remote_api_key": "xxx-xxx"
}