LMM For Classification

Class: LMMForClassificationBlockV1

Source: inference.core.workflows.core_steps.models.foundation.lmm_classifier.v1.LMMForClassificationBlockV1

Classify an image into one or more categories using a Large Multimodal Model (LMM).

You can specify arbitrary classes to an LMMBlock.

The LLMBlock supports two LMMs:

• OpenAI's GPT-4 with Vision.

You need to provide your OpenAI API key to use the GPT-4 with Vision model.

Type identifier

Use the following identifier in step "type" field: roboflow_core/lmm_for_classification@v1to add the block as as step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
lmm_type	str	Type of LMM to be used.	✅
classes	List[str]	List of classes that LMM shall classify against.	✅
lmm_config	LMMConfig	Configuration of LMM.	❌
remote_api_key	str	Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v..	✅

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to LMM For Classification in version v1.

• inputs: LMM For Classification, Single-Label Classification Model, Florence-2 Model, Background Color Visualization, Anthropic Claude, Llama 3.2 Vision, Webhook Sink, Halo Visualization, OpenAI, Reference Path Visualization, Camera Calibration, Triangle Visualization, Mask Visualization, Dynamic Crop, Stability AI Outpainting, Ellipse Visualization, Image Threshold, Relative Static Crop, Twilio SMS Notification, VLM as Classifier, Depth Estimation, Roboflow Dataset Upload, Email Notification, Bounding Box Visualization, Roboflow Custom Metadata, LMM, Color Visualization, Classification Label Visualization, Local File Sink, CogVLM, Polygon Zone Visualization, Multi-Label Classification Model, Trace Visualization, Size Measurement, Perspective Correction, Polygon Visualization, SIFT Comparison, Stability AI Inpainting, Stitch OCR Detections, Label Visualization, Model Comparison Visualization, Stability AI Image Generation, CSV Formatter, Slack Notification, Model Monitoring Inference Aggregator, Keypoint Detection Model, Pixelate Visualization, Absolute Static Crop, Object Detection Model, Dynamic Zone, Camera Focus, Keypoint Visualization, OpenAI, Image Contours, Blur Visualization, Image Blur, Clip Comparison, Grid Visualization, Image Convert Grayscale, OpenAI, Corner Visualization, SIFT, VLM as Detector, Crop Visualization, Dot Visualization, Image Slicer, Google Gemini, Image Slicer, Circle Visualization, Buffer, Florence-2 Model, OCR Model, Roboflow Dataset Upload, Dimension Collapse, Image Preprocessing, Line Counter Visualization, Google Vision OCR, Instance Segmentation Model, Stitch Images, Clip Comparison
• outputs: LMM For Classification, Florence-2 Model, Background Color Visualization, Anthropic Claude, Llama 3.2 Vision, Webhook Sink, Halo Visualization, OpenAI, Reference Path Visualization, Segment Anything 2 Model, Triangle Visualization, Mask Visualization, Dynamic Crop, Stability AI Outpainting, Image Threshold, Ellipse Visualization, Twilio SMS Notification, Path Deviation, Time in Zone, Line Counter, Email Notification, Roboflow Dataset Upload, Bounding Box Visualization, Detections Classes Replacement, Roboflow Custom Metadata, Path Deviation, LMM, Color Visualization, Instance Segmentation Model, PTZ Tracking (ONVIF).md), Classification Label Visualization, Local File Sink, Distance Measurement, Polygon Zone Visualization, CogVLM, Trace Visualization, Size Measurement, Perspective Correction, YOLO-World Model, Polygon Visualization, SIFT Comparison, Stability AI Inpainting, Label Visualization, Model Comparison Visualization, Time in Zone, Stability AI Image Generation, Detections Stitch, Slack Notification, Model Monitoring Inference Aggregator, Line Counter, Keypoint Visualization, OpenAI, Clip Comparison, OpenAI, Image Blur, Corner Visualization, Pixel Color Count, Cache Get, Crop Visualization, Dot Visualization, Google Gemini, Circle Visualization, Florence-2 Model, CLIP Embedding Model, Perception Encoder Embedding Model, Roboflow Dataset Upload, Instance Segmentation Model, Image Preprocessing, Google Vision OCR, Cache Set, Line Counter Visualization

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds LMM For Classification in version v1 has.

Bindings

• input

  • images (image): The image to infer on..
  • lmm_type (string): Type of LMM to be used.
  • classes (list_of_values): List of classes that LMM shall classify against.
  • remote_api_key (Union[string, secret]): Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v..

• output

  • raw_output (string): String value.
  • top (top_class): String value representing top class predicted by classification model.
  • parent_id (parent_id): Identifier of parent for step output.
  • root_parent_id (parent_id): Identifier of parent for step output.
  • image (image_metadata): Dictionary with image metadata required by supervision.
  • prediction_type (prediction_type): String value with type of prediction.

Example JSON definition of step LMM For Classification in version v1

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/lmm_for_classification@v1",
    "images": "$inputs.image",
    "lmm_type": "gpt_4v",
    "classes": [
        "a",
        "b"
    ],
    "lmm_config": {
        "gpt_image_detail": "low",
        "gpt_model_version": "gpt-4o",
        "max_tokens": 200
    },
    "remote_api_key": "xxx-xxx"
}