LMM For Classification

Class: LMMForClassificationBlockV1

Source: inference.core.workflows.core_steps.models.foundation.lmm_classifier.v1.LMMForClassificationBlockV1

Classify an image into one or more categories using a Large Multimodal Model (LMM).

You can specify arbitrary classes to an LMMBlock.

The LLMBlock supports two LMMs:

• OpenAI's GPT-4 with Vision.

You need to provide your OpenAI API key to use the GPT-4 with Vision model.

Type identifier

Use the following identifier in step "type" field: roboflow_core/lmm_for_classification@v1to add the block as as step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
lmm_type	str	Type of LMM to be used.	✅
classes	List[str]	List of classes that LMM shall classify against.	✅
lmm_config	LMMConfig	Configuration of LMM.	❌
remote_api_key	str	Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v..	✅

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to LMM For Classification in version v1.

• inputs: Multi-Label Classification Model, Single-Label Classification Model, Classification Label Visualization, Dimension Collapse, Background Color Visualization, Webhook Sink, Dynamic Crop, Mask Visualization, Clip Comparison, Google Vision OCR, Twilio SMS Notification, Buffer, Absolute Static Crop, Model Monitoring Inference Aggregator, Stability AI Image Generation, Florence-2 Model, Image Blur, LMM For Classification, Roboflow Dataset Upload, CogVLM, Circle Visualization, OCR Model, Clip Comparison, Crop Visualization, OpenAI, Stitch OCR Detections, OpenAI, Image Preprocessing, Model Comparison Visualization, Stitch Images, Bounding Box Visualization, Keypoint Detection Model, Perspective Correction, SIFT Comparison, Relative Static Crop, Slack Notification, Color Visualization, Ellipse Visualization, Reference Path Visualization, Blur Visualization, Pixelate Visualization, Anthropic Claude, Email Notification, LMM, Llama 3.2 Vision, CSV Formatter, VLM as Detector, Keypoint Visualization, Camera Focus, Florence-2 Model, Grid Visualization, Image Convert Grayscale, Image Threshold, Trace Visualization, Polygon Visualization, Triangle Visualization, Stability AI Inpainting, Halo Visualization, Dot Visualization, Polygon Zone Visualization, Google Gemini, Local File Sink, Dynamic Zone, Size Measurement, Instance Segmentation Model, Roboflow Custom Metadata, VLM as Classifier, Camera Calibration, Object Detection Model, SIFT, Corner Visualization, Image Contours, Line Counter Visualization, Roboflow Dataset Upload, Image Slicer, Image Slicer, Label Visualization
• outputs: Classification Label Visualization, Webhook Sink, Background Color Visualization, Dynamic Crop, Cache Get, Mask Visualization, Clip Comparison, Twilio SMS Notification, Google Vision OCR, Segment Anything 2 Model, Model Monitoring Inference Aggregator, Stability AI Image Generation, LMM For Classification, Florence-2 Model, Image Blur, Cache Set, Roboflow Dataset Upload, CogVLM, Circle Visualization, Crop Visualization, Path Deviation, OpenAI, Detections Stitch, OpenAI, Pixel Color Count, Label Visualization, Path Deviation, Line Counter, Time in Zone, Model Comparison Visualization, Bounding Box Visualization, Perspective Correction, SIFT Comparison, Slack Notification, Color Visualization, Ellipse Visualization, Reference Path Visualization, Anthropic Claude, Email Notification, LMM, Llama 3.2 Vision, Instance Segmentation Model, Keypoint Visualization, Time in Zone, Florence-2 Model, YOLO-World Model, Trace Visualization, Image Threshold, Triangle Visualization, Polygon Visualization, Stability AI Inpainting, Halo Visualization, Dot Visualization, Google Gemini, Polygon Zone Visualization, CLIP Embedding Model, Local File Sink, Size Measurement, Instance Segmentation Model, Roboflow Custom Metadata, Corner Visualization, Roboflow Dataset Upload, Line Counter Visualization, Image Preprocessing, Line Counter, Distance Measurement

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds LMM For Classification in version v1 has.

Bindings

• input

  • images (image): The image to infer on..
  • lmm_type (string): Type of LMM to be used.
  • classes (list_of_values): List of classes that LMM shall classify against.
  • remote_api_key (Union[string, secret]): Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v..

• output

  • raw_output (string): String value.
  • top (top_class): String value representing top class predicted by classification model.
  • parent_id (parent_id): Identifier of parent for step output.
  • root_parent_id (parent_id): Identifier of parent for step output.
  • image (image_metadata): Dictionary with image metadata required by supervision.
  • prediction_type (prediction_type): String value with type of prediction.

Example JSON definition of step LMM For Classification in version v1

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/lmm_for_classification@v1",
    "images": "$inputs.image",
    "lmm_type": "gpt_4v",
    "classes": [
        "a",
        "b"
    ],
    "lmm_config": {
        "gpt_image_detail": "low",
        "gpt_model_version": "gpt-4o",
        "max_tokens": 200
    },
    "remote_api_key": "xxx-xxx"
}