LMM For Classification

Class: LMMForClassificationBlockV1

Source: inference.core.workflows.core_steps.models.foundation.lmm_classifier.v1.LMMForClassificationBlockV1

Classify an image into one or more categories using a Large Multimodal Model (LMM).

You can specify arbitrary classes to an LMMBlock.

The LLMBlock supports two LMMs:

• OpenAI's GPT-4 with Vision.

You need to provide your OpenAI API key to use the GPT-4 with Vision model.

Type identifier

Use the following identifier in step "type" field: roboflow_core/lmm_for_classification@v1to add the block as as step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
lmm_type	str	Type of LMM to be used.	✅
classes	List[str]	List of classes that LMM shall classify against.	✅
lmm_config	LMMConfig	Configuration of LMM.	❌
remote_api_key	str	Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v..	✅

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to LMM For Classification in version v1.

• inputs: Background Color Visualization, VLM as Classifier, Trace Visualization, Google Vision OCR, OpenAI, Dot Visualization, Classification Label Visualization, Stitch Images, Model Monitoring Inference Aggregator, Line Counter Visualization, SIFT Comparison, Anthropic Claude, Email Notification, Dimension Collapse, LMM, Multi-Label Classification Model, Model Comparison Visualization, OCR Model, Absolute Static Crop, Stability AI Image Generation, Image Contours, Dynamic Crop, CSV Formatter, LMM For Classification, Image Slicer, CogVLM, Object Detection Model, Image Preprocessing, Circle Visualization, Florence-2 Model, VLM as Detector, Keypoint Detection Model, Relative Static Crop, Image Slicer, Slack Notification, Label Visualization, Roboflow Dataset Upload, Image Threshold, Reference Path Visualization, Depth Estimation, Bounding Box Visualization, Dynamic Zone, Icon Visualization, Roboflow Custom Metadata, Local File Sink, Polygon Visualization, Florence-2 Model, Ellipse Visualization, Pixelate Visualization, SIFT, Buffer, Grid Visualization, Camera Focus, Clip Comparison, Single-Label Classification Model, Image Blur, Keypoint Visualization, Polygon Zone Visualization, Size Measurement, Crop Visualization, Corner Visualization, Triangle Visualization, Stitch OCR Detections, Twilio SMS Notification, Clip Comparison, Stability AI Inpainting, Webhook Sink, OpenAI, QR Code Generator, OpenAI, Stability AI Outpainting, Camera Calibration, Image Convert Grayscale, Llama 3.2 Vision, Roboflow Dataset Upload, Blur Visualization, Color Visualization, Instance Segmentation Model, Halo Visualization, Google Gemini, Perspective Correction, Mask Visualization
• outputs: Path Deviation, Background Color Visualization, Perception Encoder Embedding Model, Time in Zone, Trace Visualization, Google Vision OCR, OpenAI, Dot Visualization, Model Monitoring Inference Aggregator, Classification Label Visualization, Line Counter, Line Counter Visualization, SIFT Comparison, Anthropic Claude, Email Notification, Pixel Color Count, LMM, Model Comparison Visualization, Line Counter, Stability AI Image Generation, Dynamic Crop, LMM For Classification, CogVLM, Image Preprocessing, Circle Visualization, CLIP Embedding Model, Distance Measurement, Florence-2 Model, PTZ Tracking (ONVIF).md), YOLO-World Model, Moondream2, Slack Notification, Label Visualization, Roboflow Dataset Upload, Image Threshold, Reference Path Visualization, Cache Get, Bounding Box Visualization, Icon Visualization, Roboflow Custom Metadata, Local File Sink, Cache Set, Florence-2 Model, Polygon Visualization, Path Deviation, Ellipse Visualization, Instance Segmentation Model, Image Blur, Detections Classes Replacement, Keypoint Visualization, Polygon Zone Visualization, Size Measurement, Crop Visualization, Corner Visualization, Triangle Visualization, Twilio SMS Notification, Clip Comparison, Stability AI Inpainting, Webhook Sink, OpenAI, QR Code Generator, OpenAI, Stability AI Outpainting, Detections Stitch, Time in Zone, Llama 3.2 Vision, Roboflow Dataset Upload, Segment Anything 2 Model, Color Visualization, Instance Segmentation Model, Halo Visualization, Google Gemini, Perspective Correction, Mask Visualization, Time in Zone

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds LMM For Classification in version v1 has.

Bindings

• input

  • images (image): The image to infer on..
  • lmm_type (string): Type of LMM to be used.
  • classes (list_of_values): List of classes that LMM shall classify against.
  • remote_api_key (Union[secret, string]): Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v..

• output

  • raw_output (string): String value.
  • top (top_class): String value representing top class predicted by classification model.
  • parent_id (parent_id): Identifier of parent for step output.
  • root_parent_id (parent_id): Identifier of parent for step output.
  • image (image_metadata): Dictionary with image metadata required by supervision.
  • prediction_type (prediction_type): String value with type of prediction.

Example JSON definition of step LMM For Classification in version v1

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/lmm_for_classification@v1",
    "images": "$inputs.image",
    "lmm_type": "gpt_4v",
    "classes": [
        "a",
        "b"
    ],
    "lmm_config": {
        "gpt_image_detail": "low",
        "gpt_model_version": "gpt-4o",
        "max_tokens": 200
    },
    "remote_api_key": "xxx-xxx"
}