LMM For Classification

Class: LMMForClassificationBlockV1

Source: inference.core.workflows.core_steps.models.foundation.lmm_classifier.v1.LMMForClassificationBlockV1

Classify an image into one or more categories using a Large Multimodal Model (LMM).

You can specify arbitrary classes to an LMMBlock.

The LLMBlock supports two LMMs:

• OpenAI's GPT-4 with Vision.

You need to provide your OpenAI API key to use the GPT-4 with Vision model.

Type identifier

Use the following identifier in step "type" field: roboflow_core/lmm_for_classification@v1to add the block as as step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
lmm_type	str	Type of LMM to be used.	✅
classes	List[str]	List of classes that LMM shall classify against.	✅
lmm_config	LMMConfig	Configuration of LMM.	❌
remote_api_key	str	Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v..	✅

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to LMM For Classification in version v1.

• inputs: Size Measurement, LMM, Roboflow Custom Metadata, Buffer, Image Convert Grayscale, Absolute Static Crop, Multi-Label Classification Model, Relative Static Crop, Line Counter Visualization, Background Color Visualization, OCR Model, Camera Focus, Image Contours, Image Slicer, Reference Path Visualization, Keypoint Detection Model, Instance Segmentation Model, SIFT Comparison, Object Detection Model, Triangle Visualization, Depth Estimation, Google Vision OCR, Roboflow Dataset Upload, Llama 3.2 Vision, Dynamic Zone, Clip Comparison, Perspective Correction, Crop Visualization, Webhook Sink, Dot Visualization, Model Comparison Visualization, Email Notification, Classification Label Visualization, Camera Calibration, Dimension Collapse, Slack Notification, Stability AI Image Generation, Trace Visualization, Corner Visualization, Image Threshold, Blur Visualization, Local File Sink, CogVLM, Stability AI Inpainting, SIFT, Circle Visualization, OpenAI, Florence-2 Model, Twilio SMS Notification, Label Visualization, Stitch Images, Image Preprocessing, Grid Visualization, Polygon Zone Visualization, Keypoint Visualization, LMM For Classification, Stitch OCR Detections, Bounding Box Visualization, Image Blur, OpenAI, Halo Visualization, Google Gemini, Ellipse Visualization, Color Visualization, Pixelate Visualization, VLM as Detector, Roboflow Dataset Upload, Polygon Visualization, Single-Label Classification Model, VLM as Classifier, CSV Formatter, Image Slicer, Model Monitoring Inference Aggregator, Clip Comparison, Mask Visualization, Anthropic Claude, Florence-2 Model, Dynamic Crop
• outputs: Size Measurement, LMM, Roboflow Custom Metadata, Distance Measurement, Line Counter Visualization, Background Color Visualization, Reference Path Visualization, Instance Segmentation Model, SIFT Comparison, Triangle Visualization, Path Deviation, Line Counter, Llama 3.2 Vision, Roboflow Dataset Upload, Google Vision OCR, Clip Comparison, Perspective Correction, Crop Visualization, Webhook Sink, Dot Visualization, Email Notification, Model Comparison Visualization, Classification Label Visualization, Instance Segmentation Model, Slack Notification, Stability AI Image Generation, Cache Set, Time in Zone, Line Counter, Time in Zone, Corner Visualization, Trace Visualization, Image Threshold, Local File Sink, CogVLM, Stability AI Inpainting, Circle Visualization, OpenAI, Path Deviation, Florence-2 Model, Twilio SMS Notification, Label Visualization, Image Preprocessing, Detections Stitch, Polygon Zone Visualization, Keypoint Visualization, LMM For Classification, Bounding Box Visualization, Image Blur, OpenAI, CLIP Embedding Model, Google Gemini, Halo Visualization, Ellipse Visualization, Color Visualization, Pixel Color Count, YOLO-World Model, Roboflow Dataset Upload, Polygon Visualization, Segment Anything 2 Model, Cache Get, Model Monitoring Inference Aggregator, Mask Visualization, Anthropic Claude, Florence-2 Model, Dynamic Crop

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds LMM For Classification in version v1 has.

Bindings

• input

  • images (image): The image to infer on..
  • lmm_type (string): Type of LMM to be used.
  • classes (list_of_values): List of classes that LMM shall classify against.
  • remote_api_key (Union[secret, string]): Holds API key required to call LMM model - in current state of development, we require OpenAI key when lmm_type=gpt_4v..

• output

  • raw_output (string): String value.
  • top (top_class): String value representing top class predicted by classification model.
  • parent_id (parent_id): Identifier of parent for step output.
  • root_parent_id (parent_id): Identifier of parent for step output.
  • image (image_metadata): Dictionary with image metadata required by supervision.
  • prediction_type (prediction_type): String value with type of prediction.

Example JSON definition of step LMM For Classification in version v1

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/lmm_for_classification@v1",
    "images": "$inputs.image",
    "lmm_type": "gpt_4v",
    "classes": [
        "a",
        "b"
    ],
    "lmm_config": {
        "gpt_image_detail": "low",
        "gpt_model_version": "gpt-4o",
        "max_tokens": 200
    },
    "remote_api_key": "xxx-xxx"
}