OpenAI

v3

Class: OpenAIBlockV3 (there are multiple versions of this block)

Source: inference.core.workflows.core_steps.models.foundation.openai.v3.OpenAIBlockV3

Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning

Ask a question to OpenAI's GPT models with vision capabilities (including GPT-5 and GPT-4o).

You can specify arbitrary text prompts or predefined ones, the block supports the following types of prompt:

• Open Prompt (unconstrained) - Use any prompt to generate a raw response

• Text Recognition (OCR) (ocr) - Model recognizes text in the image

• Visual Question Answering (visual-question-answering) - Model answers the question you submit in the prompt

• Captioning (short) (caption) - Model provides a short description of the image

• Captioning (detailed-caption) - Model provides a long description of the image

• Single-Label Classification (classification) - Model classifies the image content as one of the provided classes

• Multi-Label Classification (multi-label-classification) - Model classifies the image content as one or more of the provided classes

• Structured Output Generation (structured-answering) - Model returns a JSON response with the specified fields

Provide your OpenAI API key or set the value to rf_key:account (or rf_key:user:<id>) to proxy requests through Roboflow's API.

Type identifier

Use the following identifier in step "type" field: roboflow_core/open_ai@v3to add the block as as step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
task_type	str	Task type to be performed by model. Value determines required parameters and output response..	❌
prompt	str	Text prompt to the OpenAI model.	✅
output_structure	Dict[str, str]	Dictionary with structure of expected JSON response.	❌
classes	List[str]	List of classes to be used.	✅
api_key	str	Your OpenAI API key.	✅
model_version	str	Model to be used.	✅
image_detail	str	Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..	✅
max_tokens	int	Maximum number of tokens the model can generate in it's response..	❌
temperature	float	Temperature to sample from the model - value in range 0.0-2.0, the higher - the more random / "creative" the generations are..	✅
max_concurrent_requests	int	Number of concurrent requests that can be executed by block when batch of input images provided. If not given - block defaults to value configured globally in Workflows Execution Engine. Please restrict if you hit OpenAI limits..	❌

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to OpenAI in version v3.

• inputs: Local File Sink, Dot Visualization, Stability AI Inpainting, Reference Path Visualization, VLM as Classifier, Stability AI Outpainting, Multi-Label Classification Model, QR Code Generator, Line Counter Visualization, Size Measurement, Ellipse Visualization, Roboflow Custom Metadata, Dimension Collapse, Background Color Visualization, Polygon Zone Visualization, CSV Formatter, Roboflow Dataset Upload, Contrast Equalization, EasyOCR, Object Detection Model, Image Slicer, Dynamic Zone, Google Gemini, Florence-2 Model, Gaze Detection, Google Vision OCR, Image Threshold, SIFT Comparison, Identify Changes, Image Preprocessing, Icon Visualization, OCR Model, Roboflow Dataset Upload, Clip Comparison, Absolute Static Crop, Pixelate Visualization, Image Blur, Buffer, Perspective Correction, Relative Static Crop, Florence-2 Model, VLM as Detector, LMM For Classification, Llama 3.2 Vision, Clip Comparison, LMM, SIFT, Halo Visualization, Model Monitoring Inference Aggregator, Image Convert Grayscale, Anthropic Claude, Triangle Visualization, Cosine Similarity, Depth Estimation, Image Contours, Mask Visualization, Keypoint Detection Model, Image Slicer, CogVLM, Model Comparison Visualization, Stitch OCR Detections, Twilio SMS Notification, Single-Label Classification Model, Polygon Visualization, Corner Visualization, Crop Visualization, Stitch Images, Blur Visualization, Dynamic Crop, Camera Focus, OpenAI, Email Notification, Color Visualization, Classification Label Visualization, Label Visualization, OpenAI, Circle Visualization, Keypoint Visualization, Trace Visualization, Camera Calibration, Instance Segmentation Model, Morphological Transformation, OpenAI, Bounding Box Visualization, Grid Visualization, Slack Notification, Webhook Sink, Stability AI Image Generation
• outputs: PTZ Tracking (ONVIF).md), Local File Sink, Dot Visualization, Stability AI Inpainting, Reference Path Visualization, CLIP Embedding Model, VLM as Classifier, Object Detection Model, VLM as Classifier, Stability AI Outpainting, Distance Measurement, Perception Encoder Embedding Model, QR Code Generator, Path Deviation, Time in Zone, Cache Set, Line Counter Visualization, Detections Classes Replacement, Size Measurement, Ellipse Visualization, Roboflow Custom Metadata, Polygon Zone Visualization, Background Color Visualization, Roboflow Dataset Upload, Contrast Equalization, Object Detection Model, Google Gemini, Florence-2 Model, Google Vision OCR, Image Threshold, SIFT Comparison, Detections Consensus, Image Preprocessing, Icon Visualization, YOLO-World Model, Roboflow Dataset Upload, Clip Comparison, Seg Preview, Image Blur, Buffer, Perspective Correction, Line Counter, Florence-2 Model, Pixel Color Count, VLM as Detector, LMM For Classification, Llama 3.2 Vision, Detections Stitch, Line Counter, Time in Zone, LMM, Clip Comparison, Halo Visualization, Model Monitoring Inference Aggregator, Cache Get, Anthropic Claude, Triangle Visualization, Mask Visualization, Keypoint Detection Model, CogVLM, Model Comparison Visualization, Stitch OCR Detections, Twilio SMS Notification, Time in Zone, Moondream2, Polygon Visualization, Corner Visualization, Crop Visualization, Dynamic Crop, Keypoint Detection Model, Instance Segmentation Model, OpenAI, Segment Anything 2 Model, Email Notification, VLM as Detector, Color Visualization, Classification Label Visualization, Label Visualization, OpenAI, Keypoint Visualization, Circle Visualization, Trace Visualization, Instance Segmentation Model, Morphological Transformation, OpenAI, Bounding Box Visualization, JSON Parser, Path Deviation, Grid Visualization, Slack Notification, Webhook Sink, Stability AI Image Generation

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds OpenAI in version v3 has.

Bindings

• input

  • images (image): The image to infer on..
  • prompt (string): Text prompt to the OpenAI model.
  • classes (list_of_values): List of classes to be used.
  • api_key (Union[string, ROBOFLOW_MANAGED_KEY, secret]): Your OpenAI API key.
  • model_version (string): Model to be used.
  • image_detail (string): Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..
  • temperature (float): Temperature to sample from the model - value in range 0.0-2.0, the higher - the more random / "creative" the generations are..

• output

  • output (Union[string, language_model_output]): String value if string or LLM / VLM output if language_model_output.
  • classes (list_of_values): List of values of any type.

Example JSON definition of step OpenAI in version v3

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/open_ai@v3",
    "images": "$inputs.image",
    "task_type": "<block_does_not_provide_example>",
    "prompt": "my prompt",
    "output_structure": {
        "my_key": "description"
    },
    "classes": [
        "class-a",
        "class-b"
    ],
    "api_key": "xxx-xxx",
    "model_version": "gpt-5",
    "image_detail": "auto",
    "max_tokens": "<block_does_not_provide_example>",
    "temperature": "<block_does_not_provide_example>",
    "max_concurrent_requests": "<block_does_not_provide_example>"
}

v2

Class: OpenAIBlockV2 (there are multiple versions of this block)

Source: inference.core.workflows.core_steps.models.foundation.openai.v2.OpenAIBlockV2

Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning

Ask a question to OpenAI's GPT models with vision capabilities (including GPT-4o and GPT-5).

You can specify arbitrary text prompts or predefined ones, the block supports the following types of prompt:

• Open Prompt (unconstrained) - Use any prompt to generate a raw response

• Text Recognition (OCR) (ocr) - Model recognizes text in the image

• Visual Question Answering (visual-question-answering) - Model answers the question you submit in the prompt

• Captioning (short) (caption) - Model provides a short description of the image

• Captioning (detailed-caption) - Model provides a long description of the image

• Single-Label Classification (classification) - Model classifies the image content as one of the provided classes

• Multi-Label Classification (multi-label-classification) - Model classifies the image content as one or more of the provided classes

• Structured Output Generation (structured-answering) - Model returns a JSON response with the specified fields

You need to provide your OpenAI API key to use the GPT-4 with Vision model.

Type identifier

Use the following identifier in step "type" field: roboflow_core/open_ai@v2to add the block as as step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
task_type	str	Task type to be performed by model. Value determines required parameters and output response..	❌
prompt	str	Text prompt to the OpenAI model.	✅
output_structure	Dict[str, str]	Dictionary with structure of expected JSON response.	❌
classes	List[str]	List of classes to be used.	✅
api_key	str	Your OpenAI API key.	✅
model_version	str	Model to be used.	✅
image_detail	str	Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..	✅
max_tokens	int	Maximum number of tokens the model can generate in it's response..	❌
temperature	float	Temperature to sample from the model - value in range 0.0-2.0, the higher - the more random / "creative" the generations are..	✅
max_concurrent_requests	int	Number of concurrent requests that can be executed by block when batch of input images provided. If not given - block defaults to value configured globally in Workflows Execution Engine. Please restrict if you hit OpenAI limits..	❌

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to OpenAI in version v2.

• inputs: Local File Sink, Dot Visualization, Stability AI Inpainting, Reference Path Visualization, VLM as Classifier, Stability AI Outpainting, Multi-Label Classification Model, QR Code Generator, Line Counter Visualization, Size Measurement, Ellipse Visualization, Roboflow Custom Metadata, Dimension Collapse, Background Color Visualization, Polygon Zone Visualization, CSV Formatter, Roboflow Dataset Upload, Contrast Equalization, EasyOCR, Object Detection Model, Image Slicer, Dynamic Zone, Google Gemini, Florence-2 Model, Gaze Detection, Google Vision OCR, Image Threshold, SIFT Comparison, Identify Changes, Image Preprocessing, Icon Visualization, OCR Model, Roboflow Dataset Upload, Clip Comparison, Absolute Static Crop, Pixelate Visualization, Image Blur, Buffer, Perspective Correction, Relative Static Crop, Florence-2 Model, VLM as Detector, LMM For Classification, Llama 3.2 Vision, Clip Comparison, LMM, SIFT, Halo Visualization, Model Monitoring Inference Aggregator, Image Convert Grayscale, Anthropic Claude, Triangle Visualization, Cosine Similarity, Depth Estimation, Image Contours, Mask Visualization, Keypoint Detection Model, Image Slicer, CogVLM, Model Comparison Visualization, Stitch OCR Detections, Twilio SMS Notification, Single-Label Classification Model, Polygon Visualization, Corner Visualization, Crop Visualization, Stitch Images, Blur Visualization, Dynamic Crop, Camera Focus, OpenAI, Email Notification, Color Visualization, Classification Label Visualization, Label Visualization, OpenAI, Circle Visualization, Keypoint Visualization, Trace Visualization, Camera Calibration, Instance Segmentation Model, Morphological Transformation, OpenAI, Bounding Box Visualization, Grid Visualization, Slack Notification, Webhook Sink, Stability AI Image Generation
• outputs: PTZ Tracking (ONVIF).md), Local File Sink, Dot Visualization, Stability AI Inpainting, Reference Path Visualization, CLIP Embedding Model, VLM as Classifier, Object Detection Model, VLM as Classifier, Stability AI Outpainting, Distance Measurement, Perception Encoder Embedding Model, QR Code Generator, Path Deviation, Time in Zone, Cache Set, Line Counter Visualization, Detections Classes Replacement, Size Measurement, Ellipse Visualization, Roboflow Custom Metadata, Polygon Zone Visualization, Background Color Visualization, Roboflow Dataset Upload, Contrast Equalization, Object Detection Model, Google Gemini, Florence-2 Model, Google Vision OCR, Image Threshold, SIFT Comparison, Detections Consensus, Image Preprocessing, Icon Visualization, YOLO-World Model, Roboflow Dataset Upload, Clip Comparison, Seg Preview, Image Blur, Buffer, Perspective Correction, Line Counter, Florence-2 Model, Pixel Color Count, VLM as Detector, LMM For Classification, Llama 3.2 Vision, Detections Stitch, Line Counter, Time in Zone, LMM, Clip Comparison, Halo Visualization, Model Monitoring Inference Aggregator, Cache Get, Anthropic Claude, Triangle Visualization, Mask Visualization, Keypoint Detection Model, CogVLM, Model Comparison Visualization, Stitch OCR Detections, Twilio SMS Notification, Time in Zone, Moondream2, Polygon Visualization, Corner Visualization, Crop Visualization, Dynamic Crop, Keypoint Detection Model, Instance Segmentation Model, OpenAI, Segment Anything 2 Model, Email Notification, VLM as Detector, Color Visualization, Classification Label Visualization, Label Visualization, OpenAI, Keypoint Visualization, Circle Visualization, Trace Visualization, Instance Segmentation Model, Morphological Transformation, OpenAI, Bounding Box Visualization, JSON Parser, Path Deviation, Grid Visualization, Slack Notification, Webhook Sink, Stability AI Image Generation

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds OpenAI in version v2 has.

Bindings

• input

  • images (image): The image to infer on..
  • prompt (string): Text prompt to the OpenAI model.
  • classes (list_of_values): List of classes to be used.
  • api_key (Union[string, secret]): Your OpenAI API key.
  • model_version (string): Model to be used.
  • image_detail (string): Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..
  • temperature (float): Temperature to sample from the model - value in range 0.0-2.0, the higher - the more random / "creative" the generations are..

• output

  • output (Union[string, language_model_output]): String value if string or LLM / VLM output if language_model_output.
  • classes (list_of_values): List of values of any type.

Example JSON definition of step OpenAI in version v2

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/open_ai@v2",
    "images": "$inputs.image",
    "task_type": "<block_does_not_provide_example>",
    "prompt": "my prompt",
    "output_structure": {
        "my_key": "description"
    },
    "classes": [
        "class-a",
        "class-b"
    ],
    "api_key": "xxx-xxx",
    "model_version": "gpt-4o",
    "image_detail": "auto",
    "max_tokens": "<block_does_not_provide_example>",
    "temperature": "<block_does_not_provide_example>",
    "max_concurrent_requests": "<block_does_not_provide_example>"
}

v1

Class: OpenAIBlockV1 (there are multiple versions of this block)

Source: inference.core.workflows.core_steps.models.foundation.openai.v1.OpenAIBlockV1

Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning

Ask a question to OpenAI's GPT-4 with Vision model.

You can specify arbitrary text prompts to the OpenAIBlock.

You need to provide your OpenAI API key to use the GPT-4 with Vision model.

This model was previously part of the LMM block.

Type identifier

Use the following identifier in step "type" field: roboflow_core/open_ai@v1to add the block as as step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
prompt	str	Text prompt to the OpenAI model.	✅
openai_api_key	str	Your OpenAI API key.	✅
openai_model	str	Model to be used.	✅
json_output_format	Dict[str, str]	Holds dictionary that maps name of requested output field into its description.	❌
image_detail	str	Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..	✅
max_tokens	int	Maximum number of tokens the model can generate in it's response..	❌

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Available Connections

Compatible Blocks

Check what blocks you can connect to OpenAI in version v1.

• inputs: Local File Sink, Dot Visualization, Stability AI Inpainting, Reference Path Visualization, VLM as Classifier, Stability AI Outpainting, Multi-Label Classification Model, QR Code Generator, Line Counter Visualization, Ellipse Visualization, Roboflow Custom Metadata, Background Color Visualization, Polygon Zone Visualization, CSV Formatter, Roboflow Dataset Upload, Contrast Equalization, EasyOCR, Object Detection Model, Image Slicer, Google Gemini, Florence-2 Model, Google Vision OCR, Image Threshold, SIFT Comparison, Image Preprocessing, Icon Visualization, OCR Model, Roboflow Dataset Upload, Absolute Static Crop, Pixelate Visualization, Image Blur, Perspective Correction, Relative Static Crop, Florence-2 Model, VLM as Detector, LMM For Classification, Llama 3.2 Vision, Clip Comparison, LMM, SIFT, Halo Visualization, Model Monitoring Inference Aggregator, Image Convert Grayscale, Anthropic Claude, Triangle Visualization, Depth Estimation, Image Contours, Mask Visualization, Keypoint Detection Model, Image Slicer, CogVLM, Model Comparison Visualization, Stitch OCR Detections, Twilio SMS Notification, Single-Label Classification Model, Polygon Visualization, Corner Visualization, Crop Visualization, Stitch Images, Blur Visualization, Dynamic Crop, Camera Focus, OpenAI, Email Notification, Color Visualization, Classification Label Visualization, Label Visualization, OpenAI, Circle Visualization, Keypoint Visualization, Trace Visualization, Camera Calibration, Instance Segmentation Model, Morphological Transformation, OpenAI, Bounding Box Visualization, Grid Visualization, Slack Notification, Webhook Sink, Stability AI Image Generation
• outputs: PTZ Tracking (ONVIF).md), Local File Sink, Barcode Detection, Stability AI Inpainting, Delta Filter, VLM as Classifier, Distance Measurement, Perception Encoder Embedding Model, QR Code Generator, Path Deviation, Time in Zone, Size Measurement, Continue If, Identify Outliers, Detections Combine, Polygon Zone Visualization, Contrast Equalization, EasyOCR, Object Detection Model, Florence-2 Model, Gaze Detection, Detections Consensus, Roboflow Dataset Upload, Clip Comparison, Detections Filter, Detection Offset, Absolute Static Crop, Pixelate Visualization, Perspective Correction, Relative Static Crop, Line Counter, VLM as Detector, Single-Label Classification Model, LMM For Classification, Detections Stitch, LMM, Multi-Label Classification Model, SmolVLM2, Model Monitoring Inference Aggregator, Image Convert Grayscale, CogVLM, Model Comparison Visualization, Template Matching, Twilio SMS Notification, First Non Empty Or Default, Byte Tracker, Single-Label Classification Model, Polygon Visualization, Keypoint Detection Model, Byte Tracker, Instance Segmentation Model, VLM as Detector, Label Visualization, OpenAI, Keypoint Visualization, Circle Visualization, Trace Visualization, Camera Calibration, Instance Segmentation Model, Expression, Stability AI Image Generation, Dot Visualization, Reference Path Visualization, Bounding Rectangle, CLIP Embedding Model, VLM as Classifier, Object Detection Model, Stability AI Outpainting, Detections Merge, Multi-Label Classification Model, Cache Set, Line Counter Visualization, Detections Classes Replacement, Ellipse Visualization, Roboflow Custom Metadata, Dimension Collapse, Background Color Visualization, CSV Formatter, Roboflow Dataset Upload, Image Slicer, Dynamic Zone, Qwen2.5-VL, Google Gemini, Byte Tracker, Google Vision OCR, Image Threshold, SIFT Comparison, Identify Changes, Image Preprocessing, Icon Visualization, OCR Model, YOLO-World Model, Seg Preview, Image Blur, Buffer, Florence-2 Model, Pixel Color Count, Llama 3.2 Vision, Line Counter, Time in Zone, Clip Comparison, SIFT, Halo Visualization, Cache Get, Anthropic Claude, Triangle Visualization, Cosine Similarity, Mask Visualization, Keypoint Detection Model, Image Contours, Depth Estimation, Image Slicer, Stitch OCR Detections, Time in Zone, QR Code Detection, Moondream2, Corner Visualization, Crop Visualization, Stitch Images, Blur Visualization, Dynamic Crop, Detections Stabilizer, Overlap Filter, Detections Transformation, OpenAI, Segment Anything 2 Model, Email Notification, Camera Focus, Color Visualization, Velocity, Data Aggregator, Classification Label Visualization, Property Definition, SIFT Comparison, Morphological Transformation, OpenAI, Bounding Box Visualization, JSON Parser, Path Deviation, Dominant Color, Grid Visualization, Slack Notification, Webhook Sink, Rate Limiter

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds OpenAI in version v1 has.

Bindings

• input

  • images (image): The image to infer on..
  • prompt (string): Text prompt to the OpenAI model.
  • openai_api_key (Union[string, secret]): Your OpenAI API key.
  • openai_model (string): Model to be used.
  • image_detail (string): Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..

• output

  • parent_id (parent_id): Identifier of parent for step output.
  • root_parent_id (parent_id): Identifier of parent for step output.
  • image (image_metadata): Dictionary with image metadata required by supervision.
  • structured_output (dictionary): Dictionary.
  • raw_output (string): String value.
  • * (*): Equivalent of any element.

Example JSON definition of step OpenAI in version v1

{
    "name": "<your_step_name_here>",
    "type": "roboflow_core/open_ai@v1",
    "images": "$inputs.image",
    "prompt": "my prompt",
    "openai_api_key": "xxx-xxx",
    "openai_model": "gpt-4o",
    "json_output_format": {
        "count": "number of cats in the picture"
    },
    "image_detail": "auto",
    "max_tokens": 450
}