OpenAI¶
v2¶
Class: OpenAIBlockV2 (there are multiple versions of this block)
Source: inference.core.workflows.core_steps.models.foundation.openai.v2.OpenAIBlockV2
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
Ask a question to OpenAI's GPT-4 with Vision model.
You can specify arbitrary text prompts or predefined ones, the block supports the following types of prompt:
-
Open Prompt (
unconstrained) - Use any prompt to generate a raw response -
Text Recognition (OCR) (
ocr) - Model recognizes text in the image -
Visual Question Answering (
visual-question-answering) - Model answers the question you submit in the prompt -
Captioning (short) (
caption) - Model provides a short description of the image -
Captioning (
detailed-caption) - Model provides a long description of the image -
Single-Label Classification (
classification) - Model classifies the image content as one of the provided classes -
Multi-Label Classification (
multi-label-classification) - Model classifies the image content as one or more of the provided classes -
Structured Output Generation (
structured-answering) - Model returns a JSON response with the specified fields
You need to provide your OpenAI API key to use the GPT-4 with Vision model.
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/open_ai@v2to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
task_type |
str |
Task type to be performed by model. Value determines required parameters and output response.. | ❌ |
prompt |
str |
Text prompt to the OpenAI model. | ✅ |
output_structure |
Dict[str, str] |
Dictionary with structure of expected JSON response. | ❌ |
classes |
List[str] |
List of classes to be used. | ✅ |
api_key |
str |
Your OpenAI API key. | ✅ |
model_version |
str |
Model to be used. | ✅ |
image_detail |
str |
Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity.. | ✅ |
max_tokens |
int |
Maximum number of tokens the model can generate in it's response.. | ❌ |
temperature |
float |
Temperature to sample from the model - value in range 0.0-2.0, the higher - the more random / "creative" the generations are.. | ✅ |
max_concurrent_requests |
int |
Number of concurrent requests that can be executed by block when batch of input images provided. If not given - block defaults to value configured globally in Workflows Execution Engine. Please restrict if you hit OpenAI limits.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to OpenAI in version v2.
- inputs:
Circle Visualization,Background Color Visualization,Corner Visualization,Twilio SMS Notification,Slack Notification,VLM as Detector,LMM,Polygon Zone Visualization,Camera Focus,Image Slicer,Image Blur,Dot Visualization,Google Gemini,Roboflow Dataset Upload,Stability AI Inpainting,Pixelate Visualization,OpenAI,Gaze Detection,Image Convert Grayscale,Absolute Static Crop,Stability AI Image Generation,Webhook Sink,Color Visualization,Image Threshold,Halo Visualization,Polygon Visualization,VLM as Classifier,Dynamic Zone,CogVLM,Camera Calibration,Email Notification,Classification Label Visualization,Cosine Similarity,Single-Label Classification Model,Llama 3.2 Vision,Google Vision OCR,Roboflow Dataset Upload,Ellipse Visualization,Size Measurement,Bounding Box Visualization,Object Detection Model,Line Counter Visualization,Image Preprocessing,Trace Visualization,Label Visualization,Clip Comparison,Local File Sink,Image Slicer,Anthropic Claude,Crop Visualization,OCR Model,Relative Static Crop,Model Comparison Visualization,Stitch OCR Detections,Perspective Correction,OpenAI,Mask Visualization,Clip Comparison,Dynamic Crop,CSV Formatter,Florence-2 Model,Instance Segmentation Model,Keypoint Detection Model,Image Contours,Buffer,SIFT,Reference Path Visualization,Florence-2 Model,Triangle Visualization,Model Monitoring Inference Aggregator,SIFT Comparison,Keypoint Visualization,Identify Changes,Multi-Label Classification Model,LMM For Classification,Roboflow Custom Metadata,Grid Visualization,Stitch Images,Dimension Collapse,Blur Visualization - outputs:
Circle Visualization,Background Color Visualization,Corner Visualization,Twilio SMS Notification,Slack Notification,VLM as Detector,VLM as Classifier,LMM,Polygon Zone Visualization,Image Blur,Cache Set,Dot Visualization,Path Deviation,Google Gemini,Roboflow Dataset Upload,Stability AI Inpainting,Line Counter,OpenAI,Detections Consensus,Distance Measurement,Stability AI Image Generation,Webhook Sink,Color Visualization,Image Threshold,Halo Visualization,Polygon Visualization,Grid Visualization,VLM as Classifier,CogVLM,Instance Segmentation Model,Email Notification,Object Detection Model,Classification Label Visualization,Llama 3.2 Vision,Google Vision OCR,Roboflow Dataset Upload,Ellipse Visualization,Size Measurement,Pixel Color Count,Cache Get,Bounding Box Visualization,JSON Parser,Object Detection Model,Line Counter Visualization,Image Preprocessing,Keypoint Detection Model,Trace Visualization,Label Visualization,Clip Comparison,Local File Sink,Anthropic Claude,Crop Visualization,Detections Stitch,YOLO-World Model,Model Comparison Visualization,Perspective Correction,OpenAI,Path Deviation,Mask Visualization,Time in Zone,Clip Comparison,Time in Zone,Dynamic Crop,Florence-2 Model,Instance Segmentation Model,Keypoint Detection Model,Buffer,Florence-2 Model,Reference Path Visualization,CLIP Embedding Model,Triangle Visualization,Model Monitoring Inference Aggregator,SIFT Comparison,VLM as Detector,Keypoint Visualization,LMM For Classification,Roboflow Custom Metadata,Line Counter,Segment Anything 2 Model
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
OpenAI in version v2 has.
Bindings
-
input
images(image): The image to infer on..prompt(string): Text prompt to the OpenAI model.classes(list_of_values): List of classes to be used.api_key(Union[string,secret]): Your OpenAI API key.model_version(string): Model to be used.image_detail(string): Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..temperature(float): Temperature to sample from the model - value in range 0.0-2.0, the higher - the more random / "creative" the generations are..
-
output
output(Union[string,language_model_output]): String value ifstringor LLM / VLM output iflanguage_model_output.classes(list_of_values): List of values of any type.
Example JSON definition of step OpenAI in version v2
{
"name": "<your_step_name_here>",
"type": "roboflow_core/open_ai@v2",
"images": "$inputs.image",
"task_type": "<block_does_not_provide_example>",
"prompt": "my prompt",
"output_structure": {
"my_key": "description"
},
"classes": [
"class-a",
"class-b"
],
"api_key": "xxx-xxx",
"model_version": "gpt-4o",
"image_detail": "auto",
"max_tokens": "<block_does_not_provide_example>",
"temperature": "<block_does_not_provide_example>",
"max_concurrent_requests": "<block_does_not_provide_example>"
}
v1¶
Class: OpenAIBlockV1 (there are multiple versions of this block)
Source: inference.core.workflows.core_steps.models.foundation.openai.v1.OpenAIBlockV1
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
Ask a question to OpenAI's GPT-4 with Vision model.
You can specify arbitrary text prompts to the OpenAIBlock.
You need to provide your OpenAI API key to use the GPT-4 with Vision model.
This model was previously part of the LMM block.
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/open_ai@v1to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
prompt |
str |
Text prompt to the OpenAI model. | ✅ |
openai_api_key |
str |
Your OpenAI API key. | ✅ |
openai_model |
str |
Model to be used. | ✅ |
json_output_format |
Dict[str, str] |
Holds dictionary that maps name of requested output field into its description. | ❌ |
image_detail |
str |
Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity.. | ✅ |
max_tokens |
int |
Maximum number of tokens the model can generate in it's response.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to OpenAI in version v1.
- inputs:
Circle Visualization,Background Color Visualization,Corner Visualization,Twilio SMS Notification,Slack Notification,VLM as Detector,LMM,Polygon Zone Visualization,Camera Focus,Image Slicer,Image Blur,Dot Visualization,Google Gemini,Roboflow Dataset Upload,Stability AI Inpainting,Pixelate Visualization,OpenAI,Image Convert Grayscale,Absolute Static Crop,Stability AI Image Generation,Webhook Sink,Color Visualization,Image Threshold,Halo Visualization,Polygon Visualization,VLM as Classifier,CogVLM,Camera Calibration,Email Notification,Classification Label Visualization,Single-Label Classification Model,Llama 3.2 Vision,Google Vision OCR,Roboflow Dataset Upload,Ellipse Visualization,Bounding Box Visualization,Object Detection Model,Line Counter Visualization,Image Preprocessing,Trace Visualization,Label Visualization,Local File Sink,Image Slicer,Anthropic Claude,Crop Visualization,OCR Model,Relative Static Crop,Model Comparison Visualization,Stitch OCR Detections,Perspective Correction,OpenAI,Mask Visualization,Clip Comparison,Dynamic Crop,CSV Formatter,Florence-2 Model,Instance Segmentation Model,Keypoint Detection Model,Image Contours,SIFT,Reference Path Visualization,Florence-2 Model,Triangle Visualization,Model Monitoring Inference Aggregator,SIFT Comparison,Keypoint Visualization,Multi-Label Classification Model,LMM For Classification,Roboflow Custom Metadata,Grid Visualization,Stitch Images,Blur Visualization - outputs:
Corner Visualization,Slack Notification,VLM as Detector,VLM as Classifier,Polygon Zone Visualization,Image Slicer,Cache Set,Dot Visualization,Path Deviation,Roboflow Dataset Upload,Single-Label Classification Model,Dominant Color,Stability AI Image Generation,Image Convert Grayscale,Halo Visualization,Data Aggregator,Detections Classes Replacement,Dynamic Zone,CogVLM,Email Notification,Camera Calibration,Object Detection Model,Cosine Similarity,Single-Label Classification Model,Llama 3.2 Vision,Byte Tracker,Ellipse Visualization,Size Measurement,Pixel Color Count,Bounding Box Visualization,Line Counter Visualization,Image Preprocessing,Label Visualization,Local File Sink,Image Slicer,Detections Transformation,Anthropic Claude,Crop Visualization,Detections Stitch,OCR Model,Model Comparison Visualization,Relative Static Crop,QR Code Detection,Delta Filter,Path Deviation,Detections Filter,Clip Comparison,Time in Zone,CSV Formatter,Florence-2 Model,Keypoint Detection Model,Buffer,SIFT Comparison,Florence-2 Model,Multi-Label Classification Model,Rate Limiter,Keypoint Visualization,Identify Changes,Multi-Label Classification Model,Roboflow Custom Metadata,Line Counter,Stitch Images,Circle Visualization,Background Color Visualization,Twilio SMS Notification,Property Definition,LMM,Camera Focus,Image Blur,First Non Empty Or Default,Detections Merge,Google Gemini,Detection Offset,Stability AI Inpainting,Pixelate Visualization,Line Counter,OpenAI,Detections Consensus,Distance Measurement,Gaze Detection,Absolute Static Crop,Webhook Sink,Color Visualization,Image Threshold,Polygon Visualization,VLM as Classifier,Continue If,Instance Segmentation Model,Classification Label Visualization,Google Vision OCR,Roboflow Dataset Upload,Expression,Cache Get,JSON Parser,Object Detection Model,Keypoint Detection Model,Trace Visualization,Clip Comparison,Identify Outliers,YOLO-World Model,Stitch OCR Detections,Perspective Correction,OpenAI,Byte Tracker,Qwen2.5-VL,Mask Visualization,Time in Zone,Dynamic Crop,Template Matching,Byte Tracker,Barcode Detection,Instance Segmentation Model,Image Contours,SIFT,Reference Path Visualization,CLIP Embedding Model,Triangle Visualization,Model Monitoring Inference Aggregator,Bounding Rectangle,Velocity,SIFT Comparison,VLM as Detector,LMM For Classification,Grid Visualization,Segment Anything 2 Model,Detections Stabilizer,Dimension Collapse,Blur Visualization
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
OpenAI in version v1 has.
Bindings
-
input
images(image): The image to infer on..prompt(string): Text prompt to the OpenAI model.openai_api_key(Union[string,secret]): Your OpenAI API key.openai_model(string): Model to be used.image_detail(string): Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..
-
output
parent_id(parent_id): Identifier of parent for step output.root_parent_id(parent_id): Identifier of parent for step output.image(image_metadata): Dictionary with image metadata required by supervision.structured_output(dictionary): Dictionary.raw_output(string): String value.*(*): Equivalent of any element.
Example JSON definition of step OpenAI in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/open_ai@v1",
"images": "$inputs.image",
"prompt": "my prompt",
"openai_api_key": "xxx-xxx",
"openai_model": "gpt-4o",
"json_output_format": {
"count": "number of cats in the picture"
},
"image_detail": "auto",
"max_tokens": 450
}