OpenAI¶
v2¶
Class: OpenAIBlockV2 (there are multiple versions of this block)
Source: inference.core.workflows.core_steps.models.foundation.openai.v2.OpenAIBlockV2
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
Ask a question to OpenAI's GPT-4 with Vision model.
You can specify arbitrary text prompts or predefined ones, the block supports the following types of prompt:
-
Open Prompt (
unconstrained) - Use any prompt to generate a raw response -
Text Recognition (OCR) (
ocr) - Model recognizes text in the image -
Visual Question Answering (
visual-question-answering) - Model answers the question you submit in the prompt -
Captioning (short) (
caption) - Model provides a short description of the image -
Captioning (
detailed-caption) - Model provides a long description of the image -
Single-Label Classification (
classification) - Model classifies the image content as one of the provided classes -
Multi-Label Classification (
multi-label-classification) - Model classifies the image content as one or more of the provided classes -
Structured Output Generation (
structured-answering) - Model returns a JSON response with the specified fields
You need to provide your OpenAI API key to use the GPT-4 with Vision model.
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/open_ai@v2to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
task_type |
str |
Task type to be performed by model. Value determines required parameters and output response.. | ❌ |
prompt |
str |
Text prompt to the OpenAI model. | ✅ |
output_structure |
Dict[str, str] |
Dictionary with structure of expected JSON response. | ❌ |
classes |
List[str] |
List of classes to be used. | ✅ |
api_key |
str |
Your OpenAI API key. | ✅ |
model_version |
str |
Model to be used. | ✅ |
image_detail |
str |
Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity.. | ✅ |
max_tokens |
int |
Maximum number of tokens the model can generate in it's response.. | ❌ |
temperature |
float |
Temperature to sample from the model - value in range 0.0-2.0, the higher - the more random / "creative" the generations are.. | ✅ |
max_concurrent_requests |
int |
Number of concurrent requests that can be executed by block when batch of input images provided. If not given - block defaults to value configured globally in Workflows Execution Engine. Please restrict if you hit OpenAI limits.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to OpenAI in version v2.
- inputs:
Stitch Images,Pixelate Visualization,Multi-Label Classification Model,LMM For Classification,Gaze Detection,Blur Visualization,Single-Label Classification Model,Mask Visualization,OCR Model,Object Detection Model,SIFT,Model Monitoring Inference Aggregator,Polygon Visualization,Halo Visualization,Grid Visualization,Google Vision OCR,Model Comparison Visualization,Email Notification,Camera Focus,CogVLM,Image Threshold,Keypoint Visualization,Image Preprocessing,Roboflow Dataset Upload,Slack Notification,Stitch OCR Detections,Identify Changes,Relative Static Crop,Background Color Visualization,Clip Comparison,Bounding Box Visualization,Ellipse Visualization,Image Contours,Label Visualization,Classification Label Visualization,Line Counter Visualization,LMM,Stability AI Inpainting,Reference Path Visualization,VLM as Detector,Dynamic Crop,Triangle Visualization,Absolute Static Crop,Florence-2 Model,SIFT Comparison,Keypoint Detection Model,Corner Visualization,Perspective Correction,Cosine Similarity,Local File Sink,Polygon Zone Visualization,VLM as Classifier,Dimension Collapse,Image Slicer,Trace Visualization,Webhook Sink,OpenAI,Twilio SMS Notification,Roboflow Custom Metadata,Size Measurement,Crop Visualization,Instance Segmentation Model,Buffer,Roboflow Dataset Upload,Clip Comparison,Anthropic Claude,Dynamic Zone,Image Blur,Circle Visualization,Image Convert Grayscale,Dot Visualization,Google Gemini,Florence-2 Model,OpenAI,Color Visualization,CSV Formatter,Llama 3.2 Vision - outputs:
Path Deviation,LMM For Classification,Keypoint Detection Model,Line Counter,Instance Segmentation Model,CLIP Embedding Model,Mask Visualization,Object Detection Model,Line Counter,YOLO-World Model,Model Monitoring Inference Aggregator,Cache Get,Polygon Visualization,Halo Visualization,VLM as Detector,Grid Visualization,Google Vision OCR,Model Comparison Visualization,Email Notification,CogVLM,Image Threshold,Keypoint Visualization,Image Preprocessing,Roboflow Dataset Upload,Slack Notification,Background Color Visualization,Clip Comparison,Bounding Box Visualization,Ellipse Visualization,Classification Label Visualization,Label Visualization,Line Counter Visualization,LMM,Stability AI Inpainting,Reference Path Visualization,VLM as Detector,Dynamic Crop,Triangle Visualization,Object Detection Model,Distance Measurement,Time in Zone,Detections Stitch,Florence-2 Model,SIFT Comparison,Keypoint Detection Model,Corner Visualization,Perspective Correction,Local File Sink,Polygon Zone Visualization,VLM as Classifier,Twilio SMS Notification,Trace Visualization,Webhook Sink,Size Measurement,OpenAI,Roboflow Custom Metadata,Detections Consensus,Cache Set,Crop Visualization,Instance Segmentation Model,Buffer,Roboflow Dataset Upload,Clip Comparison,VLM as Classifier,Anthropic Claude,Image Blur,Circle Visualization,Dot Visualization,Google Gemini,Segment Anything 2 Model,JSON Parser,Florence-2 Model,Time in Zone,Path Deviation,OpenAI,Color Visualization,Pixel Color Count,Llama 3.2 Vision
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
OpenAI in version v2 has.
Bindings
-
input
images(image): The image to infer on.prompt(string): Text prompt to the OpenAI model.classes(list_of_values): List of classes to be used.api_key(Union[secret,string]): Your OpenAI API key.model_version(string): Model to be used.image_detail(string): Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..temperature(float): Temperature to sample from the model - value in range 0.0-2.0, the higher - the more random / "creative" the generations are..
-
output
output(Union[string,language_model_output]): String value ifstringor LLM / VLM output iflanguage_model_output.classes(list_of_values): List of values of any type.
Example JSON definition of step OpenAI in version v2
{
"name": "<your_step_name_here>",
"type": "roboflow_core/open_ai@v2",
"images": "$inputs.image",
"task_type": "<block_does_not_provide_example>",
"prompt": "my prompt",
"output_structure": {
"my_key": "description"
},
"classes": [
"class-a",
"class-b"
],
"api_key": "xxx-xxx",
"model_version": "gpt-4o",
"image_detail": "auto",
"max_tokens": "<block_does_not_provide_example>",
"temperature": "<block_does_not_provide_example>",
"max_concurrent_requests": "<block_does_not_provide_example>"
}
v1¶
Class: OpenAIBlockV1 (there are multiple versions of this block)
Source: inference.core.workflows.core_steps.models.foundation.openai.v1.OpenAIBlockV1
Warning: This block has multiple versions. Please refer to the specific version for details. You can learn more about how versions work here: Versioning
Ask a question to OpenAI's GPT-4 with Vision model.
You can specify arbitrary text prompts to the OpenAIBlock.
You need to provide your OpenAI API key to use the GPT-4 with Vision model.
This model was previously part of the LMM block.
Type identifier¶
Use the following identifier in step "type" field: roboflow_core/open_ai@v1to add the block as
as step in your workflow.
Properties¶
| Name | Type | Description | Refs |
|---|---|---|---|
name |
str |
Enter a unique identifier for this step.. | ❌ |
prompt |
str |
Text prompt to the OpenAI model. | ✅ |
openai_api_key |
str |
Your OpenAI API key. | ✅ |
openai_model |
str |
Model to be used. | ✅ |
json_output_format |
Dict[str, str] |
Holds dictionary that maps name of requested output field into its description. | ❌ |
image_detail |
str |
Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity.. | ✅ |
max_tokens |
int |
Maximum number of tokens the model can generate in it's response.. | ❌ |
The Refs column marks possibility to parametrise the property with dynamic values available
in workflow runtime. See Bindings for more info.
Available Connections¶
Compatible Blocks
Check what blocks you can connect to OpenAI in version v1.
- inputs:
Stitch Images,Pixelate Visualization,Multi-Label Classification Model,LMM For Classification,Blur Visualization,Single-Label Classification Model,Mask Visualization,OCR Model,Object Detection Model,SIFT,Model Monitoring Inference Aggregator,Polygon Visualization,Halo Visualization,Grid Visualization,Google Vision OCR,Model Comparison Visualization,Email Notification,Camera Focus,CogVLM,Image Threshold,Keypoint Visualization,Image Preprocessing,Roboflow Dataset Upload,Slack Notification,Stitch OCR Detections,Relative Static Crop,Background Color Visualization,Bounding Box Visualization,Ellipse Visualization,Image Contours,Label Visualization,Classification Label Visualization,Line Counter Visualization,LMM,Stability AI Inpainting,Reference Path Visualization,VLM as Detector,Dynamic Crop,Triangle Visualization,Absolute Static Crop,Florence-2 Model,SIFT Comparison,Keypoint Detection Model,Corner Visualization,Perspective Correction,Local File Sink,Polygon Zone Visualization,VLM as Classifier,Image Slicer,Trace Visualization,Webhook Sink,OpenAI,Twilio SMS Notification,Roboflow Custom Metadata,Crop Visualization,Instance Segmentation Model,Roboflow Dataset Upload,Clip Comparison,Anthropic Claude,Image Blur,Circle Visualization,Image Convert Grayscale,Dot Visualization,Google Gemini,Florence-2 Model,OpenAI,Color Visualization,CSV Formatter,Llama 3.2 Vision - outputs:
Pixelate Visualization,Gaze Detection,CLIP Embedding Model,Blur Visualization,OCR Model,Mask Visualization,Object Detection Model,SIFT,Line Counter,YOLO-World Model,Cache Get,Halo Visualization,Grid Visualization,Google Vision OCR,Email Notification,Camera Focus,Image Threshold,Byte Tracker,Template Matching,Image Preprocessing,Roboflow Dataset Upload,Relative Static Crop,Background Color Visualization,Bounding Box Visualization,Image Contours,Triangle Visualization,Bounding Rectangle,Absolute Static Crop,Distance Measurement,Time in Zone,Detections Stitch,Florence-2 Model,SIFT Comparison,Keypoint Detection Model,Local File Sink,Expression,Roboflow Custom Metadata,Cache Set,Crop Visualization,Clip Comparison,Dynamic Zone,SIFT Comparison,Image Convert Grayscale,Single-Label Classification Model,Identify Outliers,Florence-2 Model,Time in Zone,Path Deviation,OpenAI,Color Visualization,Pixel Color Count,Multi-Label Classification Model,Property Definition,Path Deviation,Multi-Label Classification Model,Stitch Images,LMM For Classification,First Non Empty Or Default,Keypoint Detection Model,Line Counter,Instance Segmentation Model,Rate Limiter,Single-Label Classification Model,Detections Filter,Model Monitoring Inference Aggregator,Polygon Visualization,VLM as Detector,Model Comparison Visualization,CogVLM,Keypoint Visualization,Detections Classes Replacement,Data Aggregator,Detection Offset,Slack Notification,Stitch OCR Detections,Identify Changes,Clip Comparison,Ellipse Visualization,Classification Label Visualization,Label Visualization,Line Counter Visualization,Byte Tracker,LMM,Stability AI Inpainting,Reference Path Visualization,VLM as Detector,Dynamic Crop,Dominant Color,Byte Tracker,Object Detection Model,Barcode Detection,Corner Visualization,Perspective Correction,Cosine Similarity,Polygon Zone Visualization,VLM as Classifier,Dimension Collapse,Continue If,Twilio SMS Notification,Trace Visualization,Detections Consensus,Webhook Sink,Size Measurement,OpenAI,Image Slicer,Instance Segmentation Model,Buffer,Roboflow Dataset Upload,VLM as Classifier,Anthropic Claude,Image Blur,Circle Visualization,Dot Visualization,Google Gemini,QR Code Detection,Segment Anything 2 Model,JSON Parser,Detections Stabilizer,Delta Filter,CSV Formatter,Llama 3.2 Vision,Detections Transformation
Input and Output Bindings¶
The available connections depend on its binding kinds. Check what binding kinds
OpenAI in version v1 has.
Bindings
-
input
images(image): The image to infer on.prompt(string): Text prompt to the OpenAI model.openai_api_key(Union[secret,string]): Your OpenAI API key.openai_model(string): Model to be used.image_detail(string): Indicates the image's quality, with 'high' suggesting it is of high resolution and should be processed or displayed with high fidelity..
-
output
parent_id(parent_id): Identifier of parent for step output.root_parent_id(parent_id): Identifier of parent for step output.image(image_metadata): Dictionary with image metadata required by supervision.structured_output(dictionary): Dictionary.raw_output(string): String value.*(*): Equivalent of any element.
Example JSON definition of step OpenAI in version v1
{
"name": "<your_step_name_here>",
"type": "roboflow_core/open_ai@v1",
"images": "$inputs.image",
"prompt": "my prompt",
"openai_api_key": "xxx-xxx",
"openai_model": "gpt-4o",
"json_output_format": {
"count": "number of cats in the picture"
},
"image_detail": "auto",
"max_tokens": 450
}