Skip to content

Environmental vaiables

Inference behavior can be controlled by set of environmental variables. All environmental variables are listed in inference/core/env.py

Below is a list of some environmental values that require more in-depth explanation.

Environmental variable Description Default
ONNXRUNTIME_EXECUTION_PROVIDERS List of execution providers in priority order, warning message will be displayed if provider is not supported on user platform See here
SAM2_MAX_EMBEDDING_CACHE_SIZE The number of sam2 embeddings that will be held in memory. The embeddings will be held in gpu memory. Each embedding takes 16777216 bytes. 100
SAM2_MAX_LOGITS_CACHE_SIZE The number of sam2 logits that will be held in memory. The the logits will be in cpu memory. Each logit takes 262144 bytes. 1000
DISABLE_SAM2_LOGITS_CACHE If set to True, disables the caching of SAM2 logits. This can be useful for debugging or in scenarios where memory usage needs to be minimized, but may result in slower performance for repeated similar requests. False
ENABLE_WORKFLOWS_PROFILING If set to True, in inference server allows the server to output Workflows profiler traces the client, running in Python package with InferencePipeline it enables profiling. False
WORKFLOWS_PROFILER_BUFFER_SIZE Size of profiler buffer (number of consecutive Wrofklows Execution Engine run(...) invocations to trace in buffer. 64
RUNS_ON_JETSON Boolean flag to tell if inference runs on Jetson device - set to True in all docker builds for Jetson architecture. False
WORKFLOWS_DEFINITION_CACHE_EXPIRY Number of seconds to cache Workflows definitions as a result of get_workflow_specification(...) function call 15 * 60 - 15 minutes
DOCKER_SOCKET_PATH Path to the local socket mounted to the container - by default empty, if provided - enables pooling docker container stats from the docker deamon socket. See more here Not Set
ENABLE_PROMETHEUS Boolean flag to enable Prometeus /metrics enpoint. True for docker images in dockerhub
ENABLE_STREAM_API Flag to enable Stream Management API in inference server - see more. False
STREAM_API_PRELOADED_PROCESSES In context of Stream API - this environment variable controlls how many idle processes are warmed-up ready to be a worker for InferencePipeline - helps speeding up workers processes start on GPU 0