Using NVIDIA DeepStream with docker containers

DeepStream is a highly optimized SDK to run video, audio, and image analytics in real time. It is based on GStreamer, an extensible, battle tested framework that supports many plugins out-of-the-box. 

By using DeepStream, writing turnkey applications that meet our customer needs becomes really easy, since we use python to deal with high-level abstractions, knowing that our code will be blazingly fast, getting the best of both worlds. This is possible due to its bindings to C++ and CUDA behind the scenes, combined with the threaded nature of GStreamer.

In the end, we will be able to run our DeepStream application in any hardware that provides an NVIDIA GPU. This can be either a laptop or a server, which have discrete GPUs (dGPU) and x86_64 architectures, or NVIDIA’s embedded devices, the Jetsons, which have an aarch64 architecture.

Setting up the development environment

The first step in using DeepStream requires us to set up a development environment, and for that we will use docker containers, yay! This also plays nicely with VSCode devcontainers, which allow you to open your workspace transparently inside a docker container with all the dependencies in place, so that you can start coding right away.

Up until DeepStream 6.2 NVIDIA provided a development container that works on computers with dGPUs: nvcr.io/nvidia/deepstream:6.2-devel, but since DeepStream 6.3 they have unified both Jetson’s and dGPU’s into a multiarch container: docker pull nvcr.io/nvidia/deepstream:6.3-gc-triton-devel. Isn’t that cool? We can check the NVIDIA’s documentation to run DeepStream using docker containers here: https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_docker_containers.html

Because we use python bindings, we need to install a wheel python package inside the container, so we can write a simple dockerfile, build it, and tag it accordingly, for instance: deepstream:6.3-devel-py. Instructions can be found in the GitHub repository: https://github.com/NVIDIA-AI-IOT/deepstream_python_apps.

Finally, we need to define a .devcontainer folder containing a devcontainer.json file, where we will specify the docker image that we want to open, which non-root user is defined in the container, what directories we want to mount onto the docker container other than the workspace, which is mounted by default, the runtime arguments, and we can even specify which VSCode extensions we want to install inside the container! Here you will find the complete documentation: https://code.visualstudio.com/docs/devcontainers/containers

Writing our DeepStream application

Our workspace will consist, roughly, of the following directories:

app: Where we have our python code.
config: Where we store the configuration files for the app, and for the different DeepStream plugins, like nvinfer or nvtracker.
models: Where we store the onnx (open standard format) files for each model.

In the GitHub repository for deepstream_python_apps there are plenty of examples to get started, but the basic idea of the app is to connect plugins together to form a pipeline. 

This pipeline will have one or more input elements, which can be camera streaming feeds, or video files, for instance, and it will have output elements —some examples are an onscreen display, saving to a video file, or a fakesink, which will ingest any stream silently, ideal for a production environment! 

Between the input and output elements is where we will bake the DeepStream plugins that will do the magic:

We can cascade nvinfer plugins that will run inference with the configured models, so that we can have models working on the output of a previous model. A typical example is to run an object detection model followed by an object classification on the resulting crop.
We can add an nvtracker after the nvinfer that will keep the same ID across frames on the detected objects.
We can add an nvdsanalytics plugin that will check for defined line crosses, or count objects in a given region of interest.

Many more plugins are available! Check the docs here:https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_Intro.html

Finally, we can also write our own custom plugins, and we can add probes to the input or the output of each plugin, where we can e.g. extract metadata for detections and send this information to a message broker.

Deploying the application to a production environment

The last step is to deploy the application to run somewhere in a production environment. Because we use docker containers, this is also easy!

Up until DeepStream 6.2 NVIDIA provided again separate containers for dGPU and for the Jetson, but from DeepStream 6.3 they have been unified. In addition, NVIDIA released a guide to prepare your own Dockerfiles to pull in the dependencies you require for each particular project, and to optimize the size of the resulting image: https://github.com/NVIDIA-AI-IOT/deepstream_dockers

If deploying on a machine with dGPU then we just need to install the NVIDIA drivers, docker, and nvidia-container-toolkit. The rest will be included in the docker container.

On the other hand, if deploying on a Jetson, we need to flash it with the NVIDIA matching BSP—this is installing the OS—, and then we just need to install the nvidia-container dependency.