Low-latency Full-HD video streaming from Raspberry Pi

Raspberry Pi boards are getting more and more widespread. But when it comes to real-time video streaming, you may find yourself lost in a bunch of long-reptile shell commands! In this post, I will give you some crystal clear instructions to receive a low-latency stream from a CSI or USB camera. They key to achieve this is to do the h264 encoding on the RPi GPU (not CPU). The stream is then received frame-by-frame in an OpenCV code. You can later add text, layers, or do any other process you wish.

1. CSI camera


I generally recommend to use CSI cameras with Raspberry Pi because they are directly captured by the GPU. This means that the CPU remains free for other operations. You may find the official CSI cameras unsuitable for your project. If this is true, then check ArduCam. They have a wide variety of board cameras with different sensors and lenses.

Supposing that you want to stream from an RPi to an Ubuntu Laptop:

1- Install GStreamer in the laptop using:

sudo apt -y install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev

2- In the laptop, re-compile your OpenCV with -DWITH_GSTREAMER=ON flag in CMake. Make sure your new OpenCV recognizes gstreamer:

3- In the RPi, install gst-rpicamsrc

sudo apt-get install autoconf automake libtool pkg-config libgstreamer1.0-dev gstreamer1.0-tools

git clone https://github.com/thaytan/gst-rpicamsrc.git

cd gst-rpicamsrc

sudo ./autogen.sh --prefix=/usr --libdir=/usr/lib/arm-linux-gnueabihf/

sudo make -j4

sudo make install

4- Back in the laptop, build and run this code: 

#include <opencv2/opencv.hpp>

using namespace cv;

int main()
{
    VideoCapture cap("udpsrc port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! appsink sync=false");

    Mat frame;
    while (cap.isOpened())
    {
        cap >> frame;

        if (frame.empty())
            continue;

        imshow("Stream", frame);

        if (waitKey(1)=='q')
            break;
    }

    cap.release();
    return 0;
}

5- And last, in the Rpi run this command (replace host IP with your laptop’s IP):

gst-launch-1.0 rpicamsrc bitrate=6000000 ! 'video/x-h264,width=1920,height=1080' ! h264parse ! queue ! rtph264pay config-interval=1 pt=96 ! gdppay ! udpsink host=10.42.0.1 port=5000

To check the latency, I put the camera in front of a stopwatch and then captured an image by my cellphone. The camera sees 00:12:14:427 but shows 00:12:14:227. This means 200ms latency which is good for 1920×1080 resolution. Note that RPi CPU usage is below 5% and you can use it for other tasks. You can also increase bitrate to get a higher quality stream if your Laptop’s CPU is powerful enough to render the frames.

2. USB camera


USB cameras are more available than CSI’s. As I said, I don’t recommend them with RPi but in case you don’t have access to a CSI camera or want to save your budget by using an old webcam, then this section is for you.

Supposing that you want to stream from an RPi to an Ubuntu Laptop:

1- Make sure your laptop’s OpenCV support FFMpeg.

2– In the Rpi, build FFmpeg with h264 GPU encoding flags:

git clone https://github.com/FFmpeg/FFmpeg.git

cd FFmpeg

sudo ./configure --arch=armel --target-os=linux --enable-gpl --enable-omx --enable-omx-rpi –enable-nonfree

sudo make -j4

sudo make install

3Back in the laptop, build and run this code (replace the IP with your laptop’s IP):

#include <opencv2/opencv.hpp>

using namespace cv;

int main()
{
    VideoCapture cap("udp://10.42.0.1:5000");

    Mat frame;

    while (cap.isOpened())
    {
        cap >> frame;

        if (frame.empty())
            continue;

        imshow("Stream", frame);

        if (waitKey(1)=='q')
            break;
    }

    cap.release();
    return 0;
}

4And last, in the Rpi run this command (replace the IP with your laptop’s IP):

ffmpeg -s 1280x720 -i /dev/video0 -codec:v h264_omx -b:v 2048k -f mpegts udp://10.42.0.1:5000

Repeating the same stopwatch test shows that the USB camera has more latency (around 800ms). But I must mention that a part of this latency is due to my camera which provides only 10 FPS at HD resolution. Also, note that the RPi CPU is used more (17%) in this case because the frames are captured by the CPU and only the encoding is handled by the GPU. With a CSI camera, both capturing and encoding were handled by the GPU.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *

10 + one =

Related Posts: