Working with Video Using OpenCV and QT - Part 2

This tutorial was written because of a request from a previous tutorial. In this tutorial, we improve upon the work done in that tutorial by adding a track-bar and display duration of the video. Also a few errors, I discovered will also be corrected in this tutorial. So let get cracking.

Add widgets to GUI

This is an extension to a previous tutorial so I will only point out changes or additions to the previous work so as not to repeat myself. Open the mainwindow.ui file, this file could be edited manually but for this tutorial we would use the designer.

• Add a horizontal track-bar to the GUI, this would be used to the adjust the position in the video.  
• Add two labels to the GUI; place one on the left and another on the right of the horizontal track-bar ( or however suits your needs) the left one would show the current time into the video whereas the right  one would be used to show the total time of the video.  The GUI should now look similar to the image above.

Player Class Definition

Now we add a few function definitions to Player class header file -player.h.

#ifndef PLAYER_H
#ifndef PLAYER_H
#define PLAYER_H
#include <QMutex>
#include <QThread>
#include <QImage>
#include <QWaitCondition>
#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
using namespace cv;
class Player : public QThread
{    Q_OBJECT
 private:
    ....
    VideoCapture *capture;
    Mat RGBframe;
    QImage img;
 signals:
     ......
 protected:
     ......
 public:
    .......
    //set video properties 
    void setCurrentFrame( int frameNumber);
    

    //Get video properties
    double getFrameRate();
    double getCurrentFrame();
    double getNumberOfFrames();
};

#endif // VIDEOPLAYER_H

The class definition is still simple and straightforward. We add a few public setter and getter functions to enable us grab some important video parameters. Also I change the capture variable to a pointer I discovered that it was not possible to reload a new video once one has been loaded so to correct this I initialize a new VideoCapture instance when a new video is loaded. So the method used to access the VideoCapture instance members must be changed from the "." to the "->" notation.

Player Class Implementation

Here is the constructor for the Player class.

bool Player::loadVideo(string filename) {
    capture  =  new cv::VideoCapture(filename);

    if (capture->isOpened())
    {
        frameRate = (int) capture->get(CV_CAP_PROP_FPS);
        return true;
    }
    else
        return false;
}

void Player::run()
{
    int delay = (1000/frameRate);
    while(!stop){
        if (!capture->read(frame))
        {
            stop = true;
        }
        .....
    }
}

In the loadVideo() method, we use the instance of the VideoCapture class to load the video and set the frame rate. As you should already know the VideoCapture class is from the OpenCV library

double Player::getCurrentFrame(){

    return capture->get(CV_CAP_PROP_POS_FRAMES);
}

double Player::getNumberOfFrames(){

    return capture->get(CV_CAP_PROP_FRAME_COUNT);
}

double Player::getFrameRate(){
    return frameRate;
}

void Player::setCurrentFrame( int frameNumber )
{
    capture->set(CV_CAP_PROP_POS_FRAMES, frameNumber);
}

Here are the getter and setter function to access the video information. I ran into a glitch, with ffmpeg where the capture->get(CV_CAP_PROP_FRAME_COUNT) function returned the wrong frame count. The solution might be to update my ffmpeg. I would update the post once the solution is confirmed.

Player::~Player()
{
    mutex.lock();
    stop = true;
    capture.release();
    delete capture;
    condition.wakeOne();
    mutex.unlock();
    wait();
}

Here is the rest of the Player class, in the destructor we release the VideoCapture object, also we allocated memory with the new keyword it must be freed.

MainWindow Class Definition

#ifndef MAINWINDOW_H
#define MAINWINDOW_H
#include <QMainWindow>
#include <QFileDialog>
#include <QMessageBox>
#include <player.h>
#include <QTime>

namespace Ui {
class MainWindow;
}
class MainWindow : public QMainWindow
{
    Q_OBJECT
    
public:
    .......
    
private slots:
    .........
    QString getFormattedTime(int timeInSeconds);

    void on_horizontalSlider_sliderPressed();

    void on_horizontalSlider_sliderReleased();

    void on_horizontalSlider_sliderMoved(int position);
private:
    ........
};
#endif // MAINWINDOW_H

Here is the class definition for the MainWindow class, we include the 3 event slots for the new horizontal slider and a update some other functions

Mainwindow class implementation

MainWindow::MainWindow(QWidget *parent) :
    QMainWindow(parent),
    ui(new Ui::MainWindow)
{
    .......

    ui->pushButton_2->setEnabled(false);
    ui->horizontalSlider->setEnabled(false);
}

We disable the play button and the horizontal slider; these would be enabled once a video is loaded.

void MainWindow::updatePlayerUI(QImage img)
{
    if (!img.isNull())
    {
        ui->label->setAlignment(Qt::AlignCenter);
        ui->label->setPixmap(QPixmap::fromImage(img).scaled(ui->label->size(),
                                           Qt::KeepAspectRatio, Qt::FastTransformation));
        ui->horizontalSlider->setValue(myPlayer->getCurrentFrame());
        ui->label_2->setText( getFormattedTime( (int)myPlayer->getCurrentFrame()/(int)myPlayer->getFrameRate()) );
    }
}

The updatePlayerUI slot receives a QImage and resizes it to fit the label (keeping the aspect ratio) which will be used to display. It displays the image by setting the label pixmap. We also update the horizontal slider position and the label that displays the elapsed time. We calculate the duration of the video but dividing the total number of frames by the frame rate.

void MainWindow::on_pushButton_clicked()
{
    QString filename = QFileDialog::getOpenFileName(this,
                                          tr("Open Video"), ".",
                                          tr("Video Files (*.avi *.mpg *.mp4)"));
    QFileInfo name = filename;

    if (!filename.isEmpty()){
        if (!myPlayer->loadVideo(filename.toAscii().data()))
        {
            QMessageBox msgBox;
            msgBox.setText("The selected video could not be opened!");
            msgBox.exec();
        }
        else{
            this->setWindowTitle(name.fileName());
            ui->pushButton_2->setEnabled(true);
            ui->horizontalSlider->setEnabled(true);
            ui->horizontalSlider->setMaximum(myPlayer->getNumberOfFrames());
            ui->label_3->setText( getFormattedTime( (int)myPlayer->getNumberOfFrames()/(int)myPlayer->getFrameRate()) );
        }
    }
}

QString MainWindow::getFormattedTime(int timeInSeconds){

    int seconds = (int) (timeInSeconds) % 60 ;
    int minutes = (int) ((timeInSeconds / 60) % 60);
    int hours   = (int) ((timeInSeconds / (60*60)) % 24);

    QTime t(hours, minutes, seconds);
    if (hours == 0 )
        return t.toString("mm:ss");
    else
        return t.toString("h:mm:ss");
}

void MainWindow::on_horizontalSlider_sliderPressed()
{
    myPlayer->Stop();
}

void MainWindow::on_horizontalSlider_sliderReleased()
{
    myPlayer->Play();
}

void MainWindow::on_horizontalSlider_sliderMoved(int position)
{
    myPlayer->setCurrentFrame(position);
    ui->label_2->setText( getFormattedTime( position/(int)myPlayer->getFrameRate()) );
}

This is the remaining part of the MainWindow Class, the getFormattedTime function takes the time in seconds and formats it for display, the rest is self explanatory. You can Download the full code Here.

Final words...

This is just a simple tutorial to help anyone get started with videos in OpenCV and QT. Please let me know if this was helpful and ask questions and give suggestions(if any) in the comments. Happy Coding!

Working with Video Using OpenCV and QT

Video processing is a very important task in computer vision applications. OpenCV comes with its own GUI library (Highgui); but this library has no support for buttons and some other GUI components. Therefore it could be  preferable to use a QT GUI application, but displaying a video in a QT GUI is not as intuitive as it is with Highgui. This tutorial will show you how to display video in a QT GUI, without the GUI becoming unresponsive. We would be creating a simple video player as shown below and would be programming in C++.

STEP 1: Create a new QT GUI Project

If you don't know how to do this check out the guide here. The guide shows you how to create an OpenCV console project in Qt-creator, but this time instead of using a Qt Console Application, create a Qt GUI Application. Once created the following files are automatically added into the project:

main.cpp	This contains the main function which is the starting point of all C++ applications. It is the main function the loads the main window for the GUI application.
mainwindow.cpp	This is the source file that contains the MainWindow class implementation.
mainwindow.h	This contains the class declaration for the MainWindow class.
mainwindow.ui	This is the UI designer file that could be used to tweak the GUI.
<projectName>.pro	This contains settings that are used for the project compilation.

Add widgets to GUI

Open the mainwindow.ui file, this file could be edited manually but for this tutorial we would use the designer.

• From the list of widgets on the left of the designer, drag in a label and two pushbutton widgets.
• Change the text on the first button to “Load Video” and on the second button to “Play”. Then Clear the text on the label. To change the text on a widget just double click on the widget the text would be highlighted, you can now change it and press enter when finished.
• Change the background colour of the label to a darker colour. The best way to change the background colour of a QT widget is to use Cascading StyleSheets or CSS. Select the label find the stylesheet property in the property window click on the button with Three dots (ellipsis) and add this line of CSS into the “Edit Style sheet” window and save it.

  Background-color: #000;

• The GUI should now look something like this:

Player Class Definition

Now we add a new class to handle our video player control, we will call this the “Player” Class the following class definition should be added in player.h

#ifndef PLAYER_H
#define PLAYER_H
#include <QMutex>
#include <QThread>
#include <QImage>
#include <QWaitCondition>
#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
using namespace cv;
class Player : public QThread
{    Q_OBJECT
 private:
    bool stop;
    QMutex mutex;
    QWaitCondition condition;
    Mat frame;
    int frameRate;
    VideoCapture capture;
    Mat RGBframe;
    QImage img;
 signals:
 //Signal to output frame to be displayed
      void processedImage(const QImage &image);
 protected:
     void run();
     void msleep(int ms);
 public:
    //Constructor
    Player(QObject *parent = 0);
    //Destructor
    ~Player();
    //Load a video from memory
    bool loadVideo(string filename);
    //Play the video
    void Play();
    //Stop the video
    void Stop();
    //check if the player has been stopped
    bool isStopped() const;
};
#endif // VIDEOPLAYER_H

The class definition is simple and straightforward. The first thing to note is that Player class inherits from the QThread Class which will allow it to run on its own thread this is very important so that the main window remains responsive while the video is playing without this the video will cause the screen to freeze until it has finished playing. The processedImage(...) signal will be used to output the video frames to the main window (we would see how this work later).

Player Class Implementation

Here is the constructor for the Player class

Player::Player(QObject *parent)
 : QThread(parent)
{
    stop = true;
}

Here we simply initialise the value of the class variable stop

bool Player::loadVideo(string filename) {
    capture.open(filename);
    if (capture.isOpened())
    {
        frameRate = (int) capture.get(CV_CAP_PROP_FPS);
        return true;
    }
    else
        return false;
}

In the loadVideo() method, we use the instance of the VideoCapture class to load the video and set the frame rate. As you should already know the VideoCapture class is from the OpenCV library

void Player::Play()
{
    if (!isRunning()) {
        if (isStopped()){
            stop = false;
        }
        start(LowPriority);
    }
}

The public method play() simply starts the thread by calling the run() method which is an override of the QThread run method.

void Player::run()
{
    int delay = (1000/frameRate);
    while(!stop){
        if (!capture.read(frame))
        {
            stop = true;
        }
        if (frame.channels()== 3){
            cv::cvtColor(frame, RGBframe, CV_BGR2RGB);
            img = QImage((const unsigned char*)(RGBframe.data),
                              RGBframe.cols,RGBframe.rows,QImage::Format_RGB888);
        }
        else
        {
            img = QImage((const unsigned char*)(frame.data),
                                 frame.cols,frame.rows,QImage::Format_Indexed8);
        }
        emit processedImage(img);
        this->msleep(delay);
    }
}

In the run method, we utilise a while loop to play the video after reading the frame, it is converted into a QImage and the QImage is emitted to the MainWindow object using the processedImage(...) signal; at the end of the loop we wait for a number of milliseconds (delay) which is calculated using the frame rate of the video. If the video frame was been processed it would be advisable to factor the processing time into the delay.

Player::~Player()
{
    mutex.lock();
    stop = true;
    capture.release();
    condition.wakeOne();
    mutex.unlock();
    wait();
}
void Player::Stop()
{
    stop = true;
}
void Player::msleep(int ms){
    struct timespec ts = { ms / 1000, (ms % 1000) * 1000 * 1000 };
    nanosleep(&ts, NULL);
}
bool Player::isStopped() const{
    return this->stop;
}

Here is the rest of the Player class, in the destructor we release the VideoCapture object and wait for the run method to exit.

MainWindow Class Definition

#ifndef MAINWINDOW_H
#define MAINWINDOW_H
#include <QMainWindow>
#include <QFileDialog>
#include <QMessageBox>
#include <player.h>
namespace Ui {
class MainWindow;
}
class MainWindow : public QMainWindow
{
    Q_OBJECT
    
public:
    explicit MainWindow(QWidget *parent = 0);
    ~MainWindow();
    
private slots:
    //Display video frame in player UI
    void updatePlayerUI(QImage img);
    //Slot for the load video push button.
    void on_pushButton_clicked();
    // Slot for the play push button.
    void on_pushButton_2_clicked();
private:
    Ui::MainWindow *ui;
    Player* myPlayer;
};
#endif // MAINWINDOW_H

Here is the class definition for the MainWindow class, we include the clicked event slots for both buttons and a updatePlayerUI slot. We also include a myPlayer variable which is an instance of the Player Class

Mainwindow class implementation

MainWindow::MainWindow(QWidget *parent) :
    QMainWindow(parent),
    ui(new Ui::MainWindow)
{
    myPlayer = new Player();
    QObject::connect(myPlayer, SIGNAL(processedImage(QImage)),
                              this, SLOT(updatePlayerUI(QImage)));
    ui->setupUi(this);
}

we initialise myPlayer and we connect the signal emitted from the player class to the updatePlayerUI(...) slot, so every time a frame has been emitted it would be passed to this slot.

void MainWindow::updatePlayerUI(QImage img)
{
    if (!img.isNull())
    {
        ui->label->setAlignment(Qt::AlignCenter);
        ui->label->setPixmap(QPixmap::fromImage(img).scaled(ui->label->size()
                                           Qt::KeepAspectRatio, Qt::FastTransformation));
    }
}

The updatePlayerUI slot receives a QImage and resizes it to fit the label (keeping the aspect ratio) which will be used to display. It displays the image by setting the label pixmap

MainWindow::~MainWindow()
{
    delete myPlayer;
    delete ui;
}

void MainWindow::on_pushButton_clicked()
{
    QString filename = QFileDialog::getOpenFileName(this,
                                          tr("Open Video"), ".",
                                          tr("Video Files (*.avi *.mpg *.mp4)"));
    if (!filename.isEmpty()){
        if (!myPlayer->loadVideo(filename.toAscii().data()))
        {    
            QMessageBox msgBox;
            msgBox.setText("The selected video could not be opened!");
            msgBox.exec();
        }
    }
}
void MainWindow::on_pushButton_2_clicked()
{
    if (myPlayer->isStopped())
    {
        myPlayer->Play();
        ui->pushButton_2->setText(tr("Stop"));
    }else
    {
        myPlayer->Stop();
        ui->pushButton_2->setText(tr("Play"));
    }
}

This is the remaining part of the MainWindow Class, we have the destructor for the class, "load Video"(pushbutton) button slot and the "Play" (pushbutton_2) button slot which all pretty straightforward.

Main() Function

int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
    MainWindow* w = new MainWindow();
    w->setAttribute(Qt::WA_DeleteOnClose, true);

    w->show();
    
    return a.exec();
}

And finally, the main function we create an instance of the MainWindow class and set the delete on close attribute so that the objects created are destroyed.

Final words...

This is just a simple tutorial to help anyone get started with videos in OpenCV and QT. It should also be noted that there are other ways to handle videos in QT like the Phonon multimedia framework. Please let me know if this was helpful and ask questions (if any) in the comments. Happy Coding!
UPDATE: SEE PART 2 OF THIS TUTORIAL HERE. WE SHOW YOU HOW TO ADD A TRACK-BAR TO ALLOW THE USER CONTROL THE VIDEO.