Table of Contents:

1. Introduction
2. Design of the System
3. Implementation of the Server-side
   1. Frame Grabber
   2. Stream Server
4. Implementation of the Client-side
   1. Stream Client
   2. Video Player
5. Compiling
6. Experiments & Gallery
7. Summary
8. Resources

1. Introduction

One of the most asked questions by OpenCV users is, "How can I stream videos over the network so I can process it on different computer?" OpenCV does not provide such a function by nature (or so I thought), so we have to write custom code to accomplish this task.

If you have experiences with network programming before, this should be quite easy. Just like when you send text files over the network, the same apply for this one. only this time we need to convert the received data into OpenCV's IplImage format.

In this tutorial, I will explain to you the system I built to stream OpenCV videos over TCP/IP network. Keep in mind that there are many ways to achieve this. Not to mention the freely available video streaming library such as ffmpeg and VLC. This is not about "this one is better", it is just about sharing the knowledge.

2. Design of the System

The system follows the client-server model. The computer that has the video input acts as the server. It waits for a client to connect and stream the videos once the connection has established. The diagram is shown below.

Fig 1. Streaming OpenCV videos over the network.

The diagram above shows several clients connect to the server and receive the streaming video simultaneously. However, to keep things simple, I made it that the server only accepts one client at a time.

If we look deeper into the server side, it should consists from two parts. One who read the video input in a loop, and one who waits for the client and send the video frames. It is impossible to have both parts as a single block of code, since they have to run simultaneously at the same time. To overcome this, we have to write a multi-threaded program.

The same also apply for the client side.

But another problem occurs, Windows and Unix-like systems have different way for handling with threads. While it is possible to write a code that compile and runs on both systems (using C preprocessor), it doesn't necessarily to. Let's just use Unix and throw Windows away.

In addition, I use Berkeley Sockets that is widely available on Unix-like systems for the networking code.

In summary, to make this as simplest as possible we keep these things in mind:

• The Operating System is Unix-like. Therefore we're using POSIX Threads and Berkeley Sockets. (If you use Windows, install Cygwin first).
• Only one client connected at a time.
• The client knows the width and the height of the expected frame.
• The client receives grayscaled (single channel) frames.

3. Implementation of the Server-side

The server side is the computer that has the video input to be streamed. And like I mentioned before, it consists of two parts. One who read the video input in a loop, and the other waits for the client to connect and send the video frames.

Fig 2. Stream server diagram.

In the diagram above, we see two threads running on the server side: Frame Grabber and Stream Server. The input is taken from a webcam, but you can use other resources too, like an avi file.

Frame Grabber grabs a frame from the webcam and stores it to a global variable img. Stream Server waits for a client to connect. Once the connection has established, it sends img to the client repeatedly whenever a newer version of img is available.

The full listing of the server side is in stream_server.c. Next we'll see the detail of both threads.

3.a. Frame Grabber

This is the main thread of the server-side. Its just like the usual code to display video from webcam. Below is the code snippet from stream_server.c.

Listing 1: Frame Grabber

/* run the stream server */ 
if (pthread_create(&thread_s, NULL, streamServer, NULL)) { 
    quit("pthread_create failed.", 1); 
} 
 
while(key != 'q') 
{ 
    /* get a frame */ 
    img0 = cvQueryFrame(capture); 
 
    /* convert to grayscale, thread safe */ 
    
    pthread_mutex_lock(&mutex); 
    
    cvCvtColor(img0, img1, CV_BGR2GRAY); 
    is_data_ready = 1; 
    
    pthread_mutex_unlock(&mutex); 
    
    cvShowImage("stream_server", img0); 
    key = cvWaitKey(30); 
} 
 
/* stop the stream server */ 
if (pthread_cancel(thread_s)) { 
    quit("pthread_cancel failed.", 1); 
} 

The code above should somehow look familiar with you. It grabs a frame in a loop, save it to a global variable img1, and wait until the user press the 'q' button.

But there are also some additional lines you should have notices. When the program starts, it creates a new thread for the streaming server by the line:


pthread_create(&thread_s, NULL, streamServer, NULL); 

After that, the function streamServer runs simultaneously at the same time with the main code. When the program finish (the user pressed the 'q' button) the main code terminates the streamServer thread:


pthread_cancel(thread_s); 

Also note that the global variable img1 is used by both threads. Frame Grabber writes it, while Stream Server reads it. It is important to avoid both threads access img1 at the same time, since it will give unwanted result. So we add a lock mechanism when writing img1.

pthread_mutex_lock(&mutex); 
cvCvtColor(img0, img1, CV_BGR2GRAY); 
is_data_ready = 1; 
pthread_mutex_unlock(&mutex);  

Just that simple. Now we move to the Stream Server code.

3.b. Stream Server

This thread waits for a client to connect to server on a predefined port. Once the connection has established, it sends the global variable img1 repeatedly whenever a newer version is available.

The code snippet of the main loop is shown below.

Listing 2: Stream Server

int imgsize = img1->imageSize; 
 
while(1) 
{ 
    /* send the grayscaled frame, thread safe */ 
    
    pthread_mutex_lock(&mutex); 
    
    if (is_data_ready) { 
        bytes = send(clientsock, img1->imageData, imgsize, 0); 
        is_data_ready = 0; 
    } 
    
    pthread_mutex_unlock(&mutex); 
 
    /* if something went wrong, restart the connection */ 
    if (bytes != imgsize) 
    { 
        fprintf(stderr, "Connection closed.\n"); 
        close(clientsock); 
 
        if ((clientsock = accept(serversock, NULL, NULL)) == -1) { 
            quit("accept() failed", 1); 
        } 
    } 
    
    pthread_testcancel(); 
    usleep(1000); 
} 

In the main loop above, the raw data of img1 is sent over the network if a newer version is available.

if (is_data_ready) { 
    bytes = send(clientsock, img1->imageData, imgsize, 0); 
    is_data_ready = 0; 
}  

Note that we make it thread safe by enclosing the lines above with pthread_mutex_lock().

The return value of send() is the number of bytes actually sent, or -1 on error. We need to check the return value in case something wrong has happened, like the client closing the connection. To make things simple, we're just restart the connection if something isn't right.

if (bytes != imgsize) { 
    close(clientsock); 
    if ((clientsock = accept(serversock, NULL, NULL)) == -1) { 
        quit("accept() failed", 1); 
    } 
}  

And the line:

pthread_testcancel();  

checks if main code has issued pthread_cancel() to terminate this thread.

4. Implementation of the Client-side

With the server now has set up and ready to accept connection, now we want to write a client to receive and display the streaming video.

Just like the code at the server side, this one consists of two parts. One who connects to the server and receives the frames, and the other display the received frame whenever a newer version exist.

Fig 3. Stream client diagram.

From the diagram above, we see that there are 2 threads running at the client code. Stream Client receives frames from server and store it to img, meanwhile Video Player reads img and display it in a player window.

Since the client receives frames from a TCP/IP network, it must specify the server's IP address and port number. They are passed as command line arguments when we run the code:

./stream_client 192.168.0.1 8888 320 240

The last 2 arguments are the width and the height of the expected frame. Keep in mind that the server sends only the data, without the image header. So the client must know the width and the height of the expected frame.

Q: Why don't we make it that the image header also being sent?
A: For the sake of code simplicity.

Q: Where do we find the width and the height of the expected frame?
A: From stream_server.c. It prints out the width and the height of the frame it captured.

Q: What if I put numbers that don't match with the received frame?
A: It will give unpredictable results, if you're lucky.

The full listing of the client side is in stream_client.c. Now we move into the details of the client side.

4.a. Stream Client

This thread connects to the server, given its IP address and port number. Once the connection has established, it receives the frames sent from server.

Below is the code snippet from stream_client.c.

Listing 3: Stream Client

width  = atoi(argv[3]); 
height = atoi(argv[4]); 
img    = cvCreateImage(cvSize(width, height), IPL_DEPTH_8U, 1); 
 
... 
 
int  imgsize = img->imageSize; 
char sockdata[imgsize]; 
 
while(1) 
{ 
    for (i = 0; i < imgsize; i += bytes) { 
        if ((bytes = recv(sock, sockdata + i, imgsize - i, 0)) == -1) { 
            quit("recv failed", 1); 
        } 
    } 
 
    /* convert the received data to OpenCV's IplImage format, 
       thread safe */ 
    
    pthread_mutex_lock(&mutex); 
 
    for (i = 0, k = 0; i < img->height; i++) { 
        for (j = 0; j < img->width; j++) { 
            ((uchar*)(img->imageData + i * img->widthStep))[j] = sockdata[k++]; 
        } 
    } 
    
    is_data_ready = 1; 
    
    pthread_mutex_unlock(&mutex); 
    pthread_testcancel(); 
    usleep(1000); 
} 

In the loop above, the client receives the data sent by server with recv() function. However, there is a chance that the delivery of an image is divided into several packets. So we put recv() in a loop to guarantee that we receive a complete image.

for (i = 0; i < imgsize; i += bytes) { 
    if ((bytes = recv(sock, sockdata + i, imgsize - i, 0)) == -1) { 
        quit("recv failed", 1); 
    } 
}  

If nothing went wrong, now we have the image data in sockdata. We convert the data to OpenCV's IplImage format by copying it to img.

for (i = 0, k = 0; i < img->height; i++) { 
    for (j = 0; j < img->width; j++) { 
        ((uchar*)(img->imageData + i * img->widthStep))[j] = sockdata[k++]; 
    } 
}  

The frame has successfully transmitted! Now its the turn for Video Player to read img and display it in a player window.

4.b. Video Player

This should be the easiest part from the whole system. Basically it just display the image in a loop.

Listing 4: Video Player

/* run the stream client thread */ 
if (pthread_create(&thread_c, NULL, streamClient, NULL)) { 
    quit("pthread_create failed.", 1); 
} 
 
cvNamedWindow("stream_client", CV_WINDOW_AUTOSIZE); 
 
while(key != 'q') 
{ 
    pthread_mutex_lock(&mutex); 
    
    if (is_data_ready) { 
        cvShowImage("stream_client", img); 
        is_data_ready = 0; 
    } 
    
    pthread_mutex_unlock(&mutex); 
 
    key = cvWaitKey(10); 
} 
 
/* terminate the stream client thread */ 
if (pthread_cancel(thread_c)) { 
    quit("pthread_cancel failed.", 1); 
} 

First, run the Stream Client thread:

pthread_create(&thread_c, NULL, streamClient, NULL);

After that, Stream Client does its job nicely. It receives the frames and store it in the global variable img. And displaying the video cannot be simpler:

cvShowImage("stream_client", img);

Don't forget to terminate Stream Client when the program finished:

pthread_cancel(thread_c);

What else?

5. Compiling

With stream_server.c and stream_client.c now in hand, we want to compile and try to stream our cool videos over the network. Here it is.

Compiling in *nix:

$ gcc stream_server.c -o stream_server \
      `pkg-config --cflags opencv` \
      `pkg-config --libs opencv` -lpthread

$ gcc stream_client.c -o stream_client \
      `pkg-config --cflags opencv` \
      `pkg-config --libs opencv` -lpthread

Compiling in Windows (under Cygwin):

$ gcc stream_server.c -o stream_server \
      -I"C:\OpenCV\cxcore\include" \
      -I"C:\OpenCV\cv\include" \
      -I"C:\OpenCV\otherlibs\highgui" \
      -L"C:\OpenCV\lib" -lcxcore -lcv -lhighgui -lpthread

$ gcc stream_client.c -o stream_client \
      -I"C:\OpenCV\cxcore\include" \
      -I"C:\OpenCV\cv\include" \
      -I"C:\OpenCV\otherlibs\highgui" \
      -L"C:\OpenCV\lib" -lcxcore -lcv -lhighgui -lpthread

Replace C:\OpenCV with the directory where you installed OpenCV.

6. Experiments & Gallery

I tested the system on both Windows and BSD operating systems. The thing is, I only got one laptop, so I use VMWare to obtain virtual PCs and simulate TCP/IP network.

In my first test, I ran the stream server and stream client on Windows. The input was taken from an avi file which I obtained from Learning OpenCV's code samples. The screenshot is shown below. Click it to view its original size.

Fig 4. Windows - Windows streaming video.

In my second test, I ran the stream server on Windows and stream client on BSD. The input was taken from webcam. Here it is.

Fig 5. Windows - BSD streaming video.

Note that I added face detection code at the client side. Obviously, you can do any image/video processing to the received video. That's the point of this project.

In my third test, I ran the stream server and stream client on BSD. The input was the movie 300, in MPEG format. Note that you should compile OpenCV with ffmpeg for reading MPEG files.

Fig 6. BSD - BSD streaming video.

And the last, BSD - Windows streaming server. The input was the movie Defiance, in MPEG format. Again, I added face detection code at the client side.

Fig 7. BSD - Windows streaming video.

As you can see, the system runs very well and smooth. However, this streaming system uses a lot of bandwidth. This is true since the server streams raw data. You may have to add some video compression if you want to use the system on the Internet.

7. Summary

In this article, I have showed you how to stream your OpenCV videos over TCP/IP network. With this system, you can have your webcam attached on a computer, and your video processing program resides on different computer.

While the project is far from perfect, generally it is all about ideas. Here are some other things that I'm planning to add to the code:

• Supports multiple clients.
• Supports multiple webcams.
• Streams Motion JPEG rather than raw data.
• Web server to simulate IP cameras.
• Video recording.
• And many more.

If you encounter any problem compiling and running the code above, feel free to mail me@nashruddin.com. Feedbacks are highly appreciated.

8. Resources

• Berkeley Sockets
• Beej's Guide to Network Programming
• Posix Thread Programming
• Multi-Threaded Programming with POSIX Threads