# Segmentation & object detection by color.

In this tutorial i go to explain how to image segmentation or detect objects byred color, in this case by red color.

This task is simple, but there are some things we must known.

Now i go to explain and get a demo code for segmentation, how to determine if each image pixel is red or no, and then, i go to explain how we  can detect object, it's similar but with diferent concept.

In first moment, we can decide get the red channel of image  and get the values, if it's a higher value then we have a red pixel otherwise no.

This is incorrect. Why?, imagine we have a pixel and in his red channel have a 255 value, this is the higher value, then we decide this is a red pixel, but if in green and blue channel have 255 value too then we have a white pixel instead a red pixel. Then how we determine if a pixel is red or no?

Ok, we go to get some color values and determine how we decide if is red or no.

Then we can decide that higher red value, and  lower values of blue and green can be defined as red color. Moreover the green and blue values don't have a higher difference between his values.

We go to construct simple graph with some values to see better.

We see in this graph that the red value is higher than a max value and there are a min value for green and blue.

The min and max we go to name as blue green threshold and red threshold (bg_threshold, r_threshold).

This code compute for each pixel if it's red or no, and create a black/white image, this is our segmentation image, and with this image we can use to label each object, we use a cvFindContours to retreive the contours of each object detected on our image.

[c language="++"]
for(y=0;yheight; y++)
{
red=((uchar*)(img->imageData + img->widthStep*y))[x*3+2];
green=((uchar*)(img->imageData + img->widthStep*y))[x*3+1];
blue=((uchar*)(img->imageData + img->widthStep*y))[x*3];
uchar* temp_ptr=&((uchar*)(img_result_threshold->imageData + img_result_threshold->widthStep*y))[x];

if((red>threshold)&&(green < BG_threshold) && (blue < BG_threshold) && (abs(green-blue)< BG_diff)){
temp_ptr[0]=255;//White to greater of threshold
}else{
temp_ptr[0]=0;//Black other
}
}
}

cvMorphologyEx(img_result_threshold, img_morph, img_temp, NULL, CV_MOP_CLOSE, 6);

cvNamedWindow( "Threshold", 1 );
cvShowImage( "Threshold", img_morph );

cvFindContours( img_morph, storage, &contour, sizeof(CvContour), CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0) );

for( ; contour != 0; contour = contour->h_next )
{
CvScalar color = CV_RGB( rand()&255, rand()&255, rand()&255 );
cvDrawContours( img, contour, color, color, -1, 3, 8, cvPoint(0,0) );
}
[/c]

And this is the result, whe can see in the image a color border with detected red objects, the system detect two objects. The notebook and the notebook corner

Demo Source

This is perfect to image segmentation by color, but imagine we have objects ofone image, and we want get each object and know if this object have a predominant red color or no.

The theory is same but applied to the object histogram, and use the median to determine the thresholds

But with this we don't have enough, because we can get a gray image as we can see in the second histogram, then we need use another parameter more, the dispersion of each channel.

We can see that in this first histogram we can ensure this object have a predominant red, and in the second histogram no.

David completed his studies in Universidad Politecnica de Valencia in IT with a Master's degree in artificial intelligence, computer graphics, and pattern recognition, focusing on pattern recognition and Computer Vision. David has more than 15 years of experience in IT, with more than ten years of experience in Computer Vision, computer graphics, and pattern recognition, working on different projects and start-ups, applying his knowledge of Computer Vision, optical character recognition, and augmented reality. Co-Author of two OpenCV books and reviewer of few more.

1. I think that it's better convert the image to Lab space color and calculate the color distance.

2. Hi aclis, of course, there are better color segmentation. I want explain the more simply color segmentation in this tutorial. There are better color segmentation as this paper and another papers.

The purpose of my blog is not explain the better solution, i want create basic approximations to introduce people to OpenCV and his functions, and basics of Computer vision.

Regards David.

3. Thank you !

4. Hi,
Thanks for your documentation, but i can not use the given code , can you send me as full version of it ? or put it somewhere in your blog?

5. Noctis there are a demo source in the post with a full version, under the image capture.

6. act,i need full version of the code. so pl. if possible, send me the code on my mail.i will be very thankful to u.........

7. awinash, there are a demo with full version under image capture.

8. First of all this is one of a great tutorial for newbee on open cv, Thanks alot. I would like to know can we use the same segmentation on human skin? I have read on many places hsv is best to provide skin color.?

9. Thanks, of course, HSV is better for color tasks.

10. it means i just need to convert the rgb to hsv and also i need to have skin colors store in shape of threshold like you did for red color? OR do i need more steps for it? Once again thanks for your reply.

12. I tried something which is written in some online document that i have to do these following things.

RGB to HSV color space
Then HSV 's H should be between 6 and 38 range.

Then make all other things as black while pixels between that range should be 255 so that result would come black and white.

Then use filtering like dilate ,erode and smooth gaussian. But it doesnt work... can you give me any tip what i am doing mistake in it?

cvNamedWindow("Example1", CV_WINDOW_AUTOSIZE );

//cvErode( img, img, NULL, 1);
//cvDilate( img, img, NULL, 1);

IplImage* hsv = cvCreateImage( cvGetSize(img), IPL_DEPTH_8U, 3 );

cvCvtColor( img, hsv, CV_BGR2HSV );

for(int x=0; xheight; x++)
{
for(int y=0;ywidth; y++)
{
float H=(float)((uchar*)(hsv->imageData + hsv->widthStep*x))[y*3+2];
float S=(float)((uchar*)(hsv->imageData + hsv->widthStep*x))[y*3+1];
float V=(float)((uchar*)(hsv->imageData + hsv->widthStep*x))[y*3];

if(( V>6 && V0.23 && HimageData + hsv->widthStep*x))[y*3+2] =255;
((uchar*)(hsv->imageData + hsv->widthStep*x))[y*3+1] =255;
((uchar*)(hsv->imageData + hsv->widthStep*x))[y*3] =255;
}
else
{
((uchar*)(hsv->imageData + hsv->widthStep*x))[y*3+2] =0;
((uchar*)(hsv->imageData + hsv->widthStep*x))[y*3+1] =0;
((uchar*)(hsv->imageData + hsv->widthStep*x))[y*3] =0;
}

}
}

cvDilate( hsv, hsv, NULL, 1);
cvErode( hsv, hsv, NULL, 1);
cvSmooth( hsv, hsv, CV_MEDIAN, 3, 3 );

cvShowImage("Example1", hsv );
cvWaitKey(0);
cvReleaseImage( &img );
cvDestroyWindow("Example1");

13. I think you have the error in conditional statement. I don't see correctly this statmente, i don't understand it.

14. Hi Damiles,

I am very new using OpenCv. I've compiled the source code from demo source but it still doesn't work. I am confused why the name of the picture is not written in the source code? I don't know yet what argv [0], argv [1], etc means?

15. Hi, this is a basic concept in console applications, the argv[0] is the app name and argv[1] is the first parameter send to app argv[2] the second ...

Then when you execute the application, under console, you call ./my_app myimage.jpg and you are passing myimage.jpg as variable stored in argv[1].

Regards. Damiles.

16. wow, thanks for the fast response. Sorry for the late reply :) Actually I've compiled it and succesfully result an application file. Then I put the image in the same folder with app file. But, it still doesn't work. I compiled it using Dev-C++.

Actually, what I have to do to make the source code function well,

thanks so much Damiles.

17. hy man..thx u for ur help us :) pls if u know any thing about facial motion capture (steps,links)pls hel me ...send to my e-mail......Great thanks

18. You can use opencv lkdemo.c sample. This is good starting point.

19. Thanks a lot for this post. I am a bit confused why the 3 is added to the index for the array. Obviously it's because 3+2 is r,, 3+1 is g, etc, but how would I have known that? Can you point me to the reference?

20. Hi c1ee, the 3 is no added to the index, its multiplied.

The input image is 3 chanel (r,g,b) and the data is stored in a width x height x numchannels data vector, then for index 0 we get the 0 data vector for blue, 1 data vector for green and 2 for red.
For index 1 of image we get 3 data vector position for blue, 4 for green and ....

The formula is [xposition * num_channels + channel_selector] where channel_selector is 0 for blue, 1 for green and 2 for red.

Regards.

21. Thanks for the post.
Hey, I would like to do the same segmentation for Yellow color, can you please tell me what changes I need to make in the source code to do that.

22. You can use it for yellow searching red and green colors are maxim values.

But the example is more simple and basic, and it's an introduction to this segmentation class, you must use this but using for example hsv instead rgb color model.

Regards.

23. Im working on extracting text from an image can u plz help me out in this respect

25. Is there any built in function in Opencv exist to create image from a weighted matrix.

26. If you have a matrix with data you can represent this as image simply.

Create image with same size of your matrix and with 1 channel to get gray image and set the matris as the image data

27. I have a project that is to segment a text image by projecting horizontal lines to obtain separate there after I segemte each line to have separate characters and then I make the Knowledge
I do just binarization and I could not do the segmentation
it's urgent:((

28. Hi I need ur help.I want to extract text and pattern from an image.I have already done preprocessing steps using Opencv like gray and binary image conversion and contour detection.
Wht should be the next step for Text block detection??plz guide me

29. What is your problem exactly? you can't segment each character from your binary data?

30. The next step for your blck detection is use cvMinAreaRect2 or cvFitEllipse2 to get each block. then you can classificate each block independent.

31. thnx for ur reply .........
but I want to know how to extact each character from a word.
For eg LIFE
I want to read 'L' 'I' 'F' 'E' individually.

plz tell me the algo for this

32. meenu, if you have the contours, you don't need any more, if you want each box you can use cvMinAreaRect2 or cvBoundingRect for each contour you found.

33. Thnx for ur help.
I m eagerly waiting for ur tutorial.

Meanwhile if u can just elaborate on the use of cvMinAreaRect2 .As I understood abt cvMinAreaRect2 use is:
it requires first param as const CvArr* points which will be the image coordinate or point.
My doubt is everytime the image changes we need to change this parameter.
is it possible for you to explain in code the use of cvMinAreaRect2 for each block independent classification.
Thnx a lot for ur instant response.

I really liked ur tutorials.They are really helpful

34. Hi,Your post is great Can you help me to find orange object ?

35. milad get the RGB value of orange color you want select and then modify the tutorial. Regards.

36. To all the over commentors. Blogs can be a lot far better to study should you can keep Your feedback basic and to the level. No-one likes to read giant comments when the concept may be conveyed utilizing a not as long remark

37. Sorry for my english but, what do you want said? I don't undersand...

38. Hi
may i know what the formula mean?
red = [x*3+2], green = [x*3+1], blue = [x*3];
i don't understand why need to add 2 for red, add 1 for green and for blue it do not required add any value..

And the formula of
uchar* temp_ptr=&((uchar*)(img_result_threshold->imageData + img_result_threshold->widthStep*y))[x];

May i know what the formula mean?

39. this formula: uchar* temp_ptr=&((uchar*)(img_result_threshold->imageData + img_result_threshold->widthStep*y))[x]
is for get the pointer of data(pixels) vector.

This vector is organized in BGR format, this is [B1 G1 R1 B2 R2 G2 .... Bn Gn Rn] where have mxnxnumcolors data if image is m width, n height and numcolors.
Then if you want get the x, y pixel the use this formula, but you get the B1, if you want get the other values only need sum 1 or 2 for Green and Red respectly.

Regards

40. Hi . . . i need to help in separating objects based on its color,
say for example if i have objects of red, green and blue color in the input image, i need to separate all these objects. my input image format is RGB. images will be captured through a USB
web-camera.

41. Hi Damiles,
Great job for posting these tutorials and especially answering people - this is really appreciated by all of us.

I want to know your approach on detecting/segmenting & classifying road markings ( speed limit, priority, arrow left,right, etc). How would you do it? What would be a good color space for segmenting this and a good classifying algorithm ?

42. Hi Damiles, the tutorial is great. I want that if I underline in a (real) paper, it should be stored in the electronic version. Can u help how to go about it.

Regards

43. Hi

Thanks so much ! ^ ^

44. Hi, ur code looks great and almost same as want i want now to start with my project work. But i'm not able to get to the 'Demo source' page as it says 'Page not found'. Can you send the source code to me through mail plz.

45. hi
how can i identify each number from a string of numbers
eg: let my string be '1235'
i need to get the result as '1', '2', '3', '5'
Note: i am capturing the string as image through webcam.
so first i take a picture of the string, save as a png file. now i need to do the segmentation...
need help ......

46. Please, visit this tutorial: http://blog.damiles.com/2010/12/segmentation-and-feature-extraction-contours-and-blob-detection/

47. thank you very much...
can u pls help me in implementing the same in python....
i am working on it but there r few errors like ''cvScalar.val[0]' values cannot be retrived', "sizeof(CvContour) gives TypeError: this type has no size, and so on.
pls help me.....

48. http://www.pasteall.org/27434/python

i have pasted the code i wrote in python
can you help me with it.

49. Hi Damiles,
i need to extract text from a complex image ...
text need not be aligned ... it can be scattered...
i need to segment those text characters alone..
how can i do this....(the basic idea)

50. Hello Damiles,
I am using OpenCV 2.1 & VS2008. In that i am able to capture an image but i can't process it in real time. By using cvSaveImage it holds image but some memory problem is showing, how do I carry out it?

51. Dear Sir,

52. what should be the value of threshold, BG_threshold and BG_diff??
thanks :D

53. Hi Damiles,
i have problem about it . It dont detect red color but It has output is black window . Can u help me about that ,plz

54. i have working on color segmentation so plz send me code for color base segmentation using k-Mean Cluster
ii)using Histogram

55. What are the values of threshold, BG_threshold and BG_diff that we put in order to get what u have got above in the image "best case"? thanks a lot for this useful tutorial

56. Hi Celine, the BG threshold and BG_diff are very different depending object to select, and this tutorial is no the best approach for efficient Color segementation, it is only a basic concept.

57. Hy color is very much effected by lightning condition .
even if you convert rgb to hsv and then segment ta color object .
Then problem is still same. hue is change as lightning condition change .
so better for real time application, fix the background lightning condition.
Could any one please tell me how to fix the background lightning condition ?
Thanks

58. Hy even then color is depended or effected by lightning condition.
if you convert rgb to hsv , then hue get change if lightning condition change .
so better to fix the background lightning condition.
Could any one please tell me how to fix the background thanks

60. hi,
very good, but "Demo Source" link is mistake, i require it... :(
plz, correct.

61. why do you use BG_diff?

62. why do you use the BG_diff in segmentation method?

63. You can send Demo source to me?
Email: muakhoc90@gmail.com
Thanks for share

64. Offtopic: why I can see only 51st comment? Where are previous 50 comments?

65. The article is very interesting but the source code link is broken. Could you be so nice and repair it? I will be thankful for giving me possibility to study your example code. Thanks in advance.