Salient Green

Next Steps

2009-07-24T14:12:00.000-07:00

I'm going to be working on this project again in the near future. The current plans are to adapt the algorithm for video data. This will probably involve considering motion and various other things, and maybe it will even get optimized or something fancier than that.

The goal will be to implement this on a GPU (in CUDA and/or OpenCL) so that it can handle high definition video streams in as close to real time as possible.

Source Code

2009-04-13T11:38:00.000-07:00

Source code is located here.

The source code is entirely unoptimized and offered with absolutely no support. In general we will not respond to email about getting it to work or bugs encountered. It is still quite useful as a reference source for understanding the algorithm or for working with OpenCV.

If you use or reference our work, we'd be interested in hearing about it!

Final Paper

2009-03-18T11:21:00.000-07:00

Here it is!

I'll try to upload our code soon as well.

Final Presentation

2009-03-12T22:34:00.000-07:00

Click here for the powerpoint version of our final presentation.

Or click here to get it as a PDF.

Testing and Evaluation

2009-03-08T23:39:00.001-07:00

We will be evaluating our model using the following procedure:

Below are 24 images with at least one salient object, selected without being tested previously with any of our saliency algorithms. We ran three algorithms (Itti, Itti with LAB, Itti with LAB/Symmetry), and have randomly placed the resulting masked images next to each image.

To help us evaluate the algorithms, we would like it if you could subjectively order the three masked images associated with each full image from best to worst. What defines best and what defines worst is entirely subjective. In general we are looking for the mask that shows as much of what you thought was important when looking at the image as possible.

The lettering next to each image is entirely random. Please evaluate as many images as you can or have patience for.

Please format your responses as follows:

Image 01 - BCA <- B is the best, A is the worst
Image 02 - (CA)B <- This would indicate no discernable difference between C and A and you judging them equally good.
Image XX - etc

You can complete the survey by posting a response to this post (anonymously if you desire), sending either of us an email, or even giving us something in person.

Click on the images to enlarge them:

Normalization working Normally

2009-03-02T15:43:00.001-08:00

So after some diligent staring at our code, we discovered the issues with our DoG filter code and got that fixed up. The mistakes were pretty good: first I had mixed up the height/width indexes, then we noticed that we weren't decrementing in a loop, and finally we had an off by one error due to using > instead of >=. Programming is fun.

So on that note, our images look really nice now. We can toggle between DoG and maximal suppression normalization and compare the results. DoG filtering removes a lot of noise compared to the other method. We perform a final round of normalization at the end of processing on the aggregate saliency map and opted to not use DoG there because it was removing a bit too much from the image.

Our normalization now looks like: DoG on all subroutine steps, then one maximal suppresion normalization at the end to highlight the salient areas. This also allows you to see any noise in the final map, which DoG would near completely eliminate (along with potentially interesting features). We think this is a good compromise and will be sticking with it.

Normalization is an area that a lot could be done with - it drastically affects the results of our algorithm.

Unfortunately we still haven't completed the symmetry feature map code yet. We're working on it though! It really is a lot of work to translate into OpenCV but it's making our code a lot better because of various improvements we are making along the way. Symmetry is the last big "to do" item on our list, as well as a simple survey before our final report. We should have symmetry done this weekend, after which we can perform our survey.

Our current ideas about how to present the maps in order for people to judge them is simply threshold the maps then filter images based upon these thresholded maps. Anything not black will be seen. The survey will show the original image and then three variants of it: RGB saliency map, LAB saliency map, and LAB saliency map with symmetry.

Here are some pictures of the new filter in action (ordering is input, LAB, RGB) (no symmetry):

Slow and Steady

2009-02-25T13:41:00.000-08:00

We've been working away on implementing the bilateral symmetry filter from Kovesi. He has Matlab code available to perform the operation which we have been translating into OpenCV/C++ code. This process takes a while since even the simplest lines in Matlab can take ten times as much code to write in C++. This is approximately half way done at this point.

In addition to this code translation, we have been doing more research on symmetry and have found several other papers that deal with the subject in relation to visual attention. Two of the papers we are reading explicitly mention Itti and Koch and contrast their symmetry methods with Itti et al.

I fixed some bugs in our normalization code so the final maps that resulted from doing operations using integers instead of floats, so our results look like normal saliency maps now. The DoG filter code we implemented based upon Walther's saliency toolbox still doesn't give us the kind of results we would like - often the images will come out black after being fed through the filter or will only have one very intense region of marginal importance. So there are still some bugs to work out with that filter, but otherwise things are moving smoothly and our results look good.

Our plans for wrapping up the project include fully implementing this bilateral symmetry filter and fixing any normalization quirks in the code. We then aim to perform a small experiment asking participants to rate our saliency maps against other methods.

And now, pretty pictures (Original, LAB, RGB):

Papers:

Biologically Inspired Saliency Map Model for Bottom-up
Visual Attention : http://www.springerlink.com/content/7wxq0npr7b09hlj3/fulltext.pdf

Paying Attention to Symmetry : http://www.ai.rug.nl/~gert/download/kootstra08bmvc.pdf

Tis the Time for an Update

2009-02-20T10:03:00.000-08:00

We have been working on the various things we mentioned last time. Some progress has been made with the normalization technique described last time, along with the floating point implementation. Normalization no longer just blurs the image, but it does not yet work consistently (the image is either normalized properly, or blacked out entirely). This seems to be due to the parameters to our D0G, which we have yet to find any that work for all images.

Also, as mentioned last time, we are working on a symmetry map. It has been implemented in MATLAB, though it works very slowly. One example:

Our symmetry map is based off the work by Peter Kovesi, Symmetry and Asymmetry From Local Phase (ps)

Most Recent Update Ever Up Until Another Update

2009-02-10T22:14:00.000-08:00

As mentioned in the previous post, images can seemingly shift towards the bottom right corner when using the standard gaussian kernel in OpenCV in its pyramid function. To get around this, we wrote our own code to do the pyramids and now this issue is far less noticeable. This is a side effect from using the pyramids - they are much faster than applying a wide bank of filters but come with this small downside.

We are experimenting with different ways to weight the various maps we combine for our final saliency map. We tried one method which waited them according to their mean intensity values, but found the results to be very inconsistent.

We're also producing both an RGB based and Lab based set of maps now so we can visually compare them. In most cases Lab seems to outperform the RGB based maps - sometimes to a great degree:

Original Image:

Final Lab Map:

Final RGB Map:

I've also been playing around with how we are representing images. Currently we are using 8 bits per channel so pixel values are integers from 0 to 255. I tried an implementation where all image data was stored as floating point values, 0 to 1, and the results were slightly different. This is something we need to investigate more as we continue.

The normalization technique we are currently using involves multiplying the image by (max - mean(local_max))^2, which has the issue of non-fixed range, which is why our final maps have large regions of similar intensities instead of a more gradual build up. A newer technique proposed by Itti et al. involves iterating a Difference of Guassians filter over the image to perform normalization. We have experimented with this but are having issues getting it to work as expected - our images come out very blurred.

Our immediate plans are to implement a low level symmetry filter and use that in our final saliency map. In addition we are going to refine our current methods as stated above and see if any improvement is detected.

Moving to OpenCV

2009-02-04T12:17:00.001-08:00

While Kevin was working on an initial Matlab implementation I began working on an OpenCV implementation to see which would perform better. OpenCV tends to run faster than even the optimized Matlab code, so we are most likely going to stick to a final implementation written in C++ using OpenCV + other stuff.

OpenCV isn't without its annoying quirks though. Often times when creating a supposedly all black new image, many pixels in the image will be filled with random values, resulting in a noisy image. This is corrected by multiplying the image with a zero filled matrix, but the extra step adds time and we haven't figured out a way around this yet.

Right now the saliency maps we are generating are essentially the same as those used by Itti, Koch, and Niebur. We deviate slightly in regard to our color implementation. Instead of using an RGB image to extract intensity, RG, and BY values, we convert use the L*a*b* color model. This model stores the image in thee channels: intensity (lightness), RG (red versus green), and BY (blue versus yellow). The end result is the same - we compute an intensity, red versus green, and blue versus yellow map of the image.

We also use a bank of eight Gabor filters on a grayscale version of our image. These filters are of varying rotations and scales and respond strongly whenever their orientation matches that found in the image.

This is the general algorithm we are following right now:

Split the image into L*, a*, b*, as well as grayscale channels
Compute center surround differences on the L*, a*, and b* images. This basically means that we feed the image through a guassian pyramid and compute differences between scales, normalizing at each step.
For each gabor filter in the bank, compute the center surround differences and normalize as in the color maps. The final gabor map is the mean of every image in the bank.
Take the four maps and compute a final map, weighting luminosity, color, and orientation equally.

We are still experimenting with different weights for the final map. Right now each color map receives half the weight of the luminosity map.

I've also been experimenting with thresholding throughout the process and have found that removing all pixels less than 1/10th the (max - min) in the L*a*b* process results in a cleaner end map.

And now some pretty pictures:

One issue you may notice from these images is that our maps seem to drift towards the bottom right of the image. This is an issue brought up by Dirk Walther that results from decimating an image after a gaussian kernel has been applied. It can be mostly corrected by convolving the image again with a simple kernel such as [1 1]/2. This image shows the effect of the second convolution (from Walther):

Papers:

Walther, Interactions of Visual Attention and Object Recognition: Computational Modeling, Algorithms, and Psychophysics, 2006

Orientation and the Gabor Filter

2009-02-04T12:13:00.001-08:00

To examine orientation as saliency channel, we adopted the model proposed by Itti et al, wherein a Gabor Filter is applied to the image. The Gabor Filter uses a 2D Gabor function (a sinusoid multiplied by a Gaussian kernel) as its impulse response. This can roughly be visualized as:
1D
2D

Note the size and ellipticity of the Gabor function, and standard deviation of the Gaussian are variable.

The filter then applies this function to the image, emphasizing edges along the rotation of the Gaussian envelope.

Below are several examples of applying different orientations of the Gabor function to an image.

pi/4
pi/2
3pi/4
pi

Note a wavelength of 10 and a circular (ellipticity = 1) are used with a StD of ~5.6.

Each orientation map is created in the usual way, by taking differences of several scales of the filtered image.

J Movellan, Tutorial on Gabor Filters
OpenCV implementation: http://www.personal.reading.ac.uk/~sir02mz/CGabor/example.html

Color Spaces

2009-02-04T11:41:00.000-08:00

Color spaces are abstract models of describing colors mathematically. Typically they consist of three or four tuples known as color components that map a color to the color space.

One of the easiest color models to understand is RGB. RGB is a three tuple consisting of the red, green, and blue content in a pixel. In this system, 0 is considered black while the maximum value is considered white (typically 255 with 24 bits per pixel). This can be pictured conceptually as a cube where one corner is pure black and the opposite corner is pure white. The line connecting the two corners is grayscale, and everything else represents some visible color in the space.

There are several other color models, such as CMYK, HSV, and YUV. The color model that interests us in our project is L*a*b*.

The CIE L*a*b*, unlike RGB or CMYK, does not use a subjective set of primary colors, which makes it device independent. It describes colors along three axes in 3D space - the L* axis corresponds to the lightness of a color - a value of 0 is pure black, while a value of 100 is pure white.

The other two axes, a* and b*, theoretically have no upper or lower bounds. Negative a* corresponds to a green color, positive to a red color. Negative b* to a blue color, positive to a yellow color. In reality these are often bounded between -128 and 127, depending on the number of bits used to represent the color.

The L*a*b* color model was designed to approximate human vision. The L component closely matches with human perception of lightness. Differences between colors can be determined in terms of distance between them - which is sometimes defined in terms of the minimum aount of color variation between two shades discernible to the human eye.

Sources:

http://www.hunterlab.com/appnotes/an07_96a.pdf
http://www.kleerdex.com/briefs/177.pdf
http://en.wikipedia.org/wiki/Lab_color_space

Progress Update

2009-01-28T13:17:00.000-08:00

The MATLAB algorithm is up and running, albeit without normalization or thresholding. The algorithm currently creates sub maps for several fields:

Intensity
|Red - Green|
|Blue - Yellow|
Orientation: pi/4
Orientation: pi/2
Orientation: 3pi/4
Orientation: pi
Final Saliency Map

Some examples of other early results:

The Pyramids of Gauss

2009-01-21T12:56:00.000-08:00

Once a saliency map is generated, a pyramid technique is applied to the image to create a sort of topological map that allows the most salient regions to be more easily identified. The general process behind the technique is to iteratively smooth and decimate the image, creating subsequant images which are half the size.

In many cases the smoothing process is carried out by a guassian filter, which preserves edges better than a similarily sized mean filter (take the k nearest neighbors to a pixel and evenly weight them). The guassian kernel weights the pixels according to a guassian distribution, which puts emphasis on nearby neighbors over distant ones (the degree of emphasis is controlled by the standard deviation of the kernel).

Since the gaussian is a symmetric distribution, the smoothing operation can be performed in the x and y directions seperately using a convolution.

Example of Intensity Pyramid Scales:

Papers:

Walther, Interactions 0f the Visual Attention and Ojection Recognition: Computational Modeling, Algorithms, and Psychophysics, 2006.

Itti, Koch, Niebur, A Model of Saliency-based Visual Attention for Rapid Scene Analysis, 1998.

Saliency Maps

2009-01-11T22:29:00.000-08:00

My current readingy deals with the creation of saliency maps, topological maps that combine multiscale image features that correspond to attentive selection.

A brief introduction to mapping is provided by Dr. Ernst Niebur here.

For the bottom-up approach, Cristof Koch and Shimon Ullman proposed that different visual features contribute to the stimulus, such as color, intensity, orientation, and movement. The saliency map integrates this information into one global measure, essentially a topographical map. Then the most 'salient' features of the image correspond to the global maximum of this topographical map. (From my reading, the most popular method to accomplish this is a variation of the Gradient Ascent Optimization method, which follows the positive gradient to a local maximum). In the bottom approach, these regions of greatest salience are then considered in sequential order, whereas top-down would override the most salient regions in favor of more relevant areas.

Examples of a saliency map:

The above process: First image is the original picture, the three images correspond to Color, Intensity, and Orientation respectively, and then a final saliency map. Note in practice, a saliency map is often composed of even more feature maps. The final several pictures provide the regions where the focus of attention (FOA) is directed, along with the amount of simulated time to 'notice' these regions (essentially how long it would take for the imaes to pop out to the visual cortex).

A brief paper that outlines a method of creating and using Saliency Maps: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis. Itti, Koch, and Niebur

More on Saliency

2009-01-07T11:18:00.000-08:00

Visual saliency provides a relatively efficient way to quickly eliminate items of disinterest from the field of view. This is largely due to the concepts behind the "bottom-up" approach mentioned in our introductory blog post.

The key objective of visual saliency is to quickly identify objects of interest. When observing a picture such as the one below:

the object that stands out is clearly the red bar amongst the green. This of course depends on the capabilities of the observer to distinguish colors and intensities. Consider that without the capability to distinguish red from green, there would be no obvious item of interest in the image.

In the next image, it is difficult to spot anything of interest without searching through the image. This is because there is little salience to guide you to the unique bar (can you find it?). The orientation and color of most of the objects in the image are very similar and as such nothing seems too important.

Looking at the same image again, it is interesting to note what happens when the size of the viewable area is decreased:

Can you spot the unique bar now? While it still may not be immediately obvious as in the first image, looking at a subset of the image can make determining which objects are significant easier. This approach can be taken further:

Now it should be clear that the vertical red bar is our object of interest. Consider what happens if the view is further restricted:

In the above image, it now seems as if the horizontal red bar is our object of interest because all other bars are vertical. Taken one final step:

Now we cannot tell which of the two bars is significant because they are the only two items in the field of view. This presents some interesting facts about saliency:

Saliency is not an inherent attribute of an object; saliency depends upon the combined effects of many stimuli to make a given stimulus more interesting.

Changing the scale of search can improve our ability to find more salient locations in an image though it may cause objects which are of no interest on the broad scale to suddenly seem interesting.

The Introduction

2009-01-05T12:29:00.000-08:00

Our CSE 190 project will deal with the topic of saliency, which is the state or quality of standing out relative to neighboring items. We intend to focus on the bottom-up approach, which is based upon stimulus-driven signals that announce that a location is sufficiently different from its surroundings to be worthy of attention.

In contrast the top-down approach focuses on object recognition using discriminant analysis.

Saliency is a plausible model for how biological systems and is considered to be a key attentional mechanism that facilitates learning and survival by enabling organisms to focus their limited perceptual and cognitive resources on the most pertinent subset of the available sensory data.

------------------------------------------

As of now our plan is to spend the next two weeks researching and learning more about saliency and the relevent algorithms associated with the bottom-up approach. The goal of our project is to create a bottom-up saliency detector and apply it specifically to crowds, though the detector should be general enough for use in other applications.

------------------------------------------

Video of bottom-up saliency mapping on a busy freeway: