Next Steps

I'm going to be working on this project again in the near future.  The current plans are to adapt the algorithm for video data.  This will probably involve considering motion and various other things, and maybe it will even get optimized or something fancier than that.

The goal will be to implement this on a GPU (in CUDA and/or OpenCL) so that it can handle high definition video streams in as close to real time as possible.

Source Code

Source code is located here.

The source code is entirely unoptimized and offered with absolutely no support.  In general we will not respond to email about getting it to work or bugs encountered.  It is still quite useful as a reference source for understanding the algorithm or for working with OpenCV.

If you use or reference our work, we'd be interested in hearing about it!

Final Paper

Here it is!

I'll try to upload our code soon as well.

Final Presentation

Click here for the powerpoint version of our final presentation.

Or click here to get it as a PDF.

Testing and Evaluation

We will be evaluating our model using the following procedure:

Below are 24 images with at least one salient object, selected without being tested previously with any of our saliency algorithms. We ran three algorithms (Itti, Itti with LAB, Itti with LAB/Symmetry), and have randomly placed the resulting masked images next to each image.

To help us evaluate the algorithms, we would like it if you could subjectively order the three masked images associated with each full image from best to worst.  What defines best and what defines worst is entirely subjective.  In general we are looking for the mask that shows as much of what you thought was important when looking at the image as possible.

The lettering next to each image is entirely random.  Please evaluate as many images as you can or have patience for.

Please format your responses as follows:

Image 01 - BCA <- B is the best, A is the worst
Image 02 - (CA)B <- This would indicate no discernable difference between C and A and you judging them equally good. 
Image XX - etc

You can complete the survey by posting a response to this post (anonymously if you desire), sending either of us an email, or even giving us something in person.

Click on the images to enlarge them:

Normalization working Normally

So after some diligent staring at our code, we discovered the issues with our DoG filter code and got that fixed up.  The mistakes were pretty good: first I had mixed up the height/width indexes, then we noticed that we weren't decrementing in a loop, and finally we had an off by one error due to using > instead of >=.  Programming is fun.

So on that note, our images look really nice now.  We can toggle between DoG and maximal suppression normalization and compare the results.  DoG filtering removes a lot of noise compared to the other method.  We perform a final round of normalization at the end of processing on the aggregate saliency map and opted to not use DoG there because it was removing a bit too much from the image.

Our normalization now looks like: DoG on all subroutine steps, then one maximal suppresion normalization at the end to highlight the salient areas.  This also allows you to see any noise in the final map, which DoG would near completely eliminate (along with potentially interesting features).  We think this is a good compromise and will be sticking with it.

Normalization is an area that a lot could be done with - it drastically affects the results of our algorithm.

Unfortunately we still haven't completed the symmetry feature map code yet.  We're working on it though!  It really is a lot of work to translate into OpenCV but it's making our code a lot better because of various improvements we are making along the way.  Symmetry is the last big "to do" item on our list, as well as a simple survey before our final report.  We should have symmetry done this weekend, after which we can perform our survey.

Our current ideas about how to present the maps in order for people to judge them is simply threshold the maps then filter images based upon these thresholded maps.  Anything not black will be seen.  The survey will show the original image and then three variants of it: RGB saliency map, LAB saliency map, and LAB saliency map with symmetry.

Here are some pictures of the new filter in action (ordering is input, LAB, RGB) (no symmetry):

Slow and Steady

We've been working away on implementing the bilateral symmetry filter from Kovesi. He has Matlab code available to perform the operation which we have been translating into OpenCV/C++ code. This process takes a while since even the simplest lines in Matlab can take ten times as much code to write in C++. This is approximately half way done at this point.

In addition to this code translation, we have been doing more research on symmetry and have found several other papers that deal with the subject in relation to visual attention. Two of the papers we are reading explicitly mention Itti and Koch and contrast their symmetry methods with Itti et al.

I fixed some bugs in our normalization code so the final maps that resulted from doing operations using integers instead of floats, so our results look like normal saliency maps now. The DoG filter code we implemented based upon Walther's saliency toolbox still doesn't give us the kind of results we would like - often the images will come out black after being fed through the filter or will only have one very intense region of marginal importance. So there are still some bugs to work out with that filter, but otherwise things are moving smoothly and our results look good.

Our plans for wrapping up the project include fully implementing this bilateral symmetry filter and fixing any normalization quirks in the code. We then aim to perform a small experiment asking participants to rate our saliency maps against other methods.

And now, pretty pictures (Original, LAB, RGB):

Papers:

Biologically Inspired Saliency Map Model for Bottom-up
Visual Attention : http://www.springerlink.com/content/7wxq0npr7b09hlj3/fulltext.pdf

Paying Attention to Symmetry : http://www.ai.rug.nl/~gert/download/kootstra08bmvc.pdf