Testing and Evaluation

We will be evaluating our model using the following procedure:

Below are 24 images with at least one salient object, selected without being tested previously with any of our saliency algorithms. We ran three algorithms (Itti, Itti with LAB, Itti with LAB/Symmetry), and have randomly placed the resulting masked images next to each image.

To help us evaluate the algorithms, we would like it if you could subjectively order the three masked images associated with each full image from best to worst.  What defines best and what defines worst is entirely subjective.  In general we are looking for the mask that shows as much of what you thought was important when looking at the image as possible.

The lettering next to each image is entirely random.  Please evaluate as many images as you can or have patience for.

Please format your responses as follows:

Image 01 - BCA <- B is the best, A is the worst
Image 02 - (CA)B <- This would indicate no discernable difference between C and A and you judging them equally good. 
Image XX - etc

You can complete the survey by posting a response to this post (anonymously if you desire), sending either of us an email, or even giving us something in person.

Click on the images to enlarge them: