Visualizing the Data Behind Your Images

I think it's an interesting exercise to visualize data contained in a photograph as we might other datasets. Some cameras and photo-editing software do just this, constructing histograms from the red (R), green (G) and blue (B) values (the three "primaries") in the image to help expert photographers/editors judge and improve color balance. To illustrate this idea, I'll use the color photograph and its grayscale counterpart below.

It's easiest to start with the grayscale image where the R, G and B values are, by definition, identical for any given pixel. On the left we have a grayscale value of 0, meaning an (R, G, B) value of (0, 0, 0), a.k.a. black; on the right it is (255, 255, 255) or white. In between we have 254 shades of gray.

This is one of the few cases where use of a gradient in a chart may actually help clarify things, in this case by coloring each bar according to the grayscale value it represents. The only problem with this is then picking a background color that is neutral:

With a color image we need to plot each of the R, G, and B histograms separately. It's less cluttered to use lines rather than bars, but with a sensible color scheme they probably don't need labeling.

Aside from achieving a good color balance, I think there is another issue that is interesting to explore that is important for web development: compression. JPEG compression is a fairly complex topic and I'll confess I've yet to really get my head around it. The color photo above is a high-quality (low-compression) miniature of the original (eight megapixel) image I shot. It is 46 kilobytes in size. The image below is the same in terms of number of pixels but is more highly compressed and of lower quality. It is, however, only nine kilobytes in size.

The lower quality should be obvious in the photo itself, but does the RGB histogram look significantly different? Before trying this out I genuinely had no idea what to expect. While the histograms below are clearly different to the ones above they don't really scream "this image has been compressed much more than the other one".

An alternative to the JPEG is the PNG. Here's an 8-bit PNG version of the photograph:

This image looks a lot better than the low-quality JPEG, but it's also six times bigger than it (in terms of file size) and a third bigger than the high-quality JPEG. The histogram is also very very different and difficult to read (note also the difference in the vertical scale).

This chaotic histogram is a result of the fact that only 256 different colors can be used in an 8-bit PNG (just like a GIF).

The histogram data above can be extracted from (at least some versions of) Photoshop and presumably other photo editing software. But having not found any major difference between the histograms for the high- and low-quality JPEGs I was curious to see whether more complex representations of the RGB data would highlight the difference. This required extracting all the individual RGB values, not just sums for the different channels. I didn't know how to do this in Photoshop so I used R (other options are available). In an ideal world a 3D scatter plot would work brilliantly - one dimension for each of R, G and B. But 3D scatter plots on 2D screens rarely work. So I opted for a more conventional 2D scatter plot, using point color to show the third color. In the examples below I (somewhat arbitrarily) opted to plot blue value against red value, and colored the points according to the green value. For instance, a data point representing an image pixel with an RGB value of (30, 96, 92) would be plotted at the point (30, 92) on the chart and have an RGB color of (0, 96, 0).

My original attempt to do this suffered from some serious overplotting issues. To reduce, though not remove, this issue I made the points much smaller and took a random sample of "just" 20,000 points (just under 20% of the data) for each image. I also added the marginal distributions (i.e. the relevant histograms) at the unlabeled extremities of the plot. These were derived from all the image pixels, not just the sample.

Now we can see a difference between the high- and low-quality JPEGs: the latter has much more pronounced diagonal bands of points and gaps. Examining the data, there are around 20,000 different RGB values in the ~105,000 pixels of the high-quality JPEG but only 15,000 in the low-quality JPEG. As for the PNG plot, that is an extreme example of overplotting. There really was 20,000 points plotted, they just occupy only a few hundred different positions.

Ultimately you can probably get through life as, say, a web designer or developer without understanding the intricacies of image compression - I'm still not all that sure how JPEG compression actually works. But I think it's interesting to know the underlying data is there to be played with and for constructing your own dataviz experiments.

Try our jQuery HTML5 controls for your web apps and take immediate advantage of stunning data visualization capabilities. Download Free Trial now!