Wednesday, 7 March 2018

Dotty election map

Well that escalated quickly...

While I've been working on the forthcoming book and mooc I've been doing some data wrangling in the background at work. For the 2012 Presidential election I made a gallery of maps that illustrated diverse styles of cartography along with some comments on the map types. Each map can tell a different story of the election. I've been in the process of updating this with a new gallery of the 2016 election results (currently around ten maps but more to come) and I got to the tricky one - the dasymetric dot density map. It requires quite a bit of manipulation of data so here is the map, and in this blog I'll explain a little of the process.

In 2012 I made a similar map for the Obama/Romney election. It was a product of the web mapping technology of the time. Made using ArcMap (disclaimer for those who don't know I work for Esri - who make ArcGIS). At the smallest scale 1 dot = 1,000 votes. At the largest, 1 dot = 10 votes and if you printed the map out it would be as large as a football field. It took 3 months to cajole the largest scale map onto the web!!! I wanted to update the map and the four years that have intervened have brought new software capabilities. For 2012 I had to generate up to 12 million points and position them. Now, using ArcGIS Pro I can use the dot density renderer and let the software take the strain and if I were going all out then why not try and make a map where 1 dot = 1 vote. So, for me, the map is a technical challenge. Part of what I do at work to push the software to see what it is capable of, to test it and to show others what capabilities it affords.

So how to make the map? Well, it's a product of a number of decisions, each one of which propagates into the map. I'll be doing a proper write-up on the ArcGIS blog in due course but, in summary, a dasymetric map takes data held at one spatial unit (in this case counties) and reapportions it to different (usually smaller) areas. It uses a technique developed by the late Waldo Tobler called pycnophylactic reallocation modellingThose different areas are, broadly, urban. The point of the map is to show where people live and vote rather than simply painting an entire county with a colour which creates a map that often misleads [Waldo sadly passed away recently and I was running the model when I heard of his death a couple of weeks ago. I met him a few times and his legacy to computational geography and cartography is immense].

I used the National Land Cover Database to extract urban areas. It's a raster dataset at 30m resolution. I used the impervious surface categories and created a polygon dataset with three classes, broadly dense urban, urban, and rural. I then did some data wrangling in ArcGIS Pro (more of that in a different blog) to reapportion the Democrat and Republican total votes at county level into the new polygons. There's some weighting involved so the dense urban polygons get (in total) 50% of the data. The urban get 35% of the data and the rural polygons get 15% of the data. Then I got the dot density renderer in ArcGIS Pro to draw the dots, one for each vote resulting in a map with nearly 130 million dots.

The result is a map that pushes the data into areas where people actually live. It leaves areas where no-one lives devoid of data. It reveals the structure of the US population surface. Most maps that take a dasymetric approach will all end up like this but I think there's value in the approach. To me it presents a better visual comparison of the amount of red and blue that the standard county level map that maps geography, not people, and overemphasises relatively sparsely populated large geographical areas.

So the map I saw on my desktop late Tuesday afternoon took 35 minutes to draw. Technical challenge achieved. ArcGIS Pro nailed it. This is a map that I couldn't have made in the previous election cycle. I was excited and so I took a quick screengrab, sent out a tweet and went home to walk Wisley the dog.

And that, I thought, was that. I'd put the map on the backburner and return to doing layout reviews for the book and doing last-minute work on the mooc over the next couple of weeks. But then something unexpected happened. My phone started pinging. Slowly at first but then a little more during the evening as people began to see the map on Twitter and like or re-tweet it. That's nice, I thought. I went to bed. Wednesday morning I woke to a relative avalanche of likes and retweets. I spent the day in Palm Springs at our Developer Summit and my phone never stopped. By the end of the day it had received around 3,000 likes and had been retweeted 2,000 times. I'm writing this Thursday morning and it's currently at 7,000 likes and a little over 3,000 retweets. The side-effect of this 15 minutes of map fame is I've picked up an extra 1,000 followers (25% increase) on my nearly 10 year old Twitter habit.

But there's a problem. The screengrab was quick and dirty and while there have been many and varied comments on the 'map' it's by no means the finished article. I want to create a hi-res version and also make a web map like the 2012 version. I don't have time to do this in the next couple of weeks but it will happen. But be assured, I am aware of a number of issues. Some have already spotted them and commented.

The symbols - I chose a very default red and blue. Each dot has 90% transparency so overlapping dots at this scale will undoubtedly coalesce into clumps. The impression will appear to bleed across the map. I need to tweak the colours (less saturated) and adjust the transparency to get a better effect. I will also likely do what I did for the 2012 map and classify the data so that at small scales 1 dot = 100 or 1,000 etc. To remove visual 'noise' at those scales. I'll also check for too many overlaps and overprinting. I actually think there's a problem in some areas with blue dots overprinting red. There should be more mixing and more purple. And no, there's no yellow dots. The map only displays Democrat and Republican votes in what remains, effectively, a binary voting outcome.

The data - it's county data, reapportioned. Dot maps convey a positioning that is a function of the processing, not where people actually live or vote. Dots are positioned randomly. Some have, quite reasonably, interpreted the map as showing where votes are and this is a fundamental drawback of the approach. No personal information is in the map at all. I also need to double-check a few areas where people have pointed out apparent anomalies in the map, compared to their personal knowledge of the areas. There may be errors. I need to check. That said, it's a function of the way I've used the NLCD so that data is the basis for reapportionment.

The geography - yes, I hold my hand up. There's no Alaska or Hawaii. I apologise. I'm not sure I'll go back as it requires doing some movement of those states to position them around the lower 48 and put them back in. It's easy but a non-trivial task when you're working in a GIS but I'll think about it. I understand this is unpalatable for some and I accept that criticism.

The interpretations - many have offered some fascinating insights into the gaps and the patterns through Twitter replies. I'll be going through these more carefully when the hullabaloo dies down and teasing out some. But more than anything I've been blown away by the nice things that have been said about the map. It shows the election result in a different way. It tells a different story. One of my favourite responses was this by Thomas de Beus...a lovely mashup and play on the classic photo of Trump's preferred view of the data to hang on the wall of the White House by Trey Yingst.

And this is the point of making a map like this. It presents the SAME data in a different way. It leads to different insights, different interpretations and a different perception. Neither of the above are right or wrong. They are different. Of course, we all have out own view on which serves our needs and which we prefer but that's for us as individuals.

My only regret is that I excitedly tweeted a rough version. I should have waited until I made the map properly. I'll do that but I suspect this is my one viral 15 minutes of fame and I regret it doesn't reflect the quality I know the final version will exhibit. A finished map likely won't get the same traction but we'll see. At the very least it has ignited a discussion. It brings different cartographic eyes to the dataset. Will it ever be hung in the White House? Unlikely.

Thanks for your interest and comments thus far!

Hurriedly written from a hotel in Palm Springs during which time the map's had many more likes, 11 more mentions and I've picked up another 86 followers. I can only apologise to them when they realise I tweet just as much about beer and football as I do about maps.

Sunday, 31 December 2017

Favourite maps from 2017

I've been a little busy writing my book this year so blog posts have been a bit thin on the ground. It was only as I was packing up for a short Christmas vacation that I realised I hadn't even compiled my list of favourite maps from the year. So, with some of the world already embarking on 2018 here's a selection of maps that piqued my interest in no particular order.

Trump's World by Phoebe McLean (age 15)
There probably isn't a better summary of the world in 2017 than this lovely hand-drawn map which won an ICA Children's map competition award

Planet Brewdog by Craig Fisher
It's just a map of Brewdog locations but it's massive and a perfect way to fill up an otherwise plain wall in a modern industrial brewery facility. Here, in the Columbus OH tap room.

Winter Map of Montana by Kevin Nelstead
I particularly like the halos on the white labels :-)

Tactile Atlas of Switzerland by Anna Vetter
Beautifully produced atlas for the blind and partially sighted using raised printing.

Cinemaps by Andrew Degraff
Stunning axonometric maps of over 30 films charting major plot lines and characters as they move through the film's landscape.

The new Swiss World Atlas by the Institute of Cartography and GeoInformation at ETH Zurich
A stunningly beautiful, rich and detailed atlas for secondary school children...actually, for anyone!

Blue and White Dream by anon (China)
Making a map with only a single hue is hard. This is visually stunning, especially set within a large Chinese wall hanging. Apologies, I don't know who created it.

Where the Animals Go by James Cheshire and Oliver Uberti
OK, so the UK version was published in 2016 but the American version was definitely published in 2017 and therefore qualifies. After the success of London: The Information Capital, James and Oliver hit gold again with this award-winning atlas of over 50 great maps plotting animal movement.

San Diego Emoji map by Warren Vick and Europa Technologies
Using gridded emoji is a great idea for a one-off map.

Fifty years of cyclone paths inside the Philippines by David Garcia
Just the lines drawn by the data but revealing the shape of the islands and, with it, the scale of the phenomena.

World Happiness by National Geographic Magazine
I'm a sucker for a multivariate Dorling cartogram Chernoff Face combo.

Lights On and Lights Out by John Nelson
Simple idea mapping the difference between NASA's 2012 and 2017 Earth at Night imagery. Exquisitely rendered.

The best places to see the eclipse by Josh Stevens and NASA
The use of clouds to show, err, the likelihood of cloudcover (and the inverse, clear skies) for the 2017 Solar Eclipse. Simple. Effective.

The Melting of Antarctica by Lauren Tierney
Stunning spread in National Geographic. Great composition and use of angle and projection to tell this story.

U.S. Airport ID requirements by The Washington Post
Neat, interactive use of the gridded cartogram

Dirk's Lego World map by Dirk
Because...maps AND Lego!!!

Hand-stitched London Underground by Tasha Wade
Gifted to me. Unique. One-of-a-kind. Thank you :-)

Trump's Ties by Kenneth Field
It's my list so here's my effort...I still quite like it. Won the Society of Cartographers Wallis Award too :-)

Monday, 11 September 2017

Pointilist cartography

The Washington Post have published an article that explores alternative methods for mapping elections. "Toward a more perfect 2016 presidential election results map" does an excellent job of establishing the problem of mapping totals in massively different geographical units. They don't really explain you have to normalize the totals but, instead, leap to the population-equalizing density cartogram as one alternative before quickly dismissing it as hard to read.

They then offer a map that takes precinct level data and scales the results by number of votes.

What they seem to have done is created a proportional symbol map with very small circular symbols that have been scaled across a ridiculously small size range. They've used a lot of transparency to allow overlapping symbols to build a composite patch of more opaque colour in areas with a lot of small geographical areas.

This is pointilist cartography (note, I said pointilist, not pointless). Proportional symbol maps are not new. Neither are dot density maps. This version isn't particularly innovative but it does do a very good job of mitigating the perceptual problems of widely varying geographical areas. Each place gets the same symbology treatment and, so, the map provides a well balanced mix of red and blue with a lot of white space in between. They used a symbol treatment that goes from red through white to blue with the intermediate colours reserved for marginal precincts. I like this approach. It avoids the unusual purple often used for areas that are finely balanced. It means the map brings focus to those areas that are more partisan. Of course, with a shift in the symbology you could bring focus to marginal areas if that was the map you wanted to show.

A similar approach is to use solid fills for small areas and then show larger areas as small circular symbols. Mixing the techniques on a single map can be useful and also mitigates the visual impact of large areas. Here's an illustration using the technique that I recently made for my forthcoming book. The top is a standard choropleth with a diverging colour scheme. The bottom is the pointilist version.

So, overall I really like this kind of approach to deal with perceptual issues. But the article does hide a more interesting problem. The opening paragraph is at pains to say we've been over this ground before. We have - ad nauseam. Yet so many prefer the standard choropleth and, worse, sometimes with totals. But when they suggest it's a problem for the 'designer' that's where the real problem lies. Everyone these days is a bloody 'designer'. But everything is designed. I always balk when someone tells me they're a designer. A designer of what precisely? Furniture? Buildings? UI? Maps? A cartographer knows how to map election data. They know the problems and they know the solutions that best deal with particular visual issues to get to a map that matches a particular narrative. Far too many 'designers' are busy scrambling to try and figure out how to overcome problems that have already been figured out.

Talk to a cartographer. That's their job. They know what they're doing and likely have a good solution. Pointilist cartography isn't new. I'm pleased to see articles like the one I note here picking up these techniques. I just hope they get used a little more rather than being marginalized by 'designers' who default to the standard choropleth.

Monday, 28 August 2017

Too much rain for a rainbow

National Weather Service today updated its rainbow colour scheme because of the unprecedented deluge caused by Hurricane Harvey in Texas.

Bravo for NWS in modifying its cartographic approach given a change in the phenomena it's mapping. Except they didn't do a very good job.



The previous classification had 13 classes. the new one simply adds two more at the top end to deal with larger rain totals. In fact, all they've done is added detail to the 'greater than 15 inches' class and sub-divided it into three classes '15-20', '20-30' and 'greater than 30'. It'd be pedantic of me to note they still have overlapping classes (they do) but the bigger problem is they retained the same rainbow colour scheme and then added two more colours...a brighter indigo and then a pale pink.

Does that light pink area in the new map above look more to you? Or perhaps a haven of relative stillness and tranquility amongst the utter chaos of the disaster?  Yes, the colours are nested and so we can induce increases and decreases simply through the natural pattern - but the light pink could just as easily be seen as a nested low set of values than the more it is supposed to represent.

For a colour scheme that is trying to convey magnitude...more rain...more more more, you need a scheme that people perceive as more, more, more too. Different hues do not, perceptually, do that. Light pink does not suggest hideous amounts of rain compared to the dark purples it is supposed to extend.

We see light as less and dark as more. Going through a rainbow scheme where lightness changes throughout (the mid light yellow at '1.5-2.0' inches is a particular problem) isn't an effective method. Simply adding colours to the end of an already poor colour scheme and then making the class representing the largest magnitude the very lightest colour is weak symbology. But then , they've already used all the colours of the rainbow so they're out of options!

The very least they should have done is re-calibrated the classes to make the largest class encompass the new, out-of-all-known-range range. You can't simply add more classes when you're already maxed-out of options for effective symbolisation.

Better still, look around and learn how it should be done. The Washington Post has made a terrific map using a colour scheme that does have a subtle hue shift but whose main perceptual feature is the shift in lightness values. So we see more, more more as the colour scheme gets darker. It's simple. it really is.

The scientific community continues to use poor colour schemes and poor cartography to communicate to the general public. At least the mainstream media is doing a much better job.

[Update 29.8.2017 to include the New York Times piece]

New York Times today published one of the best maps I have seen in a long while. I mean 'best maps' of anything, not just the continuing deluge in Texas. Its simplicity belies its complexity and that's the trick with good cartography. Here's a pretty lo-res grab but go to the site and take a look.

They've got the colours spot on, A slight hue shift to emphasize light to dark but cleverly hooking into the way in which we 'see' deeper water as darker blue. Of course, it isn't really deeper blue but the way light is reflected, refracted and absorbed by water gives us that illusion. So, it acts as a visual anchor that we can relate to.

There's other symbology too - small gridded proportional circles that show the heaviest rainfall in each hour. The map is an animation so this gives a terrific sense of the pulsing nature of the movement of successive waves of rain (literally, waves!). The colours morph towards the higher end as the animation plays to build a cumulative total. This also has the effect of countering the natural change blindness we see when we're trying to recall the proportional symbols.

The two symbols work in harmony. And then, for those who want detail a hover gets you a graph showing the per hour total over the last few days.

These aren't the only maps in the NYT piece. The article is full of them. Each one carefully designed to explore a specific aspect of the disaster: the history of storms, reports, evacuations etc.

It's maps like those from The Washington Post and New York Times that prove that good cartography does exist and it matters. We really don't deserve the sort of maps that NWS pumps out. They're just really awful to look at, fail on a cognitive level and prove they haven't the first clue about how to effectively communicate their own science and data.

The irony is that the NYT map uses the NWS data of the rainfall data to make their own version and prove that it's perfectly possible to make terrific maps that communicate and which once again give us more reasons to #endtherainbow. Well played.