Jan 172017
 

By Steve Wexler

January 17, 2017

This is a follow-up to the post Jeff Shaffer and I wrote about what can happen when people fail to question sources and inadvertently amplify baseless findings.

Overview

There’s been great feedback on things the community can do to maintain all the good things about Makeover Monday (MM) and at the same time reduce the occurrence of bad things.  Before I go any further I want to reiterate that I both like and value MM.  I’ve seen some incredibly good dashboards that inform my consulting practice and I plan to publish a blog on some of the design and analytic masterpieces I’ve seen.

But first…

Guilty as charged

Before presenting some recommendations, I want to cite an e-mail I received from Ben Jones pointing out some hypocrisy on my part. Ben writes:

“…did you know that the Axalta coating systems Global Automotive color popularity report that you used to make this viz doesn’t even mention the sample sizes, sampling plan, or methods of their surveys anywhere in the report?”

Ben is completely correct, and I immediately followed my own recommendations for the 100+ errant visualizations from week 1 of Makeover Monday 2017 and added this disclaimer to my dashboard:

Figure 1 -- The disclaimer / warning that now appears on my dashboard.

Figure 1 — The disclaimer / warning that now appears on my dashboard.

So, why did I get in such a huff about folks not vetting their sources for the Australian pay gap but didn’t bother to vet the data for my makeover example?

It comes down to three things:

  1. The number of people participating
  2. The magnitude of the mistake
  3. The importance of the data

For the Australian pay gap makeover there were

ONE — Over 100 people making

TWO — Big mistakes with

Three — “High-stakes” data

Recommendations

Avoid “high-stakes” data sets. Some folks will make the argument that iPhone sales could be considered “high stakes” as someone might make a stock buying or selling decision based on this. Fair enough, but for MM I suggest Andy and Eva avoid data that deals with gender, race, guns, shootings, and anything else that is politically charged.

Get ideas from reputable data sources. Realize that source data from the Australian government is, in fact, good data. The problem is that it isn’t good for making meaningful gender comparisons. So, where did the assertion of giant wage gaps come from? The MM folks cited an article from Women’s Agenda.  I spoke with data journalist Chad Skelton and he suggested that for high-stakes subjects MM should stick with ideas from the likes of The Guardian, The Wall Street Journal, and other well-known entities where reporters confer with statisticians and economists before publishing conclusions.

Note: Jeff Shaffer did some more digging and it appears that Women’s Agenda had in fact republished findings from Business Insider Australia. I have sent e-mails to editors at both publications.

Place a prominent Makeover Monday logo and disclaimer on every dashboard. Since Makeover Monday is an exercise in data visualization redesign, why not have participants trumpet MM’s purpose on every dashboard?  By having a small but prominent logo along with a disclaimer we can both raise awareness of the project and caution people that the purpose of the dashboard is to practice data visualization techniques. My only concern is that people will think this gives them a free pass to present a bogus headline and/or specious findings.

While I think the logo below is spoken for, maybe something like it would help viewers know that for the dashboard in question, it’s not about the data, it’s about the design.

Figure 2 -- Possible logo for Makeover Monday?

Figure 2 — Possible logo for Makeover Monday?

Vetting the data — the birth of “Find the Flaw Friday”?

In the previous post, I indicated that the one hour MM recommends people spend on a makeover is rarely enough time to properly vet the data, let along craft a visualization.  And asking Eva and Andy to vet all the data is completely impractical as I’m sure it’s hard enough to curate the project as it stands now.

Many of the people I corresponded with have said that exploring and really understanding the data is as important, if not more important, than designing a cool dashboard (this gets us back to the substance over style discussion.) A lot of people indicated they find exploring and vetting data more interesting than fashioning visualizations.

So, how can we help people practice analysis and not just design and presentation?

At least five people indicated they would be up for what I will call “Find the Flaw Friday” (FFF) where people are tasked with exploring a data set and determining what analysis would be sound and what would likely to be flawed. I’m not sure how easy this will be to manage and how much time people will need to spend on each project, but I will put the folks that expressed an interest in touch with each other and we’ll see what materializes.

If this works perhaps the data sets from FFF could feed MM?

Final thought — Please fix (or take down) your erroneous dashboards

I consider wage inequality a big deal, and if you are going publish anything that’s about a big deal, you need to get it right.

So, now that people know that their week 1 MM dashboards are wrong, what should they do?

FIX THEM!

In conferring with Chad, he wonders how people would deal with something like this in their jobs.  For example, suppose you create a beautiful visualization for the CEO and realize later that the central point of the viz is deeply flawed?  Even if the mistake weren’t your fault would you rush to correct the record to keep your reputation intact?

Everyone — and I would argue that this should come from the Andy and Eva as well as the folks that manage Tableau Public — should tweet about the errors to make sure people don’t continue to perpetuate the misinformation.

It’s one thing to get something wrong. It’s another thing to know something is wrong and not fix it.

So please, fix it.

Postscript

Tableau Zen Master Brit Cava recommends a fascinating Freakonomics podcast with economist and gender gap expert Claudia Goldin.  See http://freakonomics.com/podcast/the-true-story-of-the-gender-pay-gap-a-new-freakonomics-radio-podcast/.  VERY worthwhile.

 

 Posted by on January 17, 2017 1) General Discussions, Blog Tagged with: ,  No Responses »
Sep 182014
 

Overview

A dashboard from Radio Free Europe / Radio Liberty has received a lot of views since it was published earlier this week and for good reason: There’s a lot of important information packed into a compelling story.

There’s a lot I like about the dashboard but two things that I believe desperately need to be corrected.

Before going any further you can see the dashboard here.

What I Like and Don’t Like

Here’s a re-sized snapshot of the dashboard.

01_Fighters

What I Like:

  • Colors
  • Hovering over a country provides more information about the country.
  • Syria and Iraq are labelled so I can find the focus of the story quickly.
  • The author has presented normalized data in the bottom chart  to show proportion of fighters from a particular country. This is brilliant and important.

What I Do Not Like:

  • There are two different axes at the bottom of each bar chart with very different values. If I don’t look at the axes I would think that Belgium has the same number of fighters per million as Tunisia. This defeats the brilliance of presenting the data in a normalized form.
  • The fighter icon takes away from the gravity of the story as this should not be a frivolous visualization. The visualization should skew more towards The Economist and less towards USA Today (see this post.)
  • The bars are in alphabetical order.

The Makeover

Here’s what I’ve changed.

  • Clicking a country name in one of the visualizations will highlight that country in the other visualization making it easy to find.
  • The axes for the two sets of bar charts at the bottom are consistent.
  • I’ve replaced the fighter icons with labeled bars.
  • You can switch between displaying normalized data (fighters per million) and the number of fighters.
  • The bars are color coded to reflect the same color legend in the top chart.

Click here or the image below to access the interactive dashboard.

02_Makeover