By Steve Wexler
January 17, 2017
This is a follow-up to the post Jeff Shaffer and I wrote about what can happen when people fail to question sources and inadvertently amplify baseless findings.
Overview
There’s been great feedback on things the community can do to maintain all the good things about Makeover Monday (MM) and at the same time reduce the occurrence of bad things. Before I go any further I want to reiterate that I both like and value MM. I’ve seen some incredibly good dashboards that inform my consulting practice and I plan to publish a blog on some of the design and analytic masterpieces I’ve seen.
But first…
Guilty as charged
Before presenting some recommendations, I want to cite an e-mail I received from Ben Jones pointing out some hypocrisy on my part. Ben writes:
“…did you know that the Axalta coating systems Global Automotive color popularity report that you used to make this viz doesn’t even mention the sample sizes, sampling plan, or methods of their surveys anywhere in the report?”
Ben is completely correct, and I immediately followed my own recommendations for the 100+ errant visualizations from week 1 of Makeover Monday 2017 and added this disclaimer to my dashboard:
Figure 1 — The disclaimer / warning that now appears on my dashboard.
So, why did I get in such a huff about folks not vetting their sources for the Australian pay gap but didn’t bother to vet the data for my makeover example?
It comes down to three things:
- The number of people participating
- The magnitude of the mistake
- The importance of the data
For the Australian pay gap makeover there were
ONE — Over 100 people making
TWO — Big mistakes with
Three — “High-stakes” data
Recommendations
Avoid “high-stakes” data sets. Some folks will make the argument that iPhone sales could be considered “high stakes” as someone might make a stock buying or selling decision based on this. Fair enough, but for MM I suggest Andy and Eva avoid data that deals with gender, race, guns, shootings, and anything else that is politically charged.
Get ideas from reputable data sources. Realize that source data from the Australian government is, in fact, good data. The problem is that it isn’t good for making meaningful gender comparisons. So, where did the assertion of giant wage gaps come from? The MM folks cited an article from Women’s Agenda. I spoke with data journalist Chad Skelton and he suggested that for high-stakes subjects MM should stick with ideas from the likes of The Guardian, The Wall Street Journal, and other well-known entities where reporters confer with statisticians and economists before publishing conclusions.
Note: Jeff Shaffer did some more digging and it appears that Women’s Agenda had in fact republished findings from Business Insider Australia. I have sent e-mails to editors at both publications.
Place a prominent Makeover Monday logo and disclaimer on every dashboard. Since Makeover Monday is an exercise in data visualization redesign, why not have participants trumpet MM’s purpose on every dashboard? By having a small but prominent logo along with a disclaimer we can both raise awareness of the project and caution people that the purpose of the dashboard is to practice data visualization techniques. My only concern is that people will think this gives them a free pass to present a bogus headline and/or specious findings.
While I think the logo below is spoken for, maybe something like it would help viewers know that for the dashboard in question, it’s not about the data, it’s about the design.
Figure 2 — Possible logo for Makeover Monday?
Vetting the data — the birth of “Find the Flaw Friday”?
In the previous post, I indicated that the one hour MM recommends people spend on a makeover is rarely enough time to properly vet the data, let along craft a visualization. And asking Eva and Andy to vet all the data is completely impractical as I’m sure it’s hard enough to curate the project as it stands now.
Many of the people I corresponded with have said that exploring and really understanding the data is as important, if not more important, than designing a cool dashboard (this gets us back to the substance over style discussion.) A lot of people indicated they find exploring and vetting data more interesting than fashioning visualizations.
So, how can we help people practice analysis and not just design and presentation?
At least five people indicated they would be up for what I will call “Find the Flaw Friday” (FFF) where people are tasked with exploring a data set and determining what analysis would be sound and what would likely to be flawed. I’m not sure how easy this will be to manage and how much time people will need to spend on each project, but I will put the folks that expressed an interest in touch with each other and we’ll see what materializes.
If this works perhaps the data sets from FFF could feed MM?
Final thought — Please fix (or take down) your erroneous dashboards
I consider wage inequality a big deal, and if you are going publish anything that’s about a big deal, you need to get it right.
So, now that people know that their week 1 MM dashboards are wrong, what should they do?
FIX THEM!
In conferring with Chad, he wonders how people would deal with something like this in their jobs. For example, suppose you create a beautiful visualization for the CEO and realize later that the central point of the viz is deeply flawed? Even if the mistake weren’t your fault would you rush to correct the record to keep your reputation intact?
Everyone — and I would argue that this should come from the Andy and Eva as well as the folks that manage Tableau Public — should tweet about the errors to make sure people don’t continue to perpetuate the misinformation.
It’s one thing to get something wrong. It’s another thing to know something is wrong and not fix it.
So please, fix it.
Postscript
Tableau Zen Master Brit Cava recommends a fascinating Freakonomics podcast with economist and gender gap expert Claudia Goldin. See http://freakonomics.com/podcast/the-true-story-of-the-gender-pay-gap-a-new-freakonomics-radio-podcast/. VERY worthwhile.
