Jul 252011


Last year, UN Global Pulse launched a large-scale mobile phone-based survey that asked people from India, Iran, Mexico, Uganda, and Ukraine how they were dealing with the effects of the global economic crisis.

The survey (conducted from May-August 2010) asked two multiple choice and three open-ended questions focusing on economic perceptions.

Note: The fully working dashboards may be found at the end of this blog post.

Key Findings

Responses from Uganda – a country that ranks in the bottom 15th percentile in the UN’s Human Development index – were consistently more optimistic than responses from other countries.

What could account for this? Is it that Ugandans are, as a group, more hopeful and optimistic than people in the other countries surveyed?

Or could it be that survey responses were somehow skewed?

Let’s explore the data to find out.

Voices of Vulnerable Populations during Times of Crisis

Clicking the second tab displays the following view.

Economic Change Index

So, why in the first graphic does Uganda warrant a positive blue bar and Mexico a negative orange bar?  By moving your mouse pointer over a bar you can see just what it is that drives the Economic Change Index.

Here are the results for Uganda…

… and here are the results for Mexico:

The index itself (1.2 for Uganda and -1.6 for Mexico) is computed by applying Likert-scale values to each of the possible question responses.  We’ll discuss the advantages of using this approach in a moment.

Fixed Responses vs. Using One’s Own Words

The first two questions in the survey gave respondents four choices from which to choose.  The remaining three questions allowed people to respond in their own words.

You can explore these responses yourself by picking a question and a country from the drop down list boxes.

So, does the sentiment shown in the first fixed-response apply the open text responses as well?

Promising vs. Uncertain

Here is how people from Uganda responded to the question “In one word, how do you feel about your future?”…

… and here is a visualization of the responses from Mexico.

This, combined with responses to other questions, left me scratching my head. What are we not seeing that would lead to responses from Uganda — a country that is arguably in worse condition than the others — being so upbeat?

If you can’t wait for the answer, click here.

A Word about Word Clouds

I’ve analyzed a lot of survey data and I hate analyzing survey results where people get to provide free-form text responses because aggregating responses based on a common sentiment can be very difficult.

In many cases Word Cloud generators can convey the overall sentiment from multiple text responses.  They are also interesting to look at and I do believe the ones shown above are a good reflection of respondent sentiment.

A problem occurs, though, when respondents use different terms that describe the same or similar sentiment.  Consider the Word Cloud shown below.

One might think that most respondents were happy, but look what happens if we “linguistically normalize” the terms that are synonyms of “sad”:

It turns out that more people are in fact sad.

Note: There are products that are capable of parsing full sentences and are able to “disambiguate” and then normalize terms under umbrella concepts. The text responses to this particular survey, however, do not warrant this type of heavy artillery.

How We Calculate the Indices

The next tab in the workbook shows some alternative ways of visualizing the fixed-response survey results.

For these questions respondents were given four choices:

Easier / Better


Worse / More Difficult

Much Worse/ Very Difficult

Notice that we display the calculated index atop the Likert-scale stacked bar charts.  There are three advantages to calculating an index for Likert-scale responses:

  1. It makes it easy to weigh sentiment across many responses.
  2. It makes is possible to track sentiment changes over time.
  3. It makes it possible to compare results against various objective economic indices (e.g., GDP, UN HDI, etc.).

Note: I have no problem using even-numbered Likert scales, but I do think in this case sentiments will be skewed towards the low end as there are two levels of pessimism (e.g., “worse” and “much worse”) and only one of optimism (e.g., “better”).

I attempted to combat this by applying the following values to the responses:

Easier / Better = 3

Same = 0

Worse / More Difficult = -2

Much Worse/ Very Difficult = -4

While I think these values make sense, users of this dashboard are welcome to use the sliders and apply different values to each of the answers.  The indices will be recalculated automatically.

A Composite Index

In an earlier version of this dashboard I created a “composite index” that combined results from the two fixed-response questions:

I think this is a valuable metric and one that I would include should UN Global Pulse make this study longitudinal (see below).

Mobile Pulse Survey Results vs. Objective Economic and Human Development Indicators

In the next tab we see survey responses (first column) vs. the United Nations Human Development Index Ranking (second column).

What could account for Ugandan survey respondents being the most optimistic despite the fact that they rank 143 out of 169 countries in the UN’s HDI Ranking?

I believe that the survey’s SMS Text-based approach is skewing the results.

Consider the third column where we see the number of mobile subscribers within a country as a percentage of that country’s population.  In Uganda, at most 29% of the population has a mobile phone suggesting that those completing the survey may be better-off financially than others within their country. Survey responses may not, therefore, be a reflection of the country as a whole. (See The CIA World Factbook for mobile phone subscription information.)

This would not be the first time premature reliance on phone polls has derailed a survey (or in this case, just part of a survey).  See A Couple of Interesting Examples of Bias and Statistical Sampling.

Make the Survey Longitudinal

Despite the shortcomings, I think there is a lot of value in conducting these types of agile, real-time surveys.

One ongoing challenge will be comparing subjective data among different countries as there are so many cultural / proclivity issues that are difficult to compare.

One way to do this would be to conduct a longitudinal study and see how sentiment changes over time.  That is, instead of comparing Uganda with Mexico or India with Ukraine for a given year, track the changes over time, using an index.  This would allow you to see the percent change in sentiment between time periods without having to worry about normalizing cultural differences.

I hope that UN Global Pulse will update this survey on a regular basis as there’s much we would be able to learn from such a study.


[suffusion-the-author display='description']
 Posted by on July 25, 2011 4) Health and Social Issues, Blog  Add comments

  2 Responses to “Hopefulness and Hopelessness – Voices of the Vulnerable During Economic Crisis”

Comments (2)
  1. While you do make some very good points on phone market saturation, I think you are also only looking at one possible cause instead of trying to understand the overall picture of the difference between 2 countries such as Ukraine and Uganda.

    For example looking at development indicators from
    Comparing specifically Uganda and Ukraine we can see that Uganda has had steady growth over the last 25 years with a usual growth rate above 5% GDP per year, even during the recession they gained nearly 9% in 2008 and 7.1% in 2009. Meanwhile, Ukraine has grew at a 2.1% in 2008, and dropped 15% in 2009.

    Comparing other data besides GDP we see similar results where Uganda has more positive results then other nations in the study. Things such as % of people with access to drinking water, agriculture land %, availability of sanitation services, etc. are all on the rise in Uganda while in Ukraine some of these have dropped or are stagnant.

    Looking at just this data would lead me to conclude that people in Uganda have much more reason to look forward to the future, because they more and more of them will be getting better water and sanitation services then what they currently have, and from the GDP figures, looks like most of them do not have to worry about jobs, or at least losing them if they have them. Ukrainians on the other hand have every reason to be scared or frightened of the future. They dropped 15% in one year, which most likely caused a very large increase in unemployment. Some Ukrainians may even still be worried about losing their jobs with the economy being fragile, and most likely the ones without jobs are having problems finding them.

    On top of those looking at Migration, Population, and Population growth rate, Uganda has a 3% population growth rate while Ukraine is losing .5% every year. Migration out of these countries is similar while their populations are at 32 million for Uganda and 42 million for Ukraine. 

    The big problem with negative population growth is as a country you need to expand your infrastructure, etc. But when you are losing people, your revenue is most likely dropping as there are less and less people paying taxes each year. This increases the tax burden on the people living in the country, and will thus most likely lead to less spending on development by the country. Another thing to notice is the huge spike in migration out of Ukraine 10-15 years ago. This is also bad, because most of the time migrants are the ones that have the financial means and desire to leave the country, which is most likely the brightest and entrepreneurs.

    While the point that you make about the % of phone users in the country of Uganda is valid and most likely does skew the results some, I do not believe after looking over all the data between the 5 countries that it skews it to the point that would make this survey invalid, or even push Uganda real far from it’s current postion.

    Overall from studying the 5 countries compared side by side, I would agree that Uganda has the most to look forward to out of all these countries, and the results show that. 

    • Chris,

      Perhaps you are correct.

      But the lack of correlation with the UN’s own HDI (which looks quite rigorous to me) set off lots of red flag.

      I’m currently reading “Half the Sky: Turning Oppression Into Opportunity for Women Worldwide” by Nicholas Kristoff and Sheryl WuDunn (very good book, BTW.) Here’s a quote that resonated with me:

      “… To be blunt, humanitarians sometimes exaggerate and oversell, eliding pitfalls. They sometimes torture frail data until it yields the demanded ‘proof’ of success.”

      Kindest regards,


Leave a Reply to Chris Cancel reply