Posts Tagged 'education education education'

Nate Silver’s numbers game

Nate Silver: hard to believe this man is a statistician

For the second US election in a row, the winner is a guy called Nate Silver, who might be the future of intelligent journalism. He rescues us from the tyranny of columnists who simply write about the comments of their own heads.

Nate blogs at fivethirtyeight.com, which is, since 2010, a New York Times blog. He analyses opinion polls, but he does it very, very well. He is entertaining and readable, even if you don’t care who just won the election in the US.

I discovered Nate’s analysis by accident in 2008 when I was looking for some statistics to undermine one of the nuttier blog opinions by data-lite controversialist Melanie Phillips (which made it very nutty indeed). Fivethirtyeight has a rigour that journalism seems to have mislaid in the internet era in a search for sensation. He does a seemingly simple thing extremely well: when an opinion poll is released, he adds it to a model which creates an aggregate. If the model is well-constructed, this has smaller margins for error and less chance of systematic bias. It is more likely to reflect the true state of the world.

The clever part is that he doesn’t just produce an average. He weights the polls, depending on their sample size, the way the information was obtained, the historical accuracy of the polling company, when it was conducted, the exact question that was asked, and so on. He looks for statistical bias – a consistent under- or over-reporting of candidate’s popularity. He adjusts his own model if he finds evidence that it is biased. Importantly, he writes nerdy blog posts about what he is doing, explaining his reasoning, and pointing out possible flaws in his work.

The result is that “outliers” – polls that, through random sampling, produce a freak result – have little importance on Fivethirtyeight – while on the internet and the news channels they tend to dominate the agenda, albeit fleetingly. This means his reporting is less shouty, but it has proved to be stunningly accurate for two elections in a row: at the time of writing, his analysis has correctly predicted the result in every state for the 2012 US presidential election, and the electoral college vote too.

Having a model doesn’t necessarily mean you will be correct – there are plenty of other statistical models which predicted the election less accurately. Fivethirtyeight carefully spells out the steps in its analytical process (though not the precise parameters of the model), so we can make an informed judgement on the quality of the findings. Any model is open to criticism from other statisticians – but this means they can have an adult, public conversation about what might be improved, or what the impact of a flaw in the analysis might be. We can learn from this, too.

This wouldn’t be important if it was just a different way to present the same news; but this type of analysis creates fresh insight. By polling day 2012, the model predicted a greater than 90 per cent chance of an Obama victory; and yet organisations like the BBC and the FT were using lazy phrases like “too close to call” and “on a knife edge”. If newspapers are prepared to do this type of analysis routinely, I suggest, it offers huge potential for creating an open, analytical type of serious journalism led by numbers and observed reality, not opinions.

Old jokes department: “And what do you do?”

Not every journalist can be a stats geek, though I think they should have more compulsory education in how to interpret data, and would prefer that newspapers enforced an in-house ban on reporting surveys that are statistical nonsense – which, in my experience, is most of them (I’ve written those survey-based articles in the past, and reported lots of rubbish data as if it were spotless, which I regret).

Newspapers and magazines are cutting back on conventional journalism. Budgets are tight. It’s probably too much to hope that we can create a new type of data-journalist, or that newspapers will suddenly grow a statistical conscience. It needn’t be expensive: a laptop and some specialist software is perfectly adequate to do the statistical research that can validate the claims that powerful people make. It’s the job of the media to investigate these claims – not just talk to one person who agrees, and another who disagrees. On Radio 4, More or Less does an entertaining job of validating reported statistics (download the podcasts, they are excellent). Ben Goldacre’s Bad Science posts are also a model of this approach.

It’s patronising to assume that readers can’t cope with statistical analysis. Clearly, many don’t like it, and some misunderstand it; but that’s true of any type of journalism that goes beyond the obvious. The conclusions (especially those that go against gut feel or conventional wisdom) may be unpopular: just read the critical comments on Nate Silver’s blog. It’s also true that science isn’t the last word on a subject, just a powerful way of testing an assumption. Statistics involves making value judgements in how you treat the numbers, in the same way as a journalist makes a judgement about how much credibility to give any source. But in statistics there is the opportunity to be explicit about those judgements, and then go where the numbers take us.

This type of insight is a fundamental tool, in an increasingly complex world, if we want to make informed decisions. The alternative is to just place trust in the conclusions of “experts”, of which there seem to be an ever-increasing number quoted on TV or in newspapers.

I’ll leave the conclusion to one of Nate’s commenters, who explains it better than I do:

Rather than cheer for Nate because we all like his Obama forecasts, how about cheering for him because he might believe in a world where numbers and rational analysis are vital to how we make decisions, even in those cases where we don’t like what the numbers imply?… It’s not about hoping you will win at Vegas. It’s about understanding how the Vegas game works.

Golden Crocoduck fever grips Talk Normal

Why post this clip? Because Sunday is a very important day for Talknormalists: it’s the day on which the annual Golden Crocoduck Award is announced. This is given by Peter Hadfield, also known as potholer54, to the creationist who offers the most flagrant breach of the 9th commandment: “the one that tells you not to tell fibs” as he explains.  The award is given every year on 28 October – the feast day of St Jude Thaddeus, the Patron Saint of Lost Causes. It’s too late for you to vote; but please check out the nominees in the video above (one of whom actually tries to convince viewers that the earth must be stationary because when we jump we land in the same place), and then the rest of his channel.

In the summer of 2006 I was asked to go on the BBC News 24 (now BBC news channel) to have a middle-aged-manchat with Peter Sissons about why there was this thing called YouTube which was becoming really popular with da kidz.  Why, he asked me, would people want to go to a web site to watch a fireman in a tumble dryer?  – he really did ask this – especially when there’s real TV on 24 hours a day, with politicians and experts standing outside buildings and stuff (he didn’t say this. I deduced it from the height of his eyebrows when he was watching the fireman video).

Get with it daddio! I sort of said – I was living in Hoxton, on the doorstep of London’s famous Silicon Roundabout, and we all spoke like that, even those of us who were old enough to know better – da peeps need to express themselves! This is the social media equivalent of the “everyone’s got a book in them” argument. It says that everyone has got a YouTube video inside them, waiting to get out: just maybe not a very good one.

So six years later I’m all for YouTube in principle, it’s just that much of it is total crap. I’m not talking about bands I don’t like or people I disagree with. There’s a ton of that on TV too. But watching the peeps express themselves on subjects such as why the earth is the centre of the universe, or how we know dinosaurs lived on earth alongside humans, can be quite disheartening. There’s eight years of material uploaded to YouTube every day. Much of it, it appears, is uploaded by people who were asleep in science lessons, aren’t really interested in looking up the answers, but still think we need to hear their half-formed ideas. Alternatively, they simply repost something by someone they decided to agree with, without stopping to think why they might learn something by checking whether it’s true.

Sometimes it’s hard to tell which bits are the worst, but luckily we have potholer54 to help us decide. If you have any interest in using logic and curiosity as a way to debunk pseudoscience, creationism and the denial of our role in global warming, he’s an excellent guide.

His videos are short (usually between 10 and 15 minutes), expertly researched with all sources listed, and very, very funny. It’s a great way to learn if ever you want to explain why there isn’t evidence for a global flood, or why Lord Christopher Monckton, the UK’s embarrassing denier of man-made climate change, is misinformed and wrong. The Monckton takedown starts here and ends (for now) here.

It’s hard to pick a favourite from his channel, but I always enjoy Hadfield’s demolition of American talk show host Bill O’Reilly:

Unlike most of his fellow ignoramuses on YouTube, O’Reilly is “paid to broadcast his ignorance to millions of people” (Hadfield’s description) via Sissons-era TV. This is one of the best arguments in favour of YouTube: without it we’d have a lot less potholer54, but the same amount of Bill O’Reilly.

Three principles of Talknormalism

Evidently I am still getting the hang of the whole motivation thing

Delighted as I am to be mentioned in the Observer yesterday, I worry that some readers might get the idea that I am a grumpy negative old man who thinks everything was better before the internet was invented.

This is incorrect in more than three ways. I’m actually middle-aged and remarkably cheerful, all things considered. Taking into account the decline in the TN Joy Index, I’m looking comparatively more chipper by the hour. I’m such a positive person that I spent 5 minutes inspiring others by creating my own motivational poster (top) using the free app at Big Huge Labs. I encourage you to go there and make your own version to share with us. If I get a few in I’ll put them in a post and I’ll send the best submission a Talk Normal mug and a signed book (or unsigned, if you’d prefer). What an incentive.

I also don’t think we should be stuck in the past. “But what does he think?”, you ask. At risk of telling you all the good bits about the book for free, I’d like to point out the three principles of Talknormalism:

1. Try to be understood by everyone who’s listening. This takes imagination. For example: Professor David Crystal, who is a wonderful writer about the history of the language, points out that there are 400 million native speakers of English – and 1.4 billion more who speak it as a second language.

It makes sense to consider people who don’t speak English first, especially if you’re in business: their domestic economies are usually more successful than ours.

2. Stop trying to sound clever for no reason. Anyone can make complicated things complicated. It takes thought to make those things easier to understand.

3. It’s about attitude, not rules. I would guess that there’s a lot of bad punctuation, in Talk Normal. Were I to be put to the test, I would not be able to remember grammar rules with which to make your writing more elegant, um, with. Therefore, if you want a set of rules to follow, try this book instead. I’m also not against new words and phrases; I make some up on this blog.

Talknormalism doesn’t look back to a fictitious golden age and ask that we preserve it; but we can do a better job with the language we have today. That’s why we need to see your motivational posters, fellow Talknormalists.

i no your gunna like dis

Thanks to talknormalist The Finch for this page of Facebook spelling and grammar errors, and the excellent putdowns that result. The moral: time spent spellchecking is time well spent. Another moral: Facebook friends are not always your friends.

Five things I learned from Google Ngram

Note: I promised that I’d ask Richard Stallman if I had represented his ideas correctly. Turns out I hadn’t. I have pasted his response in the comments section below. It’s a fascinating debate, and goes to the heart of Talknormalism: if you describe something with a misleading name, then you start to make assumptions based on the name, not on the facts.

Before I went on holiday, I pointed out the address of the Google Ngram viewer, which originally came from my creative friend Ryan Hayes. Ngram allows you to search for the frequency of a word or phrase in books going back to 1500: the database is 500 billion words. You type in the phrase, and it draws a graph for you.

You can imagine that after a couple of Mojitos on Miami Beach last month I was thinking about little else, so here’s what I thunk:

1. Intellectual property

I once wrote a book criticising the abuse of our intellectual property laws, but the more people I meet who have profited from them, the less I feel like defending IP in its current form. I particularly recall speaking at a very posh luxury goods conference in Paris which made me want to set about my fellow panellists with a cosh.

Dr Richard Stallman of the Free Software Foundation is the enemy of lazy IP thinkers, and in this article he argues that “intellectual property” is a meaningless term, popularised for propaganda purposes by the people who have most to gain financially from it, and that it was rarely heard until 1990. Here’s the Ngram of intellectual property that he created, showing how recent the concept is:

Original research: Dr Richard Stallman

2. Angels

As committed Talknormalist Brett Hetherington writes on his blog, “we live in superstitious times”. Having seen my tip, he used Ngram to go searching for “angels”, and discovered that, in a secular age, we’re actually writing twice as much about these fantasy beings as we were 30 years ago. Brett or Dr Stallman might argue that intellectual property is no more real than the idea of an angel: both concepts being a convenient construct designed to give power to, and increase the revenues of, global organisations that seek to exploit us. I’ll email Dr S to ask, and Brett can comment below if he thinks I’ve overstepped.

Original research: Brett Hetherington

3. Paedophiles (or pedophiles)

Moral panic or long-overdue recognition of a problem that was ignored for too long? Although the term was coined in the 19th century, we certainly write a lot more about paedophiles these days.

4. Low-hanging fruit

The Patient Zero of buzzword bingo was not always so pervasive. The phrase took off at about the same time as “intellectual property” did – probably because many of the same people were using both phrases. If some consultancy firm made up the phrase “low-hanging fruit” today, it would probably use IP law to protect it, and we’d all have to talk about MegaGlobalConsult Low Hanging Fruit™ instead. I think I’m saying that we got lucky, but it doesn’t feel that way.

5. Honesty and transparency

The great thing about “transparency” is that it doesn’t have ethical baggage – it’s a technical description of your activity that’s suited to amoral business relationships. Therefore transparency is a much more useful word than “honesty” if you work in marketing. Transparency is jolly popular lately, but honesty is in long-term decline – in books, anyway. And we write more often about angels than we do about honesty, which is proof that we’re collectively bonkers.

Show me the remuneration

I know not everyone had wine but we're still five Euros short

Justin Webb tweeted from Radio 4’s Today programme on Tuesday morning:

Can’t talk simple this am: pay becomes renumeration and softness emollience. coffee please

Several people tweeted while he was at the coffee machine to tell him that the word he was looking for was remuneration, but I sympathise with Justin. I didn’t realise that renumeration meant “counting something again”, and not “earned money”, until I was about 30 years old.

I still have trouble saying and typing the word. More embarrassing, I have similar problems with the word laboratory. It’s a good job I never worked for the chemistry press, or my peers would have split their sides when I asked questions at press conferences.

Justin and I are not alone with this remun/renum mix-up: journalists have used renumeration when they mean remuneration around 100 times a year for the last 10 years (You have no idea how long this blog is taking me to type). On 23 August, Chemical Week made the same error. Ha! Who’s laughing now, chemists? Wearing your white coats, reading your trade paper in your labra-, lobaro-, your places of work.

If you suffer because you can’t say or spell remuneration, I don’t care, because – unlike nucular – it’s a pointless word that doesn’t need our respect. It’s a classic case of word obesity. Take Justin’s advice and say pay instead: problem solved. Everyone understands what you mean and you don’t sound pompous. Both my 1965 edition of Fowler’s Modern English Usage (I write as if I have copies from other years), and Essential English for Journalists agree.

If you decide to use “pay”, you’re not following the trend towards increasing word obesity. I made an index of the relative frequency of remuneration against pay (both in articles also containing the word job) from 2001 to 2009. The data is from articles published in North America and Europe:

The complicated word is twice as frequent as it was in 2001, but “pay” is almost unchanged in frequency. Most of the growth in use of renumer-, remuren-, that word has happened since 2007. Perhaps it is because we are rarely more than a day away from a story about what bankers are banking for themselves, and “pay” doesn’t seem grand enough for their piles of unearned income and bonuses.

Maybe the word we use to describe income should have at least as many letters as the income has digits, which means that only cleaners without visas and bloggers earn “pay” these days.

What about the rest of us? For Talknormalisers who want to feel special about pay, I’d suggest describing it as compensation. It’s still a pompous way to talk about money, but I like it because it sounds like they gave us the cash out of sympathy.

A near-death experience and made-up fluff

The author of the definitive biography of Canadian prog-rockers Rush is featured in this post! It doesn't get better than this, you're thinking.

Three weeks without a post! You must have been worried sick. Turns out I wasn’t crawling across the floor, desperately to reach the keyboard to tap out one last post before the horrific injuries inflicted by a deranged stalker finally did for me. I was doing my University exams, thanks for asking, which were arguably less fun.

The curse of antitalknormalism pursued me even in the musty corridors of the University of London. As one of my lecturers warned me: “In macroeconomics the answers aren’t hard. It’s trying to work out what they’ve asked you that’s the problem.” The second bit of that statement, at least, turned out to be true.

I’ve got a couple of posts ready for this week. Meanwhile, there’s time to answer the question asked by analyst and part-time Renaissance man Jon Collins, rock biographer (five stars on Amazon) and managing director of Freeform Dynamics:

No, Jon, I don’t make the graphs up.

The numbers are always real. Whenever there’s data and I haven’t told you where it is from, I found it by searching Factiva.com and sticking it into a spreadsheet.

On a personal level Mr Collins is both my webcast buddy and was voted the world’s second-best analyst (and the best in Europe) by a jury of his peers, so when he questions my rigour it is naturally hurtful. I’m only partially mollified when he tells me Talk Normal is his second favourite blog, because his favourite is the one where someone writes down what her husband says when he’s asleep, and I checked and she doesn’t have a single bar chart. But, being British, I’m keeping all this negative emotion inside so the bitterness can spill out inappropriately when I’m drunk.

Here’s another thing that might appear to have been made up for a joke. It’s a pygmy jerboa: an animal that looks as if the zoo ran out of fluff half way through inventing it.


Cut out your waffle: buy my book

Type your email and click the button and you will automatically get every new post.

“This excellent collection” (Director Magazine). Click to order:

I tweet