Post has attachment
(originally posted internally only at work on 1/13/17)

Let's start the year with an example that is, at least superficially, on the light-hearted side:  librarians creating fake patrons to check out important books that would otherwise be discarded for lack of popularity.

Underneath are two deep and important dysfunctions.  The specific one -- aiming to show people only what they are likely to want to see based on popularity -- is how fake news spreads virally.  The generic one is the idea that metrics provide a straight line to decisions with no thought or interpretation required along the way.

"The problem here isn't the collection of data: it's the blind adherence to data over human judgment, the use of data as a shackle rather than a tool."

Post has attachment
(originally posted internally only at work on 11/28/16)

Here's an interesting way to take metrics dysfunction to an entirely different level.  Examples abound of optimizing for only easy-to-measure aspects of an activity and getting, well, what was measured.  The new and innovative idea described here is to select among activities based on which of them have easy-to-obtain metrics.  

Another way to look at this is to view it as a version of the problem of measuring inputs rather than outputs.  Choosing a course of action should be based on some model of expected value -- and metrics should be chosen based on the actionable insight they are likely to yield.

Post has attachment
(originally posted internally only at work on 10/25/16)

During a presidential debate in 1980 Ronald Reagan famously asked "Are you better off than you were four years ago?".  His next sentence is less well-known but it shows how quickly we move to conflate measurement of human well-being with financial and economic metrics:  "Is it easier for you to go and buy things in the stores than it was four years ago?"

A previous post here ( pointed to the omission of unpaid work -- most egregiously household work often done by women -- from national accounts.  The key problem is that there is no money transaction involved, and that's all the processes can account.  Ironically, if the stores in Reagan's second sentence had started giving everything away for free, GDP would have gone down and people could have logically answered "no, it has become impossible to buy anything".

Which brings us to today's item.  While it is relatively easy for economists to continue to gloss over the contributions of homemakers everywhere it is harder to explain how some of the most prominent -- and, according to stock markets, most valuable -- global companies add nothing.  And, in this case, the gap between the information needed and the metric used has international political consequences.

Post has attachment
(originally posted internally only at work on 9/13/16)

If Wells Fargo had set out to write a cautionary tale, a case study of data-driven dysfunction, they could not have succeeded better.

The first article posted here showed how creating strong incentives to hit numerical targets drove people over the edge. Entirely predictable, and straight out of Austin's Measuring and Managing Performance in Organizations. (The first post in this collection links to that book.)

But why? Why push for more and more accounts, particularly from existing customers? If someone is already a customer, is there really enough margin in that person having two checking accounts rather than one? Now we have a report pointing to Data Science getting used as a cudgel by someone who didn't understand the relationship between correlation and causation:

"It all stems from Wells Fargo's internal goal of selling at least eight financial products per customer. It's what Wells Fargo calls the "Gr-eight initiative." Currently, Wells Fargo boasts an average of about six financial products per customer."

While I have no direct inside knowledge, I'll bet dollars to doughnuts that someone found that customers with eight financial products were the most profitable. Without enquiring further into why that might be so, someone dreamed up Gr-eight. (A plausible explanation, by the way, is that relationship customers are more profitable than transactional customers and that people with a strong relationship with the bank are more likely to turn to it as their financial needs change.)

Post has attachment
(originally posted internally only on 9/9/16)

Wells Fargo paid $185M in fines and 5,300 people paid with their jobs to bring you this perfect example of metrics dysfunction.

The question that does not seem to have been answered is whether or not the people who set up the system -- the people who put 5,300 line workers in the impossible situation of choosing between losing their jobs when they got caught eventually or losing their jobs right away for not meeting targets -- have themselves been fired. Or do the higher level metrics show that the system was actually profitable, even net of the fines and costs of employee turnover?

"When you pay people to do something measurable (like open accounts) they do more of it — and not necessarily in a way that actually increases the underlying figure (profit) that you wanted to raise."

Post has attachment
(originally posted internally only at work on 8/31/16)

Careful, thoughtful, painstaking use of metrics; clear aspirational goals; intrinsically motivated people with no desire to game the system: all together, a big improvement.

"For five years Charlie took it upon himself to create a new workflow system for the tracing center, breaking down each step in the tracing process into equations, doing time-motion studies for actions as minute as how long on average it takes the ladies to go from their desks to the roll room. Every step was analyzed and rethought, the numbers crunched.

And now? Despite no increase in budget, no new technology, no new staff: `I'm doing twice as many guns, twice as fast, and almost twice as accurately as we did when I got here in 2005.'"

Post has attachment
(originally posted internally only at work on 8/8/16)

A large and possibly calamitous financial hole for the aged and infirm at the intersection of two metrics tied tightly to incentives.

Flexibility and discretion in the application of rules invites abuse and corruption. Inflexible application produces unintended consequences and invites efforts to game the rules.

Post has attachment
(originally posted internally only at work on 3/28/16)

"The main measure of economic activity, GDP, counts housework when it is paid, but excludes it when it is done free of charge. This is an arbitrary distinction, and leads to perverse outcomes. ... The usual defence is that measuring unpaid work is hard."

The perverse outcomes are in many ways visited upon women, who do most of the world's unpaid work; the hard work would have to be done by men, who make up most of the economics profession.

Post has attachment
(originally posted internally only at work on 3/10/16)

"`The p-value was never intended to be a substitute for scientific reasoning,' the ASA’s executive director, Ron Wasserstein, said in a press release."

Note that p-values are metrics, measures of a particular probability under certain assumptions. As usual, the Goal - Question/Signal - Metric framework is appropriate: the goal would be to determine, say, if a new drug is efficacious in the treatment of some condition; the question would be whether or not it could reasonably be concluded to be the cause of some observed effect in some experimental trial; and the p-value would be a metric that would help answer that question.

Post has attachment
(originally posted internally only at work on 2/24/16)

A colleague mentioned the hazard of biasing a research investigation towards the data that are readily available over the data that are most desirable.

That seemed a good cue to post a link to this old chestnut.
Wait while more posts are being loaded