Friday, March 2, 2018

Tinkering with Data until it looks Pretty

Experienced Data scientists know very well that you can beat data long enough to get it to say what you want. The recent playbook in economic sector in India highlights how you can get the data to speak the truth and lie at the same time.

In the last 4 years, the methodology of how a few key metrics are calculated have been changed.

Changed in
What changed?
Base year changed from 2004-05 to 2011-12
Index of Industrial Production (IIP)
Base year changed from 2004-05 to 2011-12
Wholesale Price Index (WPI)
Base year changed from 2004-05 to 2011-12
Consumer Price Index (CPI)
Changed in 2015 and now in 2018
Base year changed to 2010 and then to 2012
Base Year
Plan is to make the base year 2017-18
Min. of Statistics and Programme Implementation
is planning to change the base year for calculations
of most national statistics
The risk is that such revisions are NOT accompanied with backward calculations or back series. Hypothetical example - if you change 1inch from 2.54 cm to 4cm from today and do not revise old measurements and use (1in = 4cm) for measurements from tomorrow. There's going to be lot of confusion.
Also, changing the base year to 2017-18 by skipping a few years doesn’t add up.
And since key policy decisions are based on these national statistics, ambiguity about the trend will not help in taking accurate measures.
[In case someone is unaware - these metrics are calculated by the govt - NSO, MoSPI as only the govt. has the most comprehensive data. ]
Out of the mainstream print and electronic media - only Business Standard seemed to highlight this technical anomaly.

without the back series it becomes difficult if not impossible to make substantial and meaningful assessments about the changes. Moreover, if such changes are brought about too frequently they can also cause a fair amount of confusion and make it very hard to read economic trends and understand policy impact.

This is what the establishment seems to want to happen. The GDP methodology was revised in 2015 and the data for old years w.r.t new methodology were not released. The  GDP numbers calculated henceforth, were based on a new methodology seemed to gave it a little upward push. Now comparing with old GDP numbers prior to 2015 is pointless as they both were calculated differently and the new numbers look more robust and betray the  naked eye.

Secondly, terms like CPI/WPI/IIP are Greek and Latin to almost 98% of the populace. In election campaigns the numbers will be touted which will invariably be higher than the old numbers and the fact that the method via which it was calculated will be lost on everyone. As it is said the devil lies in the details - not many will bother to look under the hood and those who do will be relegated to academic journals and blogs like this.

It is a far-fetched dream that the masses choose their leaders based on their proposed economic policies and question them on their financial feasibility specially in India where caste and religion drive the voting patterns. Still, it is the responsibility of the educated and the fourth pillar of democracy to at least raise questions and bring it into the public discourse.

Sunday, October 22, 2017

The Conquest of Happiness - Book Review

After a few years when I will look back at this book there will be quite a few things that I’ll be able to recollect. I can easily make lot of associations with this book. I read one of the chapters of this book in a Critical Reading class, bought it at a real cool bookstore, happened to stumble upon a hidden treasure of a coffee shop where I spend some afternoons perusing it. And finally, some clarity of thought which I gained after completing it.

As is clear by the title – it is mainly about how we can stay happy.  
I am particularly struck by the mention of “the pursuit of happiness” as one of the inalienable rights of the citizens in the U.S declaration of Independence. Seems like the Committee of Five were wise enough to know what matters eventually. There is abundant material floating around which claims to provide answers to the complex questions people spend their lives struggling with. Bertrand Russell does a really good job stripping down the needs and necessities of a man/woman and come up with what are few basic things which can provide contentment/happiness/whatever you call it.

In the first half of the book – he lists the possible reasons why people are unhappy: -
  •        Mistaken views of the world
  •        Mistaken ethics
  •        Mistaken habits of life
  •        Absence of effort from life
  •        Absence of love
  •       Absence of quiet life
  •       Too much to worry
  •       Envy
  •       Fear of public opinion

He lists with examples, anecdotes going into detail as to how the above characteristics contribute to disillusionment with life and makes a person incapable to be happy. For instance – he so rightly points out that “boredom” is an essential ingredient of life. Even though most of us try to always indulge in exciting endeavors – little do we realize that their value will start diminishing and the need for stronger levels of excitement then arises. He also so wisely states that “the lives of most great men have not been exciting except at a few great moments.” The manner in which simple, unadulterated truth is presented in this book is what makes it such a classic even 80 years after it was first published. Russell clearly defines that certain great things in life are possible only with a certain degree of monotony. I had never thought about boredom through such a perspective. Point to be noted – not to feel bad if you have a low-key weekend or vacation!

Similarly, he talks about fatigue – how indecisiveness can cause fatigue and the buildup on fatigue (mental) causes a person to stop living in the moment. The best way to prevent it is to face our anxieties head-on i.e thinking of the worst-case possibility. Often, we will realize that the “worst case possibility” isn’t as big as we thought it would be. That realization can provide a sense of relief.

After listing down the possible avenues of unhappiness – he starts with whether happiness is even an achievable goal. He draws a subtle (but very interesting) distinction between the level of happiness which a scientist and an artist can attain. He points out that even though the general public cannot understand a picture or a poem – they conclude it is a bad picture or bad poem. But when they cannot understand the theory of relativity they conclude (rightly) that their education has been insufficient. In a lot of places, you can spot Russell’s dry wit humor.

The last 7 chapters are dedicated to possible ways in which happiness can be derived.

In no proper order, he implores the reader to -
  • Let your interests be as wide as possible and let your reactions to the things and persons that interest you be as far as possible friendly rather than hostile.
  • Give affection to your dear ones as affection is reciprocal by nature.
  •  How having a tight-knit family contributes to a sense of satisfaction.

  He calls upon lot of focus to be given to personal relationships and upbringing kids. 
  • Need to Work.

He could clearly see much earlier in the 1930s that no matter how dull the work would be, life would be much worse if there was no work to do. Russell blames it to the failure of man to rely on being told what to do. As per him, man has not evolved yet to intelligently use his leisure time. Thus, the need to get up in the morning and do some (any) work remains important.    
  • Put in Effort.

You have to have a life where you put in effort. Also, it is extremely important to not fret over small things like missing the train or things that upset you. A wise man does not expend emotions over things that go wrong. Russell shares a powerful trick to put things into perspective. Ask yourself when things go wrong – In the history of the cosmos does the event in question has lot of importance?

With his simple powerful words the author is successfully able to drive home the point that happiness is not elusive. It is attainable. Most of our “unhappiness” is our own doing, our improper thoughts and expectations.
If we start approaching life with the right perspective which he shared we will become A Happy Man or Woman.

P.S - The very few things I did not like was that at places Russell seems to come across very conservative by nature specially when it came to women shouldering working opportunities. If you think of the time when he wrote the book it explains his viewpoint. At that time women were not even remotely provided equal opportunities to go out and work. Probably, that is the reason why he couldn’t come to the conclusion that factors affecting happiness of women were the same as men.

P.P.S - The other thing that stands out from the book is that Russell views drinking in a negative light. He says it is a shame that people are too tired to be capable of enjoying without the help of alcohol. At one place he mocks those who drink a lot - Me no drinkee for drinkee, me drinkee for drunkee!

P.P.S.1 - A bit of trivia.
Einstein is usually praised in the mainstream literature for the stance he adopted against nuclear weapons and the Nazi party. Both Bertrand Russell and Albert Einstein signed statements against the adoption of nuclear weapons. And then Einstein went back to his research while Russell continued his struggle on the streets raising voice and trying to bring awareness on that issue. In Noam Chomsky's words - Einstein didn't rattle too many cages but Russell tried to do something about it. Thus, Russell was viciously attacked by the NYT and others. He was disregarded as a public intellectual and rather considered a crazy madman.