In the biggest project of its kind, Brian Nosek, a social psychologist and head of the Center for Open Science in Charlottesville, Virginia, and co-authors repeated work reported in 98 original papers from three psychology journals, to see if they independently came up with the same results. The studies they took on ranged from whether expressing insecurities perpetuates them to differences in how children and adults respond to fear stimuli, to effective ways to teach arithmetic. There were completed replication attempts on the 98 papers, as in two cases replication efforts were duplicated by separate teams.
But whether a replication attempt is considered successful is not straightforward. Today in Science, the team report the multiple different measures they used to answer this question 1. Another method assessed whether a statistically significant effect could be found, and produced an even bleaker result. The team also found that the average size of the effects found in the replicated studies was only half that reported in the original studies. There is no way of knowing whether any individual paper is true or false from this work, says Nosek.
Either the original or the replication work could be flawed, or crucial differences between the two might be unappreciated. Overall, however, the project points to widespread publication of work that does not stand up to scrutiny. The current amount, he says, is near-zero. The work is part of the Reproducibility Project, launched in amid high-profile reports of fraud and faulty statistical analysis that led to an identity crisis in psychology. The goal as stated before is the efficiency of feasible alternatives, which means the more things you compare and the more thought into the cost of each action the better you are to inform others about that efficiency.
Everything needs to be viewed as a measure of one action versus another, as all actions, even doing nothing, will have an outcome. Doing this also helps eliminate making false statements about causal direction as those can be deadly to overall program efficiency. Just because more people used internal search or click on your banner and you made more money does not mean that you made more money because people did that other action. At the end of the test you have exactly one data point, that both went up, or one went up and one went down, or both went down.
One data point is not enough to establish a pattern, correlation, or especially causation. What will happen though is that your preconceived notions and opinions will naturally try to explain this relationship, which can lead to wasted resources and limited gains in the future. Focus on what you know, not what you want to know or think you can deduce. The very nature of running a program this way means that you will consistently be proving people wrong.
While this is the best thing for performance it creates a constant state of cognitive dissonance that if you are not prepared to deal with can blow up your program and lead to people creating reasons to not have faith in your tests. It is painful for people to believe something that they have done for years in the past is not only not as valuable as they think, but in many cases not valuable at all and negative to the business.
If you are not coaching this before hand and if you are not prepared to deal with this you are going to run into a large number of landmines. The first and most vital step to dealing with this is to focus all discussions on the comparing of actions and not on validating opinions. It is about the various influences of each option and not about any individual idea or concept. All ideas get treated the same, whether you think they will win or not.
Are People With High IQs More Successful?
By doing this you are taking the fight out from a me versus you attack and instead focusing on the system and the outcomes. The second tactic is what this entire article is about, discussing just how valuable being wrong is. If you have that discussion outside of test ideas and if you reinforce it in every conversation then you are opening up that door to hold the conversation when it really matters.
It is even better if you are championing how great a result is to the rest of the organization when you find something that goes against conventional wisdom. Doing this the first few times prepares people for this being a consistent and good outcome of future tests. In the case of the shopping cart test I mentioned before one of our senior executives through up their hands and proclaimed how funny it is that they are constantly wrong on each test. They were prepared for it and allowed us to make the changes because we had been preparing them since day 1.
- Poetry Through My Journey: Me, Myself & God!
- Are People With High IQs More Successful?.
- Marthas Kinder (German Edition).
- Six research-based strategies to help you overcome barriers to success?
- Cambiar es posible (Otras publicaciones) (Spanish Edition)!
- Get Your Number.
- The 25 Most Influential Psychological Experiments in History.
The third tactic is to simply ensure that each test has variants that are there purely because they go against conventional wisdom and the thought process that lead to the current status quo. By having things designed to break opinions you can leverage the learning from those to build the case in the future when one wins. The last main tactic is to have an education program consistently going within your organization. Meet regularly with each key group and inform them about what testing can do, past experiences and what to expect, and help them think about future efforts you can do to assist them.
You may not win that individual battle and get them to champion an outcome, but by doing this you are opening up a conversation and allowing them to hear about what you are trying to accomplish away from being in the heat of an argument. I would strongly suggest some sort of regular conversation at least once every other month in larger organizations and ongoing conversations in smaller organizations. In all cases you have to choose your tactics based on the people and the place. By far the most important work I do is dealing with cognitive dissonance and helping grow an understanding of what you are trying to accomplish.
Trying to get people to think in terms of feasible alternatives, being wrong, and rational decision agents is a big deal and is not part of anyones day to day activities. We are wired and trained from an early age to please people and to try and get that gold star from being right. It takes a lot to realize that being right or being wrong is irrelevant, getting results in an impartial way and working together to make everyone better is what really matters.
I want to close with a few helpful rules that might make many of the concepts I discuss above and in the future be acted on.
Take a second to really think about the core focus on your program. It is so easy to fall into the trap of thinking of testing as a way to prove a point or validate a change on the site. There is value in this but it is so little and so inefficient to what you can be doing. The first step of really getting results is to change how you view testing and optimization.
Everything you do, from how you talk to your organization, what tests you run, and how you leverage tools is shaped by your fundamental understanding of what matters in testing. Your own cognitive dissonance is the the first hurdle to really changing what and how you run a program.
A Modern Look at Terman's Study of the Gifted
Really evaluate what actions you take and what you are accomplishing with your program. Most importantly avoid going off track as it can be extremely difficult to get back on track the more you allow others to drive you towards less efficient outcomes. Featured image credit. Andrew specializes in building optimization and data programs into world class and efficient revenue producers. He has 14 years experience in conversion optimization and has worked with over different organizations.
All thoughts and opinions are his own and do not reflect on any organizations with which he is affiliated. Read his personal blog for great optimization insight. Spot on with this write-up, I actually feel this site needs a lot more attention. This editorial piece is written up in the simple words with insightful secrets. After Terman's death in , other psychologists decided to carry on the research, dubbed the Terman Study of the Gifted. The study continues to this day and is the longest-running longitudinal study in history.
Among some of the original participants of the Terman study was famed educational psychologist Lee Chronbach, "I Love Lucy" writer Jess Oppenheimer, child psychologist Robert Sears, scientist Ancel Keys, and over 50 others who had since become faculty members at colleges and universities. As impressive as these results seemed, the success stories appeared to be more the exception than the rule. In his own evaluation, Terman noted that the majority of subjects pursued occupations "as humble as those of policeman, seaman, typist and filing clerk" and finally concluded that "intelligence and achievement were far from perfectly correlated.
Researcher Melita Oden, who carried on Terman's research after his death, decided to compare the most successful subjects Group A to the least successful Group C. While they essentially had the same IQ levels, those in Group C only earned slightly above the average income of the time and had higher rates of alcoholism and divorce than individuals in Group A. According to Oden, the disparity was explained, in large part, by the psychological characteristics of the groups.
Those in Group A tended to exhibit "prudence and forethought, willpower , perseverance, and the desire to excel.
This suggests that, while IQ can play a role in life success, personality traits remain the determining feature in actualizing that success. This included the impact of the Great Depression and World War II on a person's educational attainment and gender politics which limited the professional prospects of women.
Other researchers have since suggested that any randomly selected group of children with similar backgrounds would have been just as successful as Terman's original subjects. One thing that IQ scores can reliably predict is a person's academic success in school. What it doesn't suggest is that a person will be successful at work or in life as a result of those numbers.
In some cases, it may just be the opposite. In fact, some studies have suggested that children with exceptional academic skills may be more prone to depression and social isolation than less-gifted peers.
Another found that people with higher IQs were more likely to smoke marijuana and use illegal drugs. One explanation for this, according to the researchers , was a personality trait known as openness to experience. This trait is one of the key personality dimensions described in the big 5 theory of personality. Openness is a trait that essentially removes unconscious barriers that would otherwise prevent a person from experiences considered socially unacceptable. Moreover, it is moderately associated with creativity, intelligence, and knowledge.