Trusting the experts

Trust the experts? Why would I be daft enough to do that? It is not that experts are wrong, although they are often wrong. It is that we are exhorted to treat them with reverence when that defies reason. — ephektikoi

Introduction

We routinely rely on experts for services and for advice. Does this work out? Sometimes the results are acceptable and sometimes they are barely adequate. In other cases, our interaction with experts becomes a disaster.

There are experts we can be pretty confident will solve our problems, and experts we need to treat with caution, if not just ignore. In some cases, the outcomes, positive or negative, only effect our pocket book. In other cases, the outcomes affect our long term health or our lives.

We are bombarded with opinion delivered by experts on a daily basis, some sounding authoritative, but with views contradicting one and other. Logically they can’t all be right, at least in those areas where views conflict. Maybe none are correct in any significant respect on particular topics.

There is not a lot of reason to universally trust experts, since they are frequently wrong, and in some disciplines, may be right only at the level of chance – they disagree, they agree, they get it wrong, they get it right.

Expert opinion, is it better than ours? That depends. Listening to the experts is sometime the best we can do, since the non-experts are not likely to be even as good. In some fields, experts are actually quite good, but in other important areas, they are not very good at all.

What Areas Of Expertise Are We Discussing?

Expertise can reside in trades, service industries, music, art, dance, sports, various professions, academia, healthcare, retail, consulting, finance, banking, mortgage, and on. Expertise can reside anywhere where knowledge, skill and experience go beyond the average. So, under this broad umbrella, many, if not most, people will have some areas of expertise.

I have several areas of expertise. It does not mean that I am at the top of hill in any of these particular regions, but I have competency in several things that are fairly complex.

What Services Do The Experts Provide?

Experts make the world run. They may do fabrication, construction, repair, give advise, explanation things, predict things, research things, teach things and render opinions.

Opinions

Too many “experts” start from the position that our current understanding is correct, and anomalous results must be wrong. You can be an absolute master of the body of knowledge of your discipline, and also of the schisms which invariably exist in any discipline, and be dead wrong on a large number of topics, because the body of knowledge in your discipline is very flawed. You can be an expert in bullshit to be more blunt.

Opinions, under the guise of journalism, is the domain of non-experts who think they are experts. It is often called (if honest) “opinion,” or called (if less honest) “reporting.” It is also the realm of any number of people who like to pontificate on things of which they have little first hand knowledge, or relevant qualifications.

Surely We Can Trust Science

Scientists are experts, but the history of science is to a great extent the history of error, and of mistakes in expert opinion. The battleground of science is littered with the corpses of damaged findings, defeated theories and the detritus of discarded hypotheses. It is not even clear that it is as self-correcting as many would have it.

Recent informed critiques of science have pointed out the problems facing the scientific enterprise. There is a problem with the ability to replicate findings for instance. Not only is replication seldom attempted, but when it is the failure to replicate is quite high, in both soft and hard sciences. A recent critique published in PLOS Medicine “Why Most Published Research Findings Are False” by John P. A. Ioannidis from August 30, 2005 has been a much cited paper on various problems with bio-medical research. There are many other informed critique in other papers, by other authors.

I have been saying for quite a while now that not only is science broken, but all disciplines are broken. I have informed opinions on the brokenness of science and the work of academics in general. Some thoughts:

  1. Science is never settled
  2. Science is not done by consensus
  3. Science is only somewhat self-correcting
  4. Dismissal of anecdotal evidence without consideration is irrational
  5. Ockham’s razor is past its prime, it never was a very good rule
  6. Experimental science is not the only type of science
  7. Science is not the only mode of useful inquiry
  8. If the effects are strong and clear, tests of significance probably don’t add that much value
  9. Peer review serves more to impede innovative work and force group consensus than to improve scientific findings

The disagreement of experts

Experts, like all of us, base their views on imperfect information. They fail to consider all of the available evidence, for a variety of reasons. They may:

  1. not be aware of evidence
  2. dismiss evidence without due consideration
  3. evaluate evidence according to their own prior beliefs and biases, in a theory-blind fashion
  4. evaluate within a flawed, imperfect body of knowledge from their domain of expertise
  5. not be particularly good at integrating large amounts of information
  6. lack sound judgment, common sense
  7. interpret things shaped by their culture, family experience, society, emotion, bias, exposure to the strictures of their discipline and incentives.

Storytelling

A lot of experts can tell a good story, a convincing story, but they will not agree. Different experts may tell different stories, and according to the law of contradiction they can’t all be right. Maybe none are right, in whole, or even in part. Well, maybe in part, sometimes.

Consequences of Error

When an expert is relied upon for his knowledge, and is wrong, the consequences may be bad and sometimes they are life threatening consequences. We tend to deferred experts anyway, although this may not be a safe bet.

The Key Limiting Factors

Expert performance, professional capability and competence depend upon, at the least, these factors:

  1. The individual – human capabilities
  2. The incentives underlying the individual and the collective – financial and otherwise
  3. The cultural and social matrix – social pressures, social factors
  4. The Institutions – The institutions involved, and their influence on the discipline
  5. The discipline – methods and body of knowledge in the domain of the discipline and its limitations

All of these work to determine expert performance. I discuss each in turn below.

The individual

Individual and personal factors and capabilities of the expert are of course of great importance.

Competence

An expert can be competent because of his training, experience, and the quality of the discipline. That is, competence because of the discipline. An expert can also be competent despite the problems with the discipline, because of his overall intelligence and judgment, and understanding of the deficiencies in knowledge in his field.

I should note that on any dimension, half are below average in ability, and half above; this is just basic statistics.

Bias

What are the biases of the practitioner? Experts, like all of us, are prone to rush to premature judgments. They examine a subset of the available evidence, and then draw a conclusion consistent with their current beliefs, according to their biases. We can only reason from current beliefs. It is difficult for many to honestly consider new information which conflicts with current views. More and more their views are held onto with intensity, the person becoming emotionally invested in a certain perspective. More and more their understanding is limited by confirmation and dis-confirmation bias. They may also become theory blind. By that I mean they see what they expect to see according to the dominant paradigm, and disregard things outside of the theory.

Interpretation of Evidence

Evidence must be interpreted, and this is done within the framework of existing beliefs and theories. These interpretations may sometimes be right, but clearly are often in error.

We all have idiosyncratic interpretations. It is inevitable that different individuals will understand things differently, interpret the evidence differently. Not only is evidence interpreted differently by different individuals, over time, interpretations by one individual may change.

Memory

No expert will understand, or retain in memory every aspect of their professional cannon. Far from it. In a complicated discipline, the task of learning the whole body of knowledge will be impossible. Forgetting starts immediately, and after a short period of time, only some of the material studied will be retained. There are individual differences in this regard, but the general case is that we forget a lot of material.

Intelligence

Intelligence enables or restricts the performance of an expert. How bright is the expert? Do they have the ability to integrate large amounts of perhaps overwhelming information? Does the practitioner have the types of intelligence demanded by their specialty?

Intelligence has different aspects. There may be some overall general intelligence, but there are surely special type of intelligence that give capabilities beyond that.

How much time does the expert give to reflection about his discipline? I consider reflectivity an aspect of intelligence, although it probably does not show up in IQ tests. IQ is based on speed, reflection is based on slow examination, and never shallow analysis.

Emotional intelligence not only helps in dealing with people, but also is necessary for sound judgment.

I consider humility an aspect of intelligence, although it probably does not show up in IQ tests.

Is this person creative? Is the practitioner able to think outside the box, to think laterally?

I consider open-mindedness an aspect of intelligence. How open-minded are they? How close-minded? How willing is the expert to consider other points of view? How stubborn are they? This is a character trait, but it may influence intelligent thinking. When an expert is wrong, they will often hold fast to their dogma beyond all reason.

Knowledge

What is the depth of training and experience of the expert in his specialty? How well does the expert understand the body of knowledge of his discipline? How well does the expert understand the methods of his discipline?

Does he go back and refresh his knowledge at frequent intervals? How current is his knowledge?

What is the breadth of knowledge in other areas, maybe outside of the discipline, in other fields that may be of relevance? How generally broadly educated, well rounded, is the individual?

Experience and Judgment

How much experience has the expert had in trying things out in the world, to see if the theory is actually applicable? There are theory smart experts with no practical experience. How much real world smarts, as opposed to academic smarts does the expert possess? How sound is the judgment, how much common sense is evident for this person?

Careerism

How much money, prestige, or careerism are associated with the discipline? Some professions are notorious for attracting careerists. People only interested in career advantage may give short shrift to serving others. They may sacrifice integrity in order to advance their careers. Also, see the incentives section.

Integrity

Personal integrity, including honesty and self-honesty, make for a better world. Some experts are mostly honest, and some are mostly dishonest. There are those who routinely deceive, lie, shill, con others and generally show sociopathic or even psychopathic behaviour. There are others who believe in fair dealing, and work hard to show personal integrity.

Theory blind experts

Experts may be unable to see things outside of the bounds of the theories espoused by the discipline. I call this being “theory-blind.”

The Incentives

The incentives and disincentives operating on the individual and on the collective are a huge factor in shaping the judgment of the expert. “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” — Attributed to Upton Sinclair

The cultural and social matrix

The cultural and social matrix influence the body of knowledge of the discipline and of the individual. Any expert will be enmeshed within various cultures and sub-cultures and their beliefs. These will invariably play a role in the shaping the judgment of the expert. These beliefs may include shared intellectual delusions and pathological thinking such as group-think. Peer pressure can bias thinking considerably. This will shape and constrain the experts view of the world.

An expert may routinely interact with colleagues, employers, employees, clients, family, governments, representatives of institutions, bankers, regulatory bodies and others. All of these will provide the matrix for the actions of the expert. All of these will influence the judgment of the expert, and not necessarily in a favourable fashion.

The Institutions

Governments, institutions, politicians, organizations, businesses, unions, trade associations, professional organizations, regulatory agencies, bureaucracies can all influence an expert of any discipline. These agencies are engaged in such things as making recommendations, promoting view points, promulgating rules and regulations, and setting standards of practice.

There is also the issue of institutional integrity, with some institutions being more or less corrupt, becoming top heavy with individuals who wish only to enrich themselves. This becomes an institutional failure.

The Discipline

I’m going to use the word discipline to describe any organized body of knowledge. Each discipline will have methods of practice that govern how things are done.

Flawed bodies of knowledge

The body of knowledge of any discipline will of necessity be incomplete. It will have ambiguities. It will have errors. It may have self-contradictions, or be otherwise flawed. There may be competing and even contradictory schools of thought in the discipline.

Some possibilities for a metric for soundness

You could systematically arrive at a metric for soundness for various disciplines with sufficient work. We could rate disciplines on a scale according to their successes and failures in results, explanation and prediction. The omniscient one could do it for us, if we have direct access.

How sound is the body of knowledge claimed for the discipline? Does it lead to consistent, correct and positive results? How concrete is the discipline: is it one with directly observable effects, able to be seen and repeated in real time (e.g., materials science versus string theory).

How useful is the existing body of knowledge of the discipline in explaining and controlling the real world? Some disciplines have a body of knowledge and methods which have repeatedly been shown to be sound, to be useful, to be reliable. Others seem to be hopelessly inadequate in making explanations, predictions, or recommendations. This is one key dimension: the soundness of the body of knowledge.

Flawed methods yield flawed knowledge. What are the methods of practice used in the discipline? How well do they work; do they give consistent, correct and positive results?

Some possibilities for a metric for complexity

How do we measure difficulty of a discipline, that is, what is required to master it in terms of hours spent in study, training and practice and the sort and amount of intelligence required for mastery? The omniscient one could tell us, but given that we do not have divine intervention, we might try making some rough estimate of the factors and of the requirements of some representative fields of study.

Some disciplines are hard, requiring a great great deal of highly specialized training in a broad range of difficult topics. The demands placed on the practitioner are great. I will call this dimension complexity.

Some disciplines involve a large number of things to be learned, they have great size. I will group this under complexity as well.

You could systematically arrive at a metric for complexity for various disciplines with sufficient work. We could rate disciplines on a scale according to the amount of schooling required, the ability required to complete the schooling, the number of textbooks, and presumably other factors.

Cross-classification

We can have a two dimensional classification. One dimension is soundness and the other is complexity. Let’s say we use 1 to 7 scale for each. We can situate various disciplines on this matrix so for instance electrical wiring is extremely sound, near the top, but the complexity relative to some other things is modest (I know this from experience having worked in that field). Something like mathematics is also extremely sound but also very complex, at least for most people. Something like astrology, according to my biases, is neither sound nor particularly complex. I suppose I could be wrong on both points. Take something like nutrition science. It’s not really all that sound as the science goes, as far as I can see. But it’s probably moderately complex.

Feel free to quarrel with my rankings. Don’t get your nose bent out of shape if you and I disagree.

Example Ratings of Complexity Versus Soundness 1

Soundness1 – abysmal2 – low3 – less thanso-so4 – so-so5 – more thanso-so6 – high7 – stellar
Complexity
1 – dead easyDowsing





2 – easy




Ditch digging
3 – not quite easyFortune tellingHomeopathy AstrologyPseudo-skeptics

Barbering
4 – so-so

JournalismChiropractic Nutrition Science

Any trade Carpentry Plumbing Electrical Auto body

5 – somewhat hard
Stock market forecastingArt criticism Music criticismEconomics Psychology

Investigative Journalism


6 – hard

Philosophy
Oncology Neuro- psychologyAnaesthetics Materials Science
7 – very hard





Mathematics

Summary

Even if we have a highly capable individual, their judgment may be impaired by social factors. However, if the foundations of their discipline are unsound, the role of the individual is mostly irrelevant. They may be the best in their field, but if the field has a body of knowledge that is too erroneous to be worth pursuing there will be no cigar.

You might have thought this discussion was only about experts. Well, most of the same factors apply to all of us. Maintain a healthy skepticism about the opinions of mankind. This includes media, prognosticators, consultants, scientists or just about any specialty – experts and non-experts of all persuasions. Also remember, you are part of mankind.

1I had the omniscient one give me this information, so if you have a problem with it, take it upstairs. More seriously, this is my very biased and ill-informed viewpoint, and to be taken as an illustration only.

2 thoughts on “Trusting the experts

Leave a Reply

Your email address will not be published. Required fields are marked *