LAST WEEK, the Washington Post revealed that in 268 trials dating back to 1972, 26 out of 28 examiners within the FBI Laboratory’s microscopic hair comparison unit “overstated forensic matches in a way that favored prosecutors in more than 95 percent” of the cases. These included cases where 14 people have since been either executed or died in prison.
The hair analysis review — the largest-ever post-conviction review of questionable forensic evidence by the FBI — has been ongoing since 2012. The review is a joint effort by the FBI, Innocence Project and the National Association of Criminal Defense Lawyers. The preliminary results announced last week represent just a small percentage of the nearly 3,000 criminal cases in which the FBI hair examiners may have provided analysis. Of the 329 DNA exonerations to date, 74 involved flawed hair-evidence analysis.
While these revelations are certainly disturbing — and the implications alarming — the reality is that they represent the tip of the iceberg when it comes to flawed forensics.
In a landmark 2009 report, the National Academy of Sciences concluded that, aside from DNA, there was little, if any, meaningful scientific underpinning to many of the forensic disciplines. “With the exception of nuclear DNA analysis … no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source,” reads the report.
There is one thing that all troubling forensic techniques have in common: They’re all based on the idea that patterns, or impressions, are unique and can be matched to the thing, or person, who made them. But the validity of this premise has not been subjected to rigorous scientific inquiry. “The forensic science community has had little opportunity to pursue or become proficient in the research that is needed to support what it does,” the NAS report said.
Nonetheless, courts routinely allow forensic practitioners to testify in front of jurors, anointing them “experts” in these pattern-matching fields — together dubbed forensic “sciences” despite the lack of evidence to support that — based only on their individual, practical experience. These witnesses, who are largely presented as learned and unbiased arbiters of truth, can hold great sway with jurors whose expectations are often that real life mimics the television crime lab or police procedural.
But that is not the case, as the first results from the FBI hair evidence review clearly show. And given the conclusions of the NAS report, future results are not likely to improve. What’s more, if other pattern-matching disciplines were subjected to the same scrutiny as hair analysis, there is no reason to think the results would be any better. For some disciplines the results could even be worse. Consider the examples below:
Bite marks, long accepted as criminal evidence, now face doubts about reliability. (David Duprey/AP)
ASSOCIATED PRESS
Indeed, some of the harshest criticism contained in the NAS report focuses on bite-mark evidence, and concludes that there is no scientific underpinning to the discipline. In a recent four-part series on bite-mark analysis, the Washington Post’s Radley Balko described how forensic odontologists — dentists who profess expertise in bite-mark analysis (and who are qualified as such by the American Board of Forensic Odontology) not only reject the NAS’s conclusion, but actively attack anyone who dares to criticize the field. Two examples: In 2013, ABFO leadership orchestrated an aggressive — and ultimately unsuccessful — plan to expel their own colleague, Dr. Michael Bowers, from membership within the American Academy of Forensic Sciences, which would have hamstrung Bowers from testifying against the practice in court. His crime: being a vocal critic of bite-mark “science.” In 2014, speaking at an ABFO dinner, Manhattan prosecutor Melissa Mourges, a strident supporter of bite-mark evidence, not only derided Mary Bush’s work, but also peppered her remarks with petty insults about Bush’s physical appearance.
Of course, as it is with hair analysis — and, really, any of the questionable forensic disciplines critiqued by the NAS — the utter lack of a scientific foundation has done nothing to keep bite-mark evidence out of the courtroom. To date, DNA has exonerated 24 individuals sent to prison on bite-mark evidence.
If only it were that easy.
While there is some actual science involved in bloodstain-pattern analysis — knowledge of the physics of fluids is helpful, as is an understanding of the pathology of wounds — the sheer number of variables involved in the creation of any given bloodstain makes reaching any definitive conclusion about the circumstances of its origin difficult at best. “The uncertainties associated with bloodstain-pattern analysis are enormous,” the NAS report concluded.
Yet for defendants, as with other forensic disciplines, the conclusions of a bloodstain “expert,” can mean the difference between living free or behind bars. The NAS report warns that while science supports “some aspects” of bloodstain-pattern analysis — whether blood “spattered quickly or slowly” for example — some experts “extrapolate far beyond what can be concluded.” This risk was powerfully demonstrated in the bizarre case of Warren Horinek, a former Fort Worth, Texas, police officer who, based solely on the conclusions of a blood-pattern expert, was convicted and sentenced to 30 years in prison for the 1995 murder of his wife — a death that the police, medical examiner, and prosecutor all concluded was actually suicide.
Horinek remains in prison.
Keith Cook looks over tire track impressions for comparison. (John Patriquin/Portland Press Herald/Getty)
Getty Images
There are several problems with this type of evidence — not least of which is the fact that while the evidence found at a crime scene remains static, fixed in time, shoe and tire wear is continuous, meaning in part that unless you can immediately match a shoe or tire to a crime scene, the potential probative value of that evidence could quickly be irretrievably lost. But more concerning is that there is no science demonstrating that any particular marks are actually unique, nor are there any standards for how many unique characteristics it takes to declare a match between object and evidence. There is “no defined threshold that must be surpassed, nor are there any studies that associate the number of matching characteristics with the probability that the impressions were made by a common source,” reads the NAS report. “Experts in impression evidence will argue that they accumulate a sense of those probabilities through experience, which may be true. However it is difficult to avoid biases in experience-based judgments, especially in the absence of a feedback mechanism to correct an erroneous judgment.”
Indeed, spurious shoe-print evidence offered by an FBI examiner helped to send Charles Irvin Fain to death row for the 1982 kidnapping, rape and murder of a 9-year-old girl in Idaho. According to the examiner, wear on Fain’s shoes matched wear patterns in shoe prints connected to the crime — and those wear patterns, the expert concluded, were created by a person with a particular gait. The perpetrator would “have to have the same characteristic walk as the individual who owned those shoes,” the expert testified.
DNA testing ultimately led to Fain’s release from prison in 2001 after spending 18 years on death row.
Importantly, fingerprints collected from crime scenes are often only partial prints, distorted, smudged, or generally “noisy,” as one group of investigators, seeking to formulate error rates for fingerprint examination, wrote last year. And that’s where problems can happen: Consider the case of Brandon Mayfield, the Oregon lawyer who was falsely accused of participation in the 2004 Madrid, Spain train bombings based on a fingerprint collected from a bag containing detonation devices. The FBI later admitted it bungled the print match.
Fortunately, there are ongoing efforts underway within the discipline’s community of experts to validate forensic fingerprint examinations. Jennifer Mnookin, a UCLA law professor and lead investigator into fingerprint error rates, says that leaders in the field have begun to embrace the emerging “research culture” that the area is taking on. “At this point it’s not that the work is done,” she says. “It isn’t. But compared to bite marks … to handwriting [analysis], there is now a growing body of research looking at these questions [of validity and reliability] in a way that didn’t exist 10 to 15 years ago.”
Whether all of the state cases will ever be identified let alone reviewed, remains to be seen.
For Timothy Bridges, the stakes couldn’t be much higher. Bridges was convicted and sentenced to life in prison for the beating and rape of an 83-year-old woman in Charlotte, North Carolina, in the spring of 1989. The victim (who died of unrelated causes before Bridges trial), variously described her attacker and denied that she was raped. Ultimately, Bridges, who had wavy shoulder-length hair — which is how the victim once described her attacker — was charged with the crime. There was no DNA to connect Bridges to the crime and he was not a match for a bloody palm print found at the scene (that print was never matched to anyone). But there were two hairs collected that an FBI-trained examiner testified not only were “likely” Bridges, but also that there was a very low chance they could belong to anyone else: The “likelihood of two Caucasian individuals having indistinguishable head hair is very low,” expert Elinos Whitlock testified — the very sort of language unsupported by science and found in the faulty cases identified in the current FBI review.
Bridges appealed his conviction, arguing in part that there was no scientific basis to Whitlock’s testimony. In 1992, the state appeals court disagreed: “We find no reversible error,” the court ruled, concluding that testimony by a “properly qualified witness on hair identification” was admissible.
Bridges is currently seeking a new trial and the state is reportedly reviewing the matter.
Photo: Showtime


A couple of observations. Fingerprint analysis depends on what are termed points of identification. As I understand it, federal rules require 9 points of identification for an examiner to declare a match between the print recovered from a crime scene and and one obtained from a suspect. However, there is, apparently, a certain amount of subjectivity involved in determining what a point of identification is. For instance, in the case of Mayfield, the examiner identified more then 9 points of identification in common between what was collected at the crime scene and Mayfield. However, during the FBI’s investigation, it was determined that there were only 7 points of identification actually present, insufficient under the federal standard to declare a match. As I understand it, 8 are required in Virginia and 12 in California. I once read that the FBI conducted a study a decade or more ago in which the AFIS data base was examined. It turned out that a number of matches between different individuals were found at the 9 point level.
It is also interesting to note that the subject of hair identification came up during the OJ Simpson trial 20 years ago in which an FBI examiner, Douglas Dietrich testified. He was not allowed to declare a match between samples taken from Simpson and samples found at the crime scene. All he was allowed to say was that the crime scene samples were left by an African man and that they were consistent with the samples taken from Simpson. Under cross examination, he admitted that such evidence was weak.
As a crime scene reconstruction and blood pattern expert, I pretty much agree with what has said so far.
Very interesting exploration of the flaws in modern forensics, but I wonder if it stops a little short in it’s examination of the topic in general. Specifically, if we accept that forensics is an imperfect science and that it is unlikely, in the short term at least, then what controls could we put in place to provide some effective safeguards for those navigating the justice system? Here are just a small handful of suggestions – it would be interesting [hilarious] to discover what prosecutors feel about them:-
1. Prohibit the Death Penalty for Forensic-Based Convictions
In those cases where either the convicting evidence, or a significant portion of the convicting evidence comes from forensic evidence, disallow the use of the death penalty. Through this mechanism, if there has been a miscarriage of justice, then at least the accused has a chance of clearing their name before it is too late.
2. Require Positive Testing
Demand that every forensic examiner who wishes to testify in court be given a series of controlled tests in which they must prove their acumen via a rigorously applied scientific method. For example, taking the “hair” case as an illustration, why not give a so-called hair specialist a hair sample and then say 1,000 additional samples and have them identify the correct match. Conduct the test with independent observers and then, when the expert is called upon to testify, make it a legal requirement to have their accuracy read out in court, so that the judge[s], jury members and legal teams all know precisely how capable and expert the witness really is. This will help to eradicate sham expert-witnesses from the justice system.
3. Implement A Liability Requirement
The justice system imprisons people because the withdrawal of liberty is a punishment. Yet in the event that a forensic-based conviction is overturned, there is no down-side for the so-called experts that helped lead to the original conviction. So the so-called expert is in a win-win position, because as we read elsewhere, so many of these people make a profession of being experts and as such can command hefty fees just for appearing in court. So. Firstly, make all experts personally, legally liable for any miscarriage of justice they contribute to. Require them to take out hefty insurance policies that will pay out against any case in which their evidence was subsequently proven to be false. Make the “expert” at least partially and personally responsible for the restitution due to anyone wrongly convicted.
4. Revocation of “Expert” License
Set things up so that anyone wishing to testify at court as an “expert” must first be awarded a license to do so by the court. Have this set up in such a way that it is not a burden for a one-off case in which a specialist is being called in to testify for either side in a case, but such that, in the event of a proven mis-carriage, the failure of the “expert” will count against them in a meaningful way. Prevent any so-called expert who is found to have made more then “n” mistakes, I would suggest 3 mistakes being a reasonable cut-off, be automatically barred, for life, from giving any more expert evidence in court.
None of the above sort-of remedies are ideal. Rather, like the forensics process itself, flawed in a range of ways. But each is set out in a very careful way, which is that none will prove to be a burden to a legitimate expert providing valid testimony from within their respective area of expertise. However, each will help stop the charlatans and the hustlers from making money at the expense of innocent people.
A good justice system learns from it’s mistakes, it evolves, gets better with time.
A good way to judge the effectiveness of a justice system is to test that evolutionary capability. I wonder if the various justice systems of the world would be willing to acknowledge and mitigate the risks caused by their imperfections? Sadly, I doubt it.
If fingerprints are so unreliable, when will the USA, Japan, and South Korea stop fingerprinting travellers at airports?
How is the Forth Worth officer still in prison if everyone involved in putting him there concluded the murder he was tried for was actually a suicide?
America now needs a comedy show based on forensics. We’ve had far too many of the other. We also need a comedy series about the President and Congress. Laughing is better than crying.
DNA testing is not perfect. Only a small part of DNA is actually tested so in some ways it is like a fingerprint. If it is properly tested it can be 100% positive but the implementation is poor. See http://www.dailytech.com/Sweeping+Database+Searches+Call+Infallible+DNA+Evidence+Into+Question/article12454.htm
This article is overly pessimistic. Science doesn’t claim to be perfect, but it learns from its mistakes and moves on. The FBI now realizes that if you are going to plant evidence at a crime scene, DNA is the way to go.
I just wonder how many innocent people have been executed in the last 50 years with convictions based on junk science.
Its good the truth has come out now.there is also danger in dna unless quacks are not allowed to administer them.in many parts of the world obscure individuals with doubtiful forensic training or none at all have taken control of the so call labs from inception to manning and yet the courts rely on their ‘expertise’ to convict .please release all those people convicted by the evidence of the fbi ‘hairstylist’expertise.the science needs to be forensically upgraded to dna analysis.
I wonder just how bad some of those techniques really are? I wonder if some type of experiment was set up, how often the hair examiners could make the correct match if given a large enough sample? I am betting if this was done in some type scientific double blind test, the results would be piss poor. The good news is that since we have DNA testing nowadays, I would think that hair analysis would be a thing of the past. Or at least maybe it could be used as an initial step, but then DNA would always be used for confirmation? That will help in the future, but it means very little to people who were wrongly convicted in the past based on bad science.
FBI’s Amerithrax investigation is a great example of FBI investigative hocus pocus, though it was successful inasmuch as it managed to help to conceal the identity of the criminals responsible, which may well have been the goal from the outset.
McClatchy: FBI’s genetic tests didn’t nail anthrax killer, GAO says
http://www.mcclatchydc.com/2014/12/19/250715_fbis-genetic-tests-didnt-nail.html?sp=/99/200/111/&rh=1
NYT: Former F.B.I. Agent Sues, Claiming Retaliation Over Misgivings in Anthrax Case
http://www.nytimes.com/2015/04/09/us/ex-fbi-agent-claims-retaliation-for-dissent-in-anthrax-inquiry.html?_r=1
Excellent article Ms. Smith.
I would be interested in follow-up research findings pertaining to the validity of forensic science in criminal investigations.
Please write more on this topic as information is released.
Very interesting and informative article. I do however, have one thought to add. The article, most likely based on the scientific studies referenced, comes to conclusions like, “…soles and tire treads are decidedly not unique, since both are mass produced…”
Curiously that conclusion is simultaneously true and potentially misleading.
Uniqueness is a function of the sample size of any set. If I compare a tire pattern mark left at the scene of a crime to the hundreds of millions of tires around the country the probability that tire is unique among the set gets very small, that is to say it becomes more possible to find another tire with the same pattern the larger I make the sample size. The same thing is true with DNA. Note the confidence factor assigned to DNA when presented in court. A DNA expert might say that the DNA match is 1 in 10 billion. Given there are < 10 billion people on the planet we can conclude the probability that another person exists with the same DNA is low. If the world had 100 billion people in it then a 1 in 10 billion chance means 10 people could match.
Most cases however, do not compare things like tire pattern marks or DNA to the entire sample size of all tires or all people. Good old police work narrows the sample size to a few prime suspects. When you look at one of your suspects cars and find a match, that can be pretty compelling data.
There are other parts of the article that I think are much less misleading. It's very true that the number of variables involved in blood spatter formation is so high as to make anything but the most rudimentary conclusions dubious at best.
The point is always to remember what you math teacher taught you, statistics are never wrong but you can interpret them to mean almost anything.
Fascinating. Scientism in action.
On the mention of DNA reliability, did y’all catch the full story about the discovery of Richard III’s bones under a parking lot a couple of years back?
The discoverer, Philippa Langley, located the remains essentially from ‘a hunch’ (yes); DNA testing then identified the skeleton. It shows that there’s still a lot more to the processes of modern investigation than (pseudo)science.
http://snapjudgment.org/how-one-womans-secret-discovery-under-parking-lot-changed-500-years-history
Very nice article – but you omitted a significant example: police dogs do not detect drugs and are unreliable detectors of explosives and everything else. 2011 Study: “Handler beliefs affect scent detection dog outcomes” http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3078300/
Independent Workers Party of Chicago
But they’re VERY good at detecting sandwiches in airports.
But hey, on the bright side, soon we will all be subject to mamdatory DNA collection anyway. So there’s that.
/s
No one will ever be accidentally wrongfully convicted again!
Unless their DNA was planted at the scene… or they lie… or they screw up…
Sounds like a sh!t storm is headed their way.
Testing that the science actually is science seems like something somebody would have thought to do long ago.
So, you figure two plus years studying each type, assuming they can’t do more than one thing at a time, in a short ten years or so a bunch of innocent people in prison can be exonerated. Maybe add a few years for funding approval, and a few more for winding the cases through the courts.
Actually, I saw no mention if any action was going to be taken. Is anybody even asking for funding to study the rest?
If I had to guess though, I’d bet that fingerprint matching ends up with a higher probability of accuracy than the others.
But it will be good to know if that probability is 98% or 75%… or whatever… especially since everybody acts like it’s 100%.
Oh my God!!!??? Anyone at any time could be convicted of a crime they didn’t do!!?? What should be done to these
“liars under color of authority” using the “CSI” fraud with things!?
It’s all very well to focusd on the grudging uptake of discredited science. Id be more interested though in the realizatiomn that the prosecution csn and does destroy exculpatory evidence which would aquit a defendant…
And why aren’t those prosecutors in the dock, somewhere, under indictment? Is the judicial system a farce? Maybe someday, when all the power brokers are at the bottom of the sea, pigs will once again be able to fly, and tomorrows will be anticipated, todays enjoyed, and yesterdays revered.
So prime time TV not only rehabilitates the reputation of the police and legal system — and obscures the class function of these coercive institutions of the state — and disseminates ruling class ideology and social theory — but also misinforms and misleads tens of millions of civilians into having unwarranted confidence in and deference toward the authority and legitimacy of these institutions?
Gramsci was right.
Excellent points. Grimasci posited ‘Cultural Hegemony’…
the term ‘liberal media’ is a favorite trope of dissemblers intent on masking the inverted truth