In the days after Luigi Mangione’s arrest at a McDonald’s in Altoona, Penn., New York Police Department (NYPD) Commissioner Jessica Tisch publicly claimed that the gun recovered on Mangione at the time of his arrest “matched” shell casings recovered from the crime scene where health insurance CEO Brian Thompson was fatally shot in New York City.¹

Media companies quickly reported these claims – that cartridge casings recovered at the scene of the shooting “matched” the test fires from Mangione’s gun – even though this type of opinion is not supported by science. The sharing of this information without independent investigation of the reliability of the evidence is not entirely surprising given the misconceptions of forensic evidence portrayed in movies, TV, and public imagination.

As a public defender for over a decade, I have spent much of my career litigating forensic evidence and fighting against the misuse of science and surveillance both in individual criminal cases and at the systemic level. I can speak firsthand to the struggle public defenders face in the courtroom combatting the misconceptions held by judges and juries alike that forensic evidence is objective, reliable, and infallible. This is why it is so troubling to see these claims of a “match” in Mangione’s case perpetuated by not only the media but defense lawyers as well.²

Reporting these forensic opinions as facts dangerously overstates the reliability and strength of the pattern-matching evidence and is uniquely misleading to jurors.³ There are many reasons to question the validity of the firearms (and fingerprint) comparisons in Mangione’s case.⁴

Rather than accept the NYPD’s opinions at face value, lawyers can use this case to correct common misconceptions about forensics’ infallibility and to debunk the myth of the “match.” This article addresses some of the scientific critiques of the pattern-matching methods with the hope of making them part of the public discourse.

What Is Pattern Matching?

Pattern matching, also known as feature-comparison methods, refers to a wide variety of forensic disciplines, such as fingerprints, firearms and toolmarks, shoeprints, handwriting, and bite marks, that aim to determine whether an evidentiary sample (for example, from a crime scene) is or is not associated with a potential source sample (for example, from a suspect) based on the presence of similar patterns, impressions, features, or characteristics in the sample and the source.⁵

Firearms examination, which is a subset of toolmark comparisons more broadly, seeks “to determine whether ammunition is or is not associated with a specific firearm based on toolmarks produced by guns on the ammunition.”⁶ For example, was a spent bullet or cartridge casing recovered at the scene of a shooting fired from the specific gun recovered in the suspect’s car? Were two pieces of spent ammunition fired from the same gun or different guns? A firearms examiner attempts to answer these questions by visually comparing the scratches and marks on the surfaces of the spent bullets or cartridge casings.

The basic premise of firearm and tool marks is the following: When the harder metal components of the gun come in contact with the softer metal surface of the cartridge during the firing process, the gun’s metal components might stamp or scratch marks onto the surface of the cartridge.⁷

There are three different categories of markings: class characteristics, subclass characteristics, and so-called individual characteristics. A firearms examination begins with a comparison of class characteristics, which the Association of Firearm and Tool Mark Examiners (AFTE) defines as “measurable features of a specimen which indicate a restricted group source.”

Class and “Individual” Characteristics

Class characteristics result from design factors and are determined before manufacturing begins.⁸ Class characteristics are shared by guns of the same make and model. They are measurable features predetermined by the manufacturer and include caliber, firing pin shape, the number of lands and grooves in the barrel, and the direction of the twists.⁹ If a firearms examiner believes, based on examination of the class characteristics of two pieces of ammunition, that the pieces of ammunition were fired by guns with different characteristics, the firearms examiner will render an elimination opinion that the two pieces of ammunition were fired from different guns. If the class characteristics are similar, however, the examiner compares the so-called individual characteristics.¹⁰

Individual characteristics are random scratches acquired during the firing process by imperfections on the inside of the gun produced during the manufacturing process or from normal wear and tear.¹¹ These imperfections leave marks on the bullet or casing as it leaves the gun and are believed to be unique to a particular gun. However, there are no statistics about how common or rare these markings are, which begs the question: can these characteristics be considered unique?

Examiners visually compare the random markings on bullets or cartridge casings under a light-comparison microscope, which is essentially two microscopes connected by one optical bridge, to subjectively determine whether they believe the two pieces of ammunition were fired from the same gun. There are no standards about the quality or quantity of features needed to declare a match. Rather, the examiner is merely looking for “sufficient agreement” among individual characteristics, which is defined as the level of agreement that “exceeds the best agreement demonstrated between tool marks known to have been produced by different tools and is consistent with agreement demonstrated by tool marks known to have been produced by the same tool.”¹²

“Sufficient agreement” is relative to the individual examiner’s own experience (without the benefit of ground truth being known – such as when the truth of whether two samples came from the same source or different source is known, as it would be in a study, or encountered in training). In essence, sufficient agreement is a standard akin to beauty being in the eye of the beholder. It stands to reason, then, that when examiners look for those “unique” characteristics, what they really mean is, unique as far as they know. The subjectivity of the “sufficient agreement” standard is further confounded by the third category of markings: subclass characteristics.¹³

Subclass Characteristics

Subclass characteristics are unintended marks caused by unknown defects or imperfections in the manufacturing process that affect an entire batch or manufacturing run.¹⁴ Subclass characteristics are somewhat of a red herring for the discipline: These markings can be hard to distinguish from randomly acquired scratches and are easily mistaken for individual characteristics.¹⁵

There are no standards to guide differentiating between subclass and individual characteristics and “simply no telling when a subclass arises or how many firearms are in a subclass…. Examiners in the field would be utterly unaware of subclass characteristics present in a batch or batches of firearms.”¹⁶ The danger is that in declaring a match or making an identification, a firearms examiner might rely on markings the examiner believes are individual characteristics when in fact the markings are subclass characteristics shared by an unknown number of guns and are not unique.

In the absence of statistics about the uniqueness or rarity of features and without standards about the quality and quantity of features needed to declare a match, firearms comparisons completely depend on examiners’ subjective judgments.

Reliability and Inconclusive Debate

Firearms examination evidence has been routinely admitted in courts for many decades. But the trustworthiness of such evidence has not been conclusively established.

For decades, the reliability of firearms examiners’ claims that they can visually compare microscopic marks on fired bullets or cartridge casings to determine if the spent ammunition was fired from the same gun or different guns was not meaningfully evaluated.¹⁷ In 2008, the National Resource Council Ballistic Imagining Report revealed that the basic premises of uniqueness and reproducibility underlying the field of firearms and tool marks were yet to be empirically established.¹⁸

In 2009, the National Academies of Science found that studies conducted to understand repeatability and reproducibility for firearms examination were insufficient, and in 2016, the President’s Council of Advisors on Science and Technology (PCAST) concluded that the scientific validity of firearms examination had not been demonstrated.¹⁹

PCAST instructed that for subjective methods that rely heavily on human judgments, the only way to demonstrate validity is with accuracy rates established through appropriately designed black box studies. “Nothing – not training, personal experience nor professional practices – can substitute for adequate empirical demonstration of accuracy.”²⁰

Flawed Study Designs. PCAST’s review of the existing validation research uncovered two fundamental design flaws among the studies. The first was that many studies used a closed-set design that did not allow for accurate error-rate calculations because there was no way to know how many comparisons each study participant (firearms examiner) made.

Participants could leverage interdependencies to complete these studies, which also made it easier than open-set pairwise studies.²¹ For the latter, study participants compared one sample set at a time and reached a conclusion before moving on to the next sample set.²² In theory, a firearms examiner should answer either match or identification – the samples were fired by the same gun; or elimination or exclusion – the samples were fired by different guns.

However, study participants were permitted to provide a third category of answer: inconclusive. PCAST revealed that these answers were mathematically treated as correct answers when calculating the error rate, and thus, artificially lowered the reported rate of error.²³ PCAST suggested an alternative way of treating inconclusive answers, which was to drop them from the calculation altogether.²⁴

This study flaw, which came to be known as “the inconclusive response problem,” was the source of much scientific debate and the focus of some successful litigation regarding the admissibility of evidence from firearms examiners.²⁵ Scientists did not necessarily agree as to how the inconclusive answers should be treated, but those outside the community of firearms examiners agreed that the way the studies treated them was an error and resulted in the reported error rates being inaccurate. The inconclusive-response debate became the first of an evolving body of scientific critiques. In many ways, PCAST helped to put the flaws that plagued firearms examination validation research on the radar of mainstream scientists and academics. The scientific critiques only grew from there.

Critiques to Date

After PCAST, some scientists took a closer look at the error-rate studies’ design.²⁶ The discourse regarding the study design and statistical calculations became more robust, as additional independent scientists, statisticians, and researchers have examined the research and published their own scholarship on the subject.²⁷

A recent contribution to this body of scholarship is an article that provides an overview of and builds on the scientific critiques to date. Cuellar et al. analyzed each of the major flaws, which they list in Table 1 in their article, and synthesized the scientific criticism of the error-rate studies.

These flaws include inadequate sample size, nonrepresentative sample, nonrepresentative materials, nonrepresentative testing conditions and environment, inconclusive responses treated as correct or ignored, invalid or nonexistent uncertainty measurements for error rates, and missing data.²⁸ These flaws mean that the reported error rates in the studies are not generalizable to the discipline as a whole or that they are otherwise inaccurate.

As a result, there is currently no known error rate for the discipline. And “without appropriate estimates of accuracy, an examiner’s statement that two samples are similar – or even indistinguishable – is scientifically meaningless: it has no probative value, and considerable potential for prejudicial impact.”²⁹ In other words, without a known, accurate error rate, an examiner’s opinion that a piece of ammunition came from a specific gun is meaningless.

Jurors do not know how much weight to give to such evidence. Without empirical support, a firearms examiner’s opinion is baseless speculation or the very ipse dixit of the expert that the U.S. Constitution prohibits.³⁰

Recent research also suggests low rates of repeatability and reproducibility.³¹ One study found that examiners disagreed with themselves 20% of the time when examining the same sample and disagreed with each other 30-45% of the time.³² It follows that if the method is not repeatable and reproducible, it is not scientifically valid. Simply put, in the area of firearms examination, “match” is an opinion without empiric support.

Implications for Cases

In response to recent research and well-established critiques from the broader scientific community, Maryland and the District of Columbia no longer allow testimony in criminal trials that there was a “match” or identification of one specific gun.³³ Rather, the examiner can only explain that the markings on the cartridge casings are consistent or are inconsistent.³⁴ Other courts have imposed limitations on or totally precluded testimony beyond comparison of class characteristics,³⁵ which a Cook County Circuit Court found to be more prejudicial than probative and thus should be inadmissible.³⁶

These admissibility decisions might encourage lawyers (and the public at large) to also question the validity of the “match” opinions being reported in Luigi Mangione’s case and to not immediately accept the NYPD’s forensic opinions. An additional reason to take pause about the reliability of the examination and opinions formed in Mangione’s case is that the gun in question is a ghost gun.³⁷

The basic premise of firearm and tool mark identification is rooted in the manufacturing process. Ghost guns are assembled from parts printed using a 3D printer.³⁸ These guns do not go through the normal manufacturing process.

It stands to reason that the existing research is not applicable for this kind of firearm, and the examiners’ training and experience might not be either. Recent research has suggested that existing validation studies do not have sufficiently difficult test samples for the error rates, notwithstanding other study design flaws and statistical errors, to be generalizable for the field using traditionally manufactured guns, and that existing research does not adequately test performance on close nonmatches.³⁹

These factors make the applicability of existing research even more tenuous for ghost guns. Regardless of an individual’s existing beliefs regarding the validity of firearms examination, there is reason to question its use in the case against Mangione.

Conclusion

The prosecution of Luigi Mangione for the murder of Brian Thompson offers an opportunity to examine the role of forensic evidence in criminal cases and its portrayal in the public imagination. The challenges surrounding pattern-matching methods, compounded by cognitive biases and the lack of empirical validation, underscore the need for a paradigm shift in how such evidence is understood and used.

Forensic evidence should be held to the same rigorous scientific standards as other disciplines, ensuring accuracy, reliability, and transparency. Because of the significance of firearms evidence in the murder of Thompson, the prosecution of Mangione has the potential to reshape public perceptions and promote the development of scientifically grounded practices. It is imperative to challenge entrenched myths and advocate for reforms that safeguard the integrity of justice. There are many reasons to challenge the reliability and thus admissibility of the pattern-matching “match” opinions in the case, which is a fight backed by science.

Also of Interest

Forensic Justice Institute 2025

Revisit the presentation by author Emily Prokesch at the Forensic Justice Institute 2025. The live, virtual Institute, co-produced by the State Bar of Wisconsin PINNACLE® and the Center for Integrity in Forensic Sciences, was held on Jan. 24, 2025, but webcast replays are scheduled throughout the year.

Nothing but the truth. Accurate and reliable forensic science plays a critical role in the pursuit of justice. The Forensic Justice Institute empowers criminal justice professionals to deepen their understanding of scientific evidence and its application in the courtroom.

Benefit from the unbiased, straightforward insights of nationally acclaimed speakers on trailblazing topics, including:

• Recognizing and avoiding common fallacies in interpreting probabilities
• Debunking misconceptions about DNA evidence
• Handling evidence in the early stages of scientific development
• Building a strategy for managing digital evidence
• Addressing ethical responsibilities in plea negotiations
• Analyzing flaws in forensic blood samples

Gain the knowledge to ensure your next case stands on a foundation of science fact – not science fiction.

For a schedule of webcast replays and to register, visit https://marketplace.wisbar.org/products?pid=IN0235.

Learn the Nuts and Bolts of More Effective Criminal Defense

Whether you need advice on organizing your files, negotiating a plea bargain, or filing motions before trial, make the Wisconsin Criminal Defense Manual your guide. Whatever your level of experience, the Manual can help you advance your practice and obtain the best possible outcomes for your clients. When you integrate the master information list and initial interview guide into your practice, the Manual will become one of your main criminal defense resources.

The Manual is designed to make you more efficient by giving you extensive overviews of the law, checklists for completing vital procedures accurately and completely, commentary and instructions for filing necessary forms, and information on handling public defender cases.

Both the print and the Books UnBound versions of the Manual include a complete set of fillable, unannotated forms via download.

According to reader surveys, the Manual is one of the State Bar of Wisconsin’s most valued resources. Join the many defense attorneys who rely on the Wisconsin Criminal Defense Manual for sound advice and guidance.

https://marketplace.wisbar.org/products?pid=AK0041

Endnotes

1 Jessica Parker & Nadine Yousif, Luigi Mangione Fingerprints Match Crime-Scene Prints, Police Say, BBC (Dec. 11, 2024), https://www.bbc.com/news/articles/ceqlj130rxpo.

2 Corinne Ramey & James Fanelli, Luigi Mangione’s Best Defense Strategy Might Be to Avoid Arguing Over Guilt, Wall St. J. (Dec 16, 2024), https://www.wsj.com/us-news/law/luigi-mangione-insanity-defense-ceo-shooting-59ee407b?st=c8pum9&reflink=article_imessage_share [behind paywall for some readers]; Dalia Faheid et al., The Gun CEO Killing Suspect Luigi Mangione Had upon Arrest Matches Shell Casings from the Crime Scene, Police Say, CNN (Dec 11, 2024), https://www.cnn.com/2024/12/11/us/unitedhealthcare-ceo-brian-thompson-shooter-wednesday/index.html.

3 President’s Council of Advisors on Sci. & Tech., Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-comparison Methods, Executive Office of the President of the United States (2016) [hereinafter PCAST].

4 Maria Cuellar et al., Methodological Problems in Every Black-Box Study of Forensic Firearm Comparisons, L., Probability & Risk vol. 23, issue 1 (2024), https://doi.org/10.1093/lpr/mgae015; Graham Rayman, NYPD Fingerprint Error Sparks Demand for Independent Probe of Crime Lab, Case Reviews, N.Y. Daily News (Oct. 22, 2023), https://www.nydailynews.com/2023/10/22/nypd-fingerprint-error-sparks-demand-for-independent-probe-of-crime-lab-case-reviews/.

5 Id.

6 Rayman, supra note 4.

7 StrengtheningForensic Science in the United States: A Path Forward (Washington, D.C.: National Academies Press 2009) [hereinafter Strengthening Forensic Science].

8 Ass’n of Firearm & Tool Mark Examiners, Glossary (6th ed.), https://afte.org/uploads/documents/AFTE_Glossary_Version_6.091922_FINAL_COPYRIGHT.pdf.

9 StrengtheningForensic Science, supra note 7.

10 PCAST, supra note 3; Strengthening Forensic Science, supra note 7.

11 Cuellar et al., supra note 4; Rayman, supra note 4.

12 Ass’n of Firearm & Tool Mark Examiners, Theory of Identification as It Relates to Toolmarks,AFTE J. 43(4):287 (2011).

13 People v. Ross, 129 N.Y.S.3d 629 (2020).

14 R. Nichols, Subclass Characteristics: From Origin to Evaluation, AFTE J. 50(2): 68 (2018).

15 Id.

16 Ross, 129 N.Y.S.3d 629.

17 PCAST, supra note 3.

18 National Academy of Scis., Ballistic Imaging. Technical Report (Washington, D.C.: National Academies Press 2008).

19 PCAST, supra note 3; Strengthening Forensic Science, supra note 7.

20 UnitedStates v. Tibbs, No. 2016-CF1-19431, 2019 WL 4359486 (D.C. Super. Sep. 5, 2019).

21 PCAST, supra note 3.

22 Id.

23 Id.

24 Id.

25 Id.

26 Cuellar et al., supra note 4; Ross, supra note 13; Tibbs, No. 2016-CF1-19431; Alan H. Dorfman & Richard Valliant, Inconclusives, Errors, and Error Rates in Forensic Firearms Analysis: Three Perspectives, 5 Forensic Sci. Int’l: Synergy 100273 (2022); Itiel E. Dror & Nicholas Scurich, (Mis)use of Scientific Measurements in Forensic Science, 2 Forensic Sci. Int’l Synergy 333 (2020).

27 Dorfman & Valliant, supra note 26; Dror & Scurich, supra note 26; Hofman et al., Treatment of Inconclusives in the AFTE Range of Conclusions, L., Probability & Risk, 19: 317 (2020); Amicus Brief in Support of Appellant, Abruquah v. Maryland (see Abruquah v. Maryland, 296 A.3d 961 (Md. 2023)); Thomas D. Albright, How to Make Better Forensic Decisions, PNAS, vol. 119, no. 38 (2022), https://www.pnas.org/doi/10.1073/pnas.2206567119; Faigman et al., The Field of Firearms Forensics Is Flawed, Sci. Am. (May 25, 2022).

28 Cuellar et al., supra note 4.

29 Id.

30 PCAST, supra note 3.

31 General Elec. Co. v. Joiner, 522 U.S. 136 (1997).

32 Keith L. Monson et al., Accuracy of Comparison Decisions by Forensic Firearms Examiners, J. Forensic Sci. 68(1): 86 (2022). https://doi.org/10.1111/1556-4029.15152; Brandon A. Best et al., An Assessment of the Foundational Validity of Firearms Identification Using Ten Consecutively Button-Rifled Barrels, 54 AFTE J. 28 (2020).

33 Monson, supra note 32.

34 Abruquah, 296 A.3d at 998; Gardner v. United States, 140 A.3d 1172 (D.C. Ct. App. 2016).

35 Ross, 129 N.Y.S.3d 629; Tibbs, No. 2016-CF1-19431.

36 People v. Winfield, Nos. 15CR14066-01, 16CR13911-013, 17CR02660-01 (Ill. Cir. Ct. Cook Cnty. Jan. 11, 2023) (granting outright exclusion). Rather than seeking appeal of the decision, the state made a motion to reargue, which was briefed and argued before Judge Hooks, who orally denied the motion. Before Judge Hooks could issue a written decision, Judge Hooks left the bench, and a different judge granted the state’s motion and found that it was error to grant the Frye hearing in the first place. See alsoNew York v. Lewis, Ind#1717/2020 (N.Y. Sup. Ct. Feb. 6, 2023) (following Ross); United States v. Briscoe, 703 F. Supp. 3d 1288 (D.N.M. 2023) (holding that field lacks general acceptance and precluding opinion testimony regarding source); Oregon v. Moore, No. 18CR77176, at *26, *29 (Or. Cir. Ct. Aug. 8, 2023) (holding that field lacks general acceptance and precluding opinion testimony regarding source); United States v. Shipp, 422 F. Supp. 3d 762, 782-83 (E.D.N.Y. 2019) (holding that field lacks general acceptance); United States v. Adams, 444 F. Supp. 3d 1248, 1266-67 (D. Or. Mar. 16, 2020) (holding that field lacks general acceptance and precluding opinion testimony regarding source); Missouri v. Goodwin-Bey, No. 1531-CR00555-01 (Mo. Cir. Ct. Green Cnty. Dec. 16, 2016) (restricting testimony to “cannot exclude” language).

37 Nick Penzenstadler, How a CEO’s Murder Offers a Dark Glimpse into the Anonymous World of 3D-printed Guns, USA Today (Dec. 19, 2024), https://www.usatoday.com/story/news/nation/2024/12/19/luigi-mangione-ghost-gun-3d-printed-gun-design/77069151007/.

38 Id.

39 Richard E. Gutierrez & Emily J. Prokesch, The False Promise of Firearms Examination Validation Studies: Lay Controls, Simplistic Comparisons, and the Failure to Soundly Measure Misidentification Rates, J. Forensic Sci. 69(4):1334 (July 2024), https://pubmed.ncbi.nlm.nih.gov/38684627/; Richard E. Gutierrez, Bowling with Bumper Rails: How Firearms Examiners Have Duped the Courts and Generated Low Error Rates Only by Avoiding Challenging Comparison, UC L.J. 75:1535 (Aug. 2024).