JRA Collection

Permanent URI for this collection

https://hdl.handle.net/20.500.11990/10170

A collection of resources supplementing course offerings in the Justice Research Academy.

Browse

Now showing 1 - 20 of 28

A systematic review of propensity score methods in the social sciences
(Taylor & Francis Group, 2011) Thoemmes, Felix J.; Kim, Eun Sook
The use of propensity scores in psychological and educational research has been steadily increasing in the last 2 to 3 years. However, there are some common misconceptions about the use of different estimation techniques and conditioning choices in the context of propensity score analysis. In addition, reporting practices for propensity score analyses often lack important details that allow other researchers to confidently judge the appropriateness of reported analyses and potentially to replicate published findings. In this article we conduct a systematic literature review of a large number of published articles in major areas of social science that used propensity scores up until the fall of 2009. We identify common errors in estimation, conditioning, and reporting of propensity score analyses and suggest possible solutions. [Author Abstract]
Advances in propensity score analysis
(Statistical Methods in Medical Research, 2020-03) Austin, Peter C.
An introduction to propensity score methods for reducing the effects of confounding in observational studies
(Multivariate behavioral research, 2011-05) Austin, Peter. C.
The propensity score is the probability of treatment assignment conditional on observed baseline characteristics. The propensity score allows one to design and analyze an observational (nonrandomized) study so that it mimics some of the particular characteristics of a randomized controlled trial. In particular, the propensity score is a balancing score: conditional on the propensity score, the distribution of observed baseline covariates will be similar between treated and untreated subjects. I describe 4 different propensity score methods: matching on the propensity score, stratification on the propensity score, inverse probability of treatment weighting using the propensity score, and covariate adjustment using the propensity score. I describe balance diagnostics for examining whether the propensity score model has been adequately specified. Furthermore, I discuss differences between regression-based methods and propensity score-based methods for the analysis of observational data. I describe different causal average treatment effects and their relationship with propensity score analyses. [Author Abstract]
Best practices: Avoiding failures of Implementation - Lessons from process evaluations
(Bureau of Justice Assistance, 2009) Cissner, Amanda B.; Farole, Donald J.
The purpose of this paper is to identify lessons that will help practitioners and policymakers anticipate, recognize, and resolve problems that may arise when implementing new projects or attempting to replicate existing ones in new settings. It has been said that learning from your own mistakes is smart; learning from the mistakes of others is wise. We hope that this report can help smart criminal justice innovators become wise. [Author Abstract]
Combining propensity score matching and group-based trajectory analysis in an observational study
(American Psychological Association, 2007) Haviland, Amelia; Nagin, Daniel S.; Rosenbaum, Paul R.
In a nonrandomized or observational study, propensity scores may be used to balance observed covariates and trajectory groups may be used to control baseline or pretreatment measures of outcome. The trajectory groups also aid in characterizing classes of subjects for whom no good matches are available and to define substantively interesting groups between which treatment effects may vary. These and related methods are illustrated using data from a Montreal-based study. The effects on subsequent violence of gang joining at age 14 are studied while controlling for measured characteristics of boys prior to age 14. The boys are divided into trajectory groups based on violence from ages 11 to 13. Within trajectory group, joiners are optimally matched to a variable number of controls using propensity scores, Mahalanobis distances, and a combinatorial optimization algorithm. Use of variable ratio matching results in greater efficiency than pair matching and also greater bias reduction than matching at a fixed ratio. The possible impact of failing to adjust for an important but unmeasured covariate is examined using sensitivity analysis. [Author Abstract]
Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r
(American Psychology-Law Society, 2005-10) Rice, Marnie E.; Harris, Grant T.
In order to facilitate comparisons across follow-up studies that have used different measures of effect size, we provide a table of effect size equivalencies for the three most common measures: ROC area (AUC), Cohen’s d, and r. We outline why AUC is the preferred measure of predictive or diagnostic accuracy in forensic psychology or psychiatry, and we urge researchers and practitioners to use numbers rather than verbal labels to characterize effect sizes. [Author Abstract]
Comparing the effects of community service and short-term imprisonment on recidivism: A matched samples approach
(Springerlink, 2010-06-19) Wermink, Hilde; Blokland, Arjan; Nieuwbeerta, Paul; Nagin, Daniel; Tollenaar, Nikolaj
This study uses longitudinal official record data on adult offenders in The Netherlands (n=4,246) to compare recidivism after community service to that after short-term imprisonment. To account for possible bias due to selection of offenders into these types of sanctions, we control for a large set of confounding variables using a combined method of ‘matching by variable’ and ‘propensity score matching’. Our findings demonstrate that offenders recidivate significantly less after having performed community service compared to after having been imprisoned. This finding holds for both the short- and long-term. Furthermore, using the Rosenbaum bounds method, we show that the results are robust for hidden bias. [Author Abstract]
Covariate balancing propensity score
(Royal Statistical Society, 2013) Imai, Kosuke; Ratkovic, Marc
The propensity score plays a central role in a variety of causal inference settings. In particular, matching and weighting methods based on the estimated propensity score have become increasingly common in the analysis of observational data. Despite their popularity and theoretical appeal, the main practical difficulty of these methods is that the propensity score must be estimated. Researchers have found that slight misspecification of the propensity score model can result in substantial bias of estimated treatment effects. We introduce covariate balancing propensity score (CBPS) methodology, which models treatment assignment while optimizing the covariate balance. The CBPS exploits the dual characteristics of the propensity score as a covariate balancing score and the conditional probability of treatment assignment. The estimation of the CBPS is done within the generalized method-of-moments or empirical likelihood framework. We find that the CBPS dramatically improves the poor empirical performance of propensity score matching and weighting methods reported in the literature. We also show that the CBPS can be extended to other important settings, including the estimation of the generalized propensity score for non-binary treatments and the generalization of experimental estimates to a target population. Open source software is available for implementing the methods proposed. [Author Abstract]
Covariate balancing propensity score for a continuous treatment: Application to the efficacy of political advertisements
(Institute of Mathematical Statistics, 2018) Fong, Christian; Hazlett, Chad; Imai, Kosuke
Propensity score matching and weighting are popular methods when estimating causal effects in observational studies. Beyond the assumption of unconfoundedness, however, these methods also require the model for the propensity score to be correctly specified. The recently proposed covariate balancing propensity score (CBPS) methodology increases the robustness to model misspecification by directly optimizing sample covariate balance between the treatment and control groups. In this paper, we extend the CBPS to a continuous treatment. We propose the covariate balancing generalized propensity score (CBGPS) methodology, which minimizes the association between covariates and the treatment. We develop both parametric and nonparametric approaches and show their superior performance over the standard maximum likelihood estimation in a simulation study. The CBGPS methodology is applied to an observational study, whose goal is to estimate the causal effects of political advertisements on campaign contributions. We also provide open-source software that implements the proposed methods. [Author Abstract]
Developing an alternative juvenile programming effort to reduce detention overreliance
(OJJDP Journal of Juvenile Justice, 2016) van Wormer, Jacqueline G.; Campbell, Christopher
The assumption underlying juvenile detention alternatives is that youth on probation receiving programming or treatment are less likely to recidivate, whereas youth in detention will be more likely to recidivate. Under a coordinated justice reform effort, a juvenile justice court system serving two southeastern counties in Washington state developed a program (the FAST program) for probation violators that offered 2 sessions of accountability skill development to address targeted criminogenic needs in lieu of a formalized hearing and a subsequent stay in detention. The goal of the FAST program for participating youth was to reduce future probation violations and detention stays. This paper presents an evaluation of the FAST program using propensity score modeling of 434 juvenile probation violators. A comparison of matched groups shows the program does not reduce recidivism or future probation violations among participants, though it does produce the same result as those who received detention. Our explanation makes the case for increasing the dosage (number of sessions) of violator programs, which may be what is necessary to provide a more effective alternative to detention. [Author Abstract]
Effect of mental health courts on arrests and jail days
(Arch Gen Psychiatry, 2010-10-04) Steadman, Henry J.; Redlich, Allison; Callahan, Lisa; Clark Robbins, Pamela; Vesselinov, Roumen
This study is a 4-site, prospective, longitudinal, quasiexperimental study. The MHC and TAU samples were interviewed and followed up for 18 months at each site. The core research questions addressed here are (1) is participation in an MHC associated with more favorable criminal justice outcomes than processing through the regular criminal court system? and (2) for what types of defendants do MHCs produce the most favorable criminal justice outcomes? [Author Abstract}
Exploring the black box of community supervision
(Haworth Press, 2008) Bonta, James; Bourgon, Guy; Scott, Terri
Community supervision has been an integral part of corrections since the establishment of probation more than 100 years ago. It has commonly been assumed that offenders benefit from community supervision much more than if they were incarcerated. However, empirical evidence in support of the effectiveness of community supervision in reducing recidivism questions this assumption. A detailed examination of audio taped interviews between 62 probation officers and their clients found relatively poor adherence to some of the basic principles of effective intervention–the principles of Risk, Need and Responsivity. For the most part, probation officers spent too much time on the enforcement aspect of supervision (i.e., complying with the conditions of probation) and not enough time on the service delivery role of supervision. Major criminogenic needs such as antisocial attitudes and social supports for crime were largely ignored and probation officers evidenced few of the skills (e.g., prosocial modeling, differential reinforcement) that could influence behavioral change in their clients. As a snapshot of present practices, this study begins a path to a systematic and structured training agenda to help probation officers become more effective agents of change. [Author Abstract]
Final report: Process and outcome evaluation of the G.R.E.A.T. Program
(U.S. Department of Justice, 2013) Esbensen, Finn-Aage; Osgood, Wayne; Peterson, Dana; Taylor, Terrance J.; Carson, Dena; Freng, Adrienne; Matsuda, Kristy
In 2006, the University of Missouri-St. Louis was awarded a grant from the National Institute of Justice to determine what effect, if any, the G.R.E.A.T. (Gang Resistance Education and Training) program had on students. G.R.E.A.T., which is a 13-lesson general prevention program taught by uniformed law enforcement officers to middle school students, has three stated goals: 1) to reduce gang membership, 2) to reduce delinquency, especially violent offending, and 3) to improve students’ attitudes toward the police. The process evaluation consisted of multiple methods to assess program fidelity: 1) observations of G.R.E.A.T. Officer Trainings, 2) surveys and interviews of G.R.E.A.T.-trained officers and supervisors, 3) surveys of school personnel, and 4)“on-site,” direct observations of officers delivering the G.R.E.A.T. program in the study sites. Results illustrate a high level of program fidelity, providing greater confidence in any subsequent outcome results. To assess program effectiveness, we conducted a randomized control trial involving 3,820 students nested in 195 classrooms in 31 schools in 7 cities. Active parental consent was obtained for 78% (3,820 students) of the students enrolled (11 percent of parents declined and 11 percent failed to return consent forms). These students were surveyed six times (completion rates were: 98%, 95%, 87%, 83%, 75%, and 72%).in the course of five years thereby allowing assessment of both short- and long-term program effects. Approximately half of the G.R.E.A.T. grade-level classrooms within each school were randomly assigned to experimental or control groups, with102 classrooms (2,051 students) assigned to receive G.R.E.A.T. and 93 classrooms (1,769 students) assigned to the control condition. Results from analyses of data one-year post-program delivery were quite favorable; we found statistically significant differences between the treatment (i.e., G.R.E.A.T.) and control students on 14 out of 33 attitudinal and behavioral outcomes. However, the question remained whether the program had long-term impacts that persisted into high school. To address this question, we continued to survey this group of students for three more years (most of the students were in 10th or 11th grade at the time of the last survey administration). The four-year post program analyses revealed results similar to the one-year post program effects, albeit with smaller effect sizes. Across four years post program 10 positive program effects were found, including lower odds of gang joining and more positive attitudes to police. [Author Abstract]
Full matching in an observational study of coaching for the SAT
(American Statistical Association, 2004-09) Hansen, Ben B.
Among matching techniques for observational studies, full matching is in principle the best, in the sense that its alignment of comparable treated and control subjects is as good as that of any alternate method, and potentially much better. This article evaluates the practical performance of full matching for the first time, modifying it in order to minimize variance as well as bias and then using it to compare coached and uncoached takers of the SAT. In this new version, with restrictions on the ratio of treated subjects to controls within matched sets, full matching makes use of many more observations than does pair matching, but achieves far closer matches than does matching with k ≥ 2 controls. Prior to matching, the coached and uncoached groups are separated on the propensity score by 1.1 SDs. Full matching reduces this separation to 1% or 2% of an SD. In older literature comparing matching and regression, Cochran expressed doubts that any method of adjustment could substantially reduce observed bias of this magnitude. To accommodate missing data, regression-based analyses by ETS researchers rejected a subset of the available sample that differed significantly from the subsample they analyzed. Full matching on the propensity score handles the same problem simply and without rejecting observations. In addition, it eases the detection and handling of nonconstancy of treatment effects, which the regression-based analyses had obscured, and it makes fuller use of covariate information. It estimates a somewhat larger effect of coaching on the math score than did ETS’s methods. [Author Abstract]
Genetic matching for estimating causal effects: A general multivariate matching method for acheiving balance in observational studies
(Institute of Governmental Studies, UC Berkeley, 2006) Diamond, Alexis; Sekhon, Jasjeet S.
Genetic matching is a new method for performing multivariate matching which uses an evolutionary search algorithm to determine the weight each covariate is given. The method utilizes an evolutionary algorithm developed by Mebane and Sekhon (1998; Sekhon and Mebane 1998) that maximizes the balance of observed potential confounders across matched treated and control units. The method is nonparametric and does not depend on knowing or estimating the propensity score, but the method is greatly improved when a known or estimated propensity score is incorporated. Genetic matching reliably reduces both the bias and the mean square error of the estimated causal effect even when the property of equal percent bias reduction (EPBR) does not hold. When this property does not hold, matching methods—such as Mahalanobis distance and propensity score matching—often perform poorly. Even if the EPBR property does hold and the propensity score is correctly specified, in finite samples, estimates based on genetic matching have lower mean square error than those based on the usual matching methods. We present a reanalysis of the LaLonde (1986) job training dataset which demonstrates the benefits of genetic matching and which helps to resolve a longstanding debate between Dehejia and Wahba (1997; 1999; 2002; Dehejia 2005) and Smith and Todd (2001, 2005a,b) over the ability of matching to overcome LaLonde’s critique of nonexperimental estimators. Monte Carlos are also presented to demonstrate the properties of our method. [Author Abstract]
Goodness-of-fit diagnostics for the propensity score model when estimating treatment effects using covariate adjustment with the propensity score
(John Wiley & Sons, Ltd., 2008-10-29) Austin, Peter C.
The propensity score is defined to be a subject’s probability of treatment selection, conditional on observed baseline covariates. Conditional on the propensity score, treated and untreated subjects have similar distributions of observed baseline covariates. In the medical literature, there are three commonly employed propensity-score methods: stratification (subclassification) on the propensity score, matching on the propensity score, and covariate adjustment using the propensity score. Methods have been developed to assess the adequacy of the propensity score model in the context of stratification on the propensity score and propensity-score matching. However, no comparable methods have been developed for covariate adjustment using the propensity score. Inferences about treatment effect made using propensity-score methods are only valid if, conditional on the propensity score, treated and untreated subjects have similar distributions of baseline covariates. We develop both quantitative and qualitative methods to assess the balance in baseline covariates between treated and untreated subjects. The quantitative method employs the weighted conditional standardized difference. This is the conditional difference in the mean of a covariate between treated and untreated subjects, in units of the pooled standard deviation, integrated over the distribution of the propensity score. The qualitative method employs quantile regression models to determine whether, conditional on the propensity score, treated and untreated subjects have similar distributions of continuous covariates. We illustrate our methods using a large dataset of patients discharged from hospital with a diagnosis of a heart attack (acute myocardial infarction). The exposure was receipt of a prescription for a beta-blocker at hospital discharge. [Author Abstract]
Matching methods for causal inference: A review and a look forward
(Stat Sci., 2010-02-01) Stuart, Elizabeth A.
When estimating causal effects using observational data, it is desirable to replicate a randomized experiment as closely as possible by obtaining treated and control groups with similar covariate distributions. This goal can often be achieved by choosing well-matched samples of the original treated and control groups, thereby reducing bias due to the covariates. Since the 1970’s, work on matching methods has examined how to best choose treated and control subjects for comparison. Matching methods are gaining popularity in fields such as economics, epidemiology, medicine, and political science. However, until now the literature and related advice has been scattered across disciplines. Researchers who are interested in using matching methods–or developing methods related to matching–do not have a single place to turn to learn about past and current research. This paper provides a structure for thinking about matching methods and guidance on their use, coalescing the existing research (both old and new) and providing a summary of where the literature on matching methods is now and where it should be headed. [Author Abstract]
Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies
(John Wiley & Sons Ltd., 2015-07-09) Austin, Peter C.; Stuart, Elizabeth A.,
The propensity score is defined as a subject’s probability of treatment selection, conditional on observed baseline covariates. Weighting subjects by the inverse probability of treatment received creates a synthetic sample in which treatment assignment is independent of measured baseline covariates. Inverse probability of treatment weighting (IPTW) using the propensity score allows one to obtain unbiased estimates of average treatment effects. However, these estimates are only valid if there are no residual systematic differences in observed baseline characteristics between treated and control subjects in the sample weighted by the estimated inverse probability of treatment. We report on a systematic literature review, in which we found that the use of IPTW has increased rapidly in recent years, but that in the most recent year, a majority of studies did not formally examine whether weighting balanced measured covariates between treatment groups. We then proceed to describe a suite of quantitative and qualitative methods that allow one to assess whether measured baseline covariates are balanced between treatment groups in the weighted sample. The quantitative methods use the weighted standardized difference to compare means, prevalence, higher-order moments, and interactions. The qualitative methods employ graphical methods to compare the distribution of continuous baseline covariates between treated and control subjects in the weighted sample. Finally, we illustrate the application of these methods in an empirical case study. We propose a formal set of balance diagnostics that contribute towards an evolving concept of ‘best practice’ when using IPTW to estimate causal treatment effects using observational data. [Author Abstract]
Multivariate matching methods that are monotonic imbalance bounding
(American Statistical Association, 2011-03) Iacus, Stefano, M.; King, Gary; Porro, Giuseppe
We introduce a new “Monotonic Imbalance Bounding” (MIB) class of matching methods for causal inference with a surprisingly large number of attractive statistical properties. MIB generalizes and extends in several new directions the only existing class, “Equal Percent Bias Reducing” (EPBR), which is designed to satisfy weaker properties and only in expectation. We also offer strategies to obtain specific members of the MIB class, and analyze in more detail a member of this class, called Coarsened Exact Matching, whose properties we analyze from this new perspective. We offer a variety of analytical results and numerical simulations that demonstrate how members of the MIB class can dramatically improve inferences relative to EPBR-based matching methods. [Author Abstract]
Panacea or poison: Can propensity score modeling (PSM) methods replicate the results from randomized control trials (RCTs)?
(Criminal Justice Policy Research Institute, 2023-01-07) Campbell, Christopher; Labrecque, Ryan M.
In this document, the authors report on a research project to determine whether or not propensity score modelling (PSM) can replicate the results of randomized controlled trials (RCTs) for criminal justice evaluations. For the research project, the authors set out to assess the reliability and validity of seven PSM methods in replicating the results of 10 criminal justice RCT experiments. In their research, the authors focused on the following five different PSM techniques: one-to-one matching, with and without a caliper; one-to-many matching, with and without a caliper; inverse probability of the treatment weighting (IPTW); stratified weighting scheme; and optimal pairs matching. The researchers gathered the datasets of 10 publicly available and restricted RCT studies from the National Archive of Criminal Justice Data (NACJD), introduced an artificial selection bias into the treatment groups of the investigations, and then used each PSM technique to remove the selection bias. The researchers then compared the results generated from the PSM methods to those derived from the original RCT experiments and meta-analyzed the findings across all studies to reveal the true reliability and validity of PSM in relation to RCTs using criminal justice data. Results indicated that there is sufficient support for the use of PSM in criminal justice research, and the authors note that their research demonstrated that those seven PSM methods can be an effective means for estimating reliable and valid simulation of RCT experiments.

Browse

Browsing JRA Collection by Title

Results Per Page

Sort Options