Assay Validation - Replicate Experiment Analysis
A Replicate Experiment Analysis is a formal experiment that can be used to decide if an assay is ready for production use. It should be performed as part of assay validation for new assays and for existing assays being moved to a new lab (or undergoing other changes).
The goal of the replicate experiment analysis for new assays is slightly different from that for assays being relocated. For a new assay, the goal is to determine the variability of the assay result over a range of potencies, as an assay with high variability is of limited value for lead selection and/or SAR analysis. While for an assay being moved to a new lab, the goal is to determine both the variability AND the equivalency of the new assay results vs the original assay. This is especially important if the results from the old and new assay are to be combined for future lead selection and/or SAR analysis.
In both cases, a statistical comparison is done between the first two runs of the assay, under production conditions, using the same panel of 20 or more compounds covering a wide range of potencies. This is referred to as a "reproducibility comparison". In addition, for relocated assays only, an "assay comparison" is done between a run from the new assay and a run from the original assays. The calculations used for both types of comparison are the same and yield the test statistics described below. Details on how these are calculated can be found in the NIH Assay Guidance Manual (https://www.ncbi.nlm.nih.gov/books/NBK83783/).
Mean Ratio (95% CI)
Geometric mean of the ratios of the results for all compounds (run 1/run 2). This value should be close to 1 (between 0.67 and 1.5), indicating that the there is no meaningful difference in the results between runs. If the ratio falls outside this range, it may still be okay, as long as the 95% confidence interval includes 1. However, the mean ratio (MR) alone is not sufficient to determine if an assay is production ready.
MSR (within run)
Minimum Significant Ratio (within run). This is the smallest result ratio between two compounds, in the same run, that is statistically significant. This must be less than 3 for the assay to be considered production ready.
Limits of Agreement (upper and lower)
The limits of agreement (LofA) take into account both the mean ratio and MSR. If both limits fall between 0.33 and 3, the assay is ready for production.
The slideshow below shows Analyze™ screen captures of typical replicate experiment analyses of potency results. The table shows the result count, the test statistics, and the PASS/FAIL validation tests for the key statistics. The chart shows the statistics (horizontal lines) and the individual ratios of each of the compounds tested (blue points). The X-axis position of each point is the geometric mean of each paired results while the Y-axis position is the ratio of those results (run1/run2).
In order for an assay to be considered production-ready, all tests must pass AND there must be no trend observed in the charted points. Such a trend could indicate that results from the new assay don't agree with those of the old over the entire potency range, even when the test statistics are all passing.
Note that this post was focused on comparing potency results (e.g., IC50). A replicate experiment analysis can also be used for efficacy results (e.g., Max %Inh), using a different set of test statistics. Analyze™ supports this type of analysis as well.