Laboratory records are maintained with sufficient detail, so that other equally qualified analysts can reconstruct the experimental conditions and review the results obtained. When collecting data, the data should generally be obtained with more decimal places than the specification requires and rounded only after final calculations are completed as per the General Notices and Requirements.

Effective sampling is an important step in the assessment of a quality attribute of a population. The purpose of sampling is to provide representative data (the sample) for estimating the properties of the population. How to attain such a sample depends entirely on the question that is to be answered by the sample data. In general, use of a random process is considered the most appropriate way of selecting a sample. Indeed, a random and independent sample is necessary to ensure that the resulting data produce valid estimates of the properties of the population. Generating a nonrandom or “convenience” sample risks the possibility that the estimates will be biased. The most straightforward type of random sampling is called simple random sampling, a process in which every unit of the population has an equal chance of appearing in the sample. However, sometimes this method of selecting a random sample is not optimal because it cannot guarantee equal representation among factors (i.e., time, location, machine) that may influence the critical properties of the population. For example, if it requires 12 hours to manufacture all of the units in a lot and it is vital that the sample be representative of the entire production process, then taking a simple random sample after the production has been completed may not be appropriate because there can be no guarantee that such a sample will contain a similar number of units made from every time period within the 12-hour process. Instead, it is better to take a systematic random sample whereby a unit is randomly selected from the production process at systematically selected times or locations (e.g., sampling every 30 minutes from the units produced at that time) to ensure that units taken throughout the entire manufacturing process are included in the sample. Another type of random sampling procedure is needed if, for example, a product is filled into vials using four different filling machines. In this case it would be important to capture a random sample of vials from each of the filling machines. A stratified random sample, which randomly samples an equal number of vials from each of the four filling machines, would satisfy this requirement. Regardless of the reason for taking a sample (e.g., batch-release testing), a sampling plan should be established to provide details on how the sample is to be obtained to ensure that the sample is representative of the entirety of the population and that the resulting data have the required sensitivity. The optimal sampling strategy will depend on knowledge of the manufacturing and analytical measurement processes. Once the sampling scheme has been defined, it is likely that the sampling will include some element of random selection. Finally, there must be sufficient sample collected for the original analysis, subsequent verification analyses, and other analyses.

Tests discussed in the remainder of this chapter assume that simple random sampling has been performed.

Where the use of the USP Reference Standard is specified, the USP Reference Standard, or a secondary standard traceable to the USP Reference Standard, is used. Because the assignment of a value to a standard is one of the most important factors that influences the accuracy of an analysis, it is critical that this be done correctly.

Verifying an acceptable level of performance for an analytical system in routine or continuous use can be a valuable practice. This may be accomplished by analyzing a control sample at appropriate intervals, or using other means, such as, variation among the standards, background signal-to-noise ratios, etc. Attention to the measured parameter, such as charting the results obtained by analysis of a control sample, can signal a change in performance that requires adjustment of the analytical system.

All methods are appropriately validated as specified under Validation of Compendial Procedures 1225. Methods published in the USP–NF have been validated and meet the Current Good Manufacturing Practices regulatory requirement for validation as established in the Code of Federal Regulations. A validated method may be used to test a new formulation (such as a new product, dosage form, or process intermediate) only after confirming that the new formulation does not interfere with the accuracy, linearity, or precision of the method. It may not be assumed that a validated method could correctly measure the active ingredient in a formulation that is different from that used in establishing the original validity of the method.

Precision is the degree of agreement among individual test results when the analytical method is applied repeatedly to a homogeneous sample. For an alternative method to be considered to have “comparable” precision to that of a current method, its precision (see Analytical Performance Characteristics under Validation of Compendial Procedures 1225) must not be worse than that of the current method by an amount deemed important. A decrease in precision (or increase in variability) can lead to an increase in the number of results expected to fail required specifications. On the other hand, an alternative method providing improved precision is acceptable.

One way of comparing the precision of two methods is by estimating the variance for each method (the sample variance, s2, is the square of the sample standard deviation) and calculating a one-sided upper confidence interval for the ratio of (true) variances, where the ratio is defined as the variance of the alternative method to that of the current method. An example, with this assumption, is outlined under Appendix D. The one-sided upper confidence limit should be compared to an upper limit deemed acceptable, a priori, by the analytical laboratory. If the one-sided upper confidence limit is less than this upper acceptable limit, then the precision of the alternative method is considered acceptable in the sense that the use of the alternative method will not lead to an important loss in precision. Note that if the one-sided upper confidence limit is less than one, then the alternative method has been shown to have improved precision relative to the current method.

The confidence interval method just described is preferred to applying the two-sample F-test to test the statistical significance of the ratio of variances. To perform the two-sample F-test, the calculated ratio of sample variances would be compared to a critical value based on tabulated values of the F distribution for the desired level of confidence and the number of degrees of freedom for each variance. Tables providing F-values are available in most standard statistical textbooks. If the calculated ratio exceeds this critical value, a statistically significant difference in precision is said to exist between the two methods. However, if the calculated ratio is less than the critical value, this does not prove that the methods have the same or equivalent level of precision; but rather that there was not enough evidence to prove that a statistically significant difference did, in fact, exist.

Comparison of the accuracy (see Analytical Performance Characteristics under Validation of Compendial Procedures 1225) of methods provides information useful in determining if the new method is equivalent, on the average, to the current method. A simple method for making this comparison is by calculating a confidence interval for the difference in true means, where the difference is estimated by the sample mean of the alternative method minus that of the current method.

The confidence interval should be compared to a lower and upper range deemed acceptable, a priori, by the laboratory. If the confidence interval falls entirely within this acceptable range, then the two methods can be considered equivalent, in the sense that the average difference between them is not of practical concern. The lower and upper limits of the confidence interval only show how large the true difference between the two methods may be, not whether this difference is considered tolerable. Such an assessment can only be made within the appropriate scientific context.

The confidence interval method just described is preferred to the practice of applying a t-test to test the statistical significance of the difference in averages. One way to perform the t-test is to calculate the confidence interval and to examine whether or not it contains the value zero. The two methods have a statistically significant difference in averages if the confidence interval excludes zero. A statistically significant difference may not be large enough to have practical importance to the laboratory because it may have arisen as a result of highly precise data or a larger sample size. On the other hand, it is possible that no statistically significant difference is found, which happens when the confidence interval includes zero, and yet an important practical difference cannot be ruled out. This might occur, for example, if the data are highly variable or the sample size is too small. Thus, while the outcome of the t-test indicates whether or not a statistically significant difference has been observed, it is not informative with regard to the presence or absence of a difference of practical importance.

Sample size determination is based on the comparison of the accuracy and precision of the two methods4 and is similar to that for testing hypotheses about average differences in the former case and variance ratios in the latter case, but the meaning of some of the input is different. The first component to be specified is , the largest acceptable difference between the two methods that, if achieved, still leads to the conclusion of equivalence. That is, if the two methods differ by no more than , they are considered acceptably similar. The comparison can be two-sided as just expressed, considering a difference of in either direction, as would be used when comparing means. Alternatively, it can be one-sided as in the case of comparing variances where a decrease in variability is acceptable and equivalency is concluded if the ratio of the variances (new/current, as a proportion) is not more than 1.0 + . A researcher will need to state based on knowledge of the current method and/or its use, or it may be calculated. One consideration, when there are specifications to satisfy, is that the new method should not differ by so much from the current method as to risk generating out-of-specification results. One then chooses to have a low likelihood of this happening by, for example, comparing the distribution of data for the current method to the specification limits. This could be done graphically or by using a tolerance interval, an example of which is given in Appendix E. In general, the choice for must depend on the scientific requirements of the laboratory.

The next two components relate to the probability of error. The data could lead to a conclusion of similarity when the methods are unacceptably different (as defined by ). This is called a false positive or Type I error. The error could also be in the other direction; that is, the methods could be similar, but the data do not permit that conclusion. This is a false negative or Type II error. With statistical methods, it is not possible to completely eliminate the possibility of either error. However, by choosing the sample size appropriately, the probability of each of these errors can be made acceptably small. The acceptable maximum probability of a Type I error is commonly denoted as and is commonly taken as 5%, but may be chosen differently. The desired maximum probability of a Type II error is commonly denoted by . Often, is specified indirectly by choosing a desired level of 1 – , which is called the “power” of the test. In the context of equivalency testing, power is the probability of correctly concluding that two methods are equivalent. Power is commonly taken to be 80% or 90% (corresponding to a of 20% or 10%), though other values may be chosen. The protocol for the experiment should specify , , and power. The sample size will depend on all of these components. An example is given in Appendix E. Although Appendix E determines only a single value, it is often useful to determine a table of sample sizes corresponding to different choices of , , and power. Such a table often allows for a more informed choice of sample size to better balance the competing priorities of resources and risks (false negative and false positive conclusions).

This is a modified version of the ESD Test that allows for testing up to a previously specified number, r, of outliers from a normally distributed population. Let r equal 2, and n equal 10.

Similar to the ESD test, the two smallest values will be tested as outliers; again assuming the data come from a single normal population.

Hampel's Rule

Step 1— The first step in applying Hampel's Rule is to normalize the data. However, instead of subtracting the mean from each data point and dividing the difference by the standard deviation, the median is subtracted from each data value and the resulting differences are divided by MAD (see below). The calculation of MAD is done in three stages. First, the median is subtracted from each data point. Next, the absolute values of the differences are obtained. These are called the absolute deviations. Finally, the median of the absolute deviations is calculated and multiplied by the constant 1.483 to obtain MAD6.

Suppose the process mean and the standard deviation are both unknown, but a sample of size 50 produced a mean and standard deviation of 99.5 and 2.0, respectively. These values were calculated using the last 50 results generated by this specific method for a particular (control) sample. Given this information, the tolerance limits can be calculated by the following formula:

in which x is the mean; s is the standard deviation; and K is based on the level of confidence, the proportion of results to be captured in the interval, and the sample size, n. Tables providing K values are available. In this example, the value of K required to enclose 95% of the population with 95% confidence for 50 samples is 2.3828. The tolerance limits are calculated as follows:

hence, the tolerance interval is (94.7, 104.3).

Assume the specification interval for this method is (90.0, 110.0) and the process mean and standard deviation have not changed since this interval was established. The following quantities can be defined: the lower specification limit (LSL) is 90.0, the upper specification limit (USL) is 110.0, the lower tolerance limit (LTL) is 94.7, and the upper tolerance limit (UTL) is 104.3. Calculate the acceptable difference, (), in the following manner:

Fig. 2. A graph of the quantities calculated above.

With this choice of , and assuming the two methods have comparable precision, the confidence interval for the difference in means between the two methods (alternative-current) should fall within –4.7 and +4.7 to claim that no important difference exists between the two methods.

Quality control analytical laboratories sometimes deal with 99% tolerance limits, in which cases the interval will widen. Using the previous example, the value of K required to enclose 99% of the population with 99% confidence for 50 samples is 3.390. The tolerance limits are calculated as follows:

the resultant wider tolerance interval is (92.7, 106.3). Similarly, the new LTL of 92.7 and UTL of 106.3 would produce a smaller :

Choosing a larger leads to a smaller n but at the cost of increasing the risk of reaching the wrong conclusion.

Formulas are available that can be used for a specified , under the assumption that the population variances are known and equal, to calculate the number of samples required to be tested per method, n. The level of confidence and power must also be specified. [note—Power refers to the probability of correctly concluding that two identical methods are equivalent.] For example, if = 4.7, and the two population variances are assumed to equal 4.0, then, for a 5% level test9 and 80% power (with associated z-values of 1.645 and 1.282, repectively), the sample size is approximated by the following formula:

Thus, assuming each method has a population variance, 2, of 4.0, the number of samples, n, required to conclude with 80% probability that the two methods are equivalent (90% confidence interval for the difference in the true means falls between –4.7 and +4.7) when in fact they are identical (the true mean difference is zero) is 4. Because the normal distribution was used in the above formula, 4 is actually a lower bound on the needed sample size. If feasible, one might want to use a larger sample size. Values for z for common confidence levels are presented in Table 8. The formula above makes three assumptions: 1) the variance used in the sample size calculation is based on a sufficiently large amount of prior data to be treated as known; 2) the prior known variance will be used in the analysis of the new experiment, or the sample size for the new experiment is sufficiently large so that the normal distribution is a good approximation to the t distribution; and 3) the laboratory is confident that there is no actual difference in the means, the most optimistic case. It is not common for all three of these assumptions to hold. The formula above should be treated most often as an initial approximation. Deviations from the three assumptions will lead to a larger required sample size. In general, we recommend seeking assistance from someone familiar with the necessary methods.

When a log transformation is required to achieve normality, the sample size formula needs to be slightly adjusted as shown below. Instead of formulating the problem in terms of the population variance and the largest acceptable difference, , between the two methods, it now is formulated in terms of the population RSD and the largest acceptable proportional difference between the two methods.

where

and represents the largest acceptable proportional difference between the two methods ((alternative-current)/current) and the population RSDs are assumed known and equal.