PEPFAR Quality Control and Method Validation Activity 10: Presentation Slides

Selecting appropriate rules to meet the intended use of the test. ACTIVITY: How to Select Control Rules 1 On‐going Evaluation of Your Quality Control Systems Gaussian is the Key to Quality Control Alerts us to changes in Monitors and evaluates accuracy and precision method performance NS EQA TER MN Identify rule violations when Defines an acceptable change vs. a significant change occurs unacceptable change (TE < TEA) Determines how far the mean can shift before erroneous results are reported (SEc and Sigma‐metric) Select appropriate control rules based on the SEc or Sigma for that method 2 ISO 15189 5.6.2 Quality control 5.6.2.1 General The laboratory shall design quality control procedures that verify the attainment of the intended quality of results. QC procedures must take into account the quality required for each test and the performance capabilities of the test method. 3 1 HILS1758 Effective Date: 12/06/2016 An Effective QC Design • Ensures quality performance by quickly detecting medically significant errors (Ped ≥ 90%) • Generate few rule violations when there are no significant errors occurring (Pfr ≤ 5%) 4 QC Rule Attributes • Wider control limits are less sensitive to error detection but have fewer false rejections • Error detection will be greater when control limits are set closer to the mean. The tradeoff is that you will also have a higher false rejection rate • Easier to detect larger shifts than smaller shifts • Proportion of data measurements beyond the control limits increases with larger shifts 5 When SEc and Sigma-metrics are large, • the less overlap occurs between the before‐ change population and after‐change population data measurements • the easier it becomes to select rules that will alert us to a medically significant change • less QC effort is needed for methods that easily meet quality specifications 6 2 HILS1758 Effective Date: 12/06/2016 Sigma Scale 1.65 1.0 2.65 36 3.65 40 4.65 50 5.65Pfr P ed N R 13g/20132 /R4 13, 0.9 4s 150x 0.07 6 0.8 135 22g/R45 1415/8x Desirable 0.03 2 0.7 Error 35 225 R45 /4, Detection 0.03 T 0.6 Desirable 2.5s 0.5 False 0.04 4 1 Rejection 2.03 0.4 135 2251 2 1 0.3 0.072 Probability for Rejection (P) 0.2 0:00 2 0.1 13se 1.0 2.0 3.0 1.0 Control metric (ASE, multiples of s) 7 Image from Westgard, James (2007) Assuring the Right Quality Right. page127 Sigma-Metrics QC Selection Tool for 2 Levels Control Sigma Scale 2.65 30 3.65 46 5.65P Ped N R 17,20102 /2 015 5% 09 6 135.22 .R. 16 615 By 135 225 R 15/ 15 0.6 05 0.4 2 0.3 02 Probability for Rejection (P) 0.1 30 Systematic Enor (SE, multiples of s) 10 8 SigmaMetricsToolHandout.pdf from Westgard.com website. Last accessed March 12, 2012. Sigma-Metrics QC Selection Tool for 3 Levels Control Sigma Scale 1.65 10 2.65 3.65 46 5.65P Pad NR 1.36 20102 /2, 315.5x 09 007 6 0.8 003 3 2 0.7 12013,/R/3, DDS 6 0.6 17. 2013, /R 16 315 O.S 3 0.3 02 Probability for Rejection (P) 0.1 Systematic Enor (SE, multiples of s) 30 40 9 SigmaMetricsToolHandout.pdf from Westgard.com website. Last accessed March 12, 2012. 3 HILS1758 Effective Date: 12/06/2016 QC Rule Selection Guidelines • Use control materials that assess the measurement procedure at medical points of interest • Provide 90% or greater error detection • Have 5% or less false rejection rates • Fewest number of control measurements per analytical run possible to save on costs associated with • QC materials • Reagents • Consumables • Meet regulatory or accrediting body ’ s requirements for number of measurements per run 10 Sigma-metrics Rule Selection Guide  6 sigma ‐ any QC will do (just don’t use 2 SD limits)  5 sigma ‐ single‐rules such as 1:3s or 1:2.5s  4 sigma ‐ multirules  < 4 sigma ‐ multirules with look‐back to previous runs, increase the number of controls analyzed  3 or less ‐ look for better analytical methods From www.westgard.com/links ‐to ‐india ‐part i ‐best ‐qc ‐practices ‐and ‐westgard ‐rules.htm. Last accessed on March 5, 2013. ‐ 11 Activity: Predicting Probabilities Purpose What will you do? Determine the probabilities Working individually, you will: and percentages of error  Use the information supplied on Worksheet 1 detection when given either and calculate either the SEc or Sigma value. the SEc or Sigma ‐metric for a  Use Handout 1 for three levels of control measurement procedure. materials to:  Locate the value on the Sigma ‐metric scale What will you need?  Place your ruler so that it corresponds to the • Worksheet 1: Predicting Ped SEc value – ensure the ruler is straight. Capabilities  Read the probability of rejection from the Y ‐ • Handout 1: QC Rule Selection axis where the ruler and the rule intersect. • Pencils  Record your answers on Worksheet 1. • Rulers  Place a tic mark next to the rules that are appropriate to consider.  Circle the rule you select  Prepare to defend your selection 12 25 minutes 4 HILS1758 Effective Date: 12/06/2016 Example A Sigma Scale for 3 Level of Controls 1.65 1.0 2.65 30 3.65 40 165 58 Confilate| Ride P. NR 09 QC Rale 1-36/2of 375/45/3:15/6x 6 1 08 1-36/2of 375/8:45/3:15/6x DIB 3 2 0.7 1354 1-36/2of 375/45/3:15 0.6 1-35/2of 3: 25 /1 :45/3-15 31 1234 D.m. 3 1 0.3 D.m 02 Probability for Rejection (P) 0.1 00 Systematic Enor (SE, multiples of's) 10 20 30 13 Example B Sigma Scale for 3 Level of Controls 1D 3,65 46 465 50 Candidate Rule NR 09 C Rules 129 1-35/2 of 3:2/R:45/3:15/6x 6 1 0.8 1-35/2 of 3:2/R:45/3:15/6x 3 2 0.7 1-35/2 of 3:2/45/3-15 6 1 0.6 1-35/2 of 3:24/R:45/3:15 3 1 O.S MV 1-35 0.4 1.2 55 0.3 1:35 3 02 Probability for Rejection (P) 0.1 00 Systematic Enor (SE, multiples of s) 10 20 14 Example C Sigma Scale for 3 Level of Controls 1.65 10 2.65 3 3.65 46 165 58 5.6 Candidate Rule Pfr ≈Ped N R 09 QC Rules (%) (%) 1:3s/2 of 3:2s/R:4s/3:1s/6x 0.07 0.80 6 1 0.8 (7%) (80%) 1:3s/2 of 3:2s/R:4s/3:1s/6x 0.03 0.68 3 2 0.7 (3%) (68%) 1:3s/2 of 3:2s/R:4s/3:1s 0.05 0.63 6 1 0.6 (5%) (63%) 1:3s/2 of 3:2s/R:4s/3:1s 0.02 0.38 3 1 (2%) (38%) 1:3s 0.01 0.28 6 1 O 0.4 (1%) (28%) 1:2.5s 0.03 0.32 3 1 (3%) (32%) 03 1:3s 0.01 0.12 3 1 (1%) (12%) 02 1:3s 0.00 0.08 1 1 Probability for Rejection (P) (0%) (8%) 0.1 00 Systematic Enor (SE, multiples of's) 10 20 15 5 HILS1758 Effective Date: 12/06/2016 When implementing a QC strategy, remember that When Sigma and SEc are high, it is easy and cost ‐effective to control the analytical system using SQC. As Sigma and SEc decreases, it becomes harder and less cost ‐effective to control the analytical system and additional actions beyond SQC must be taken. 16 Select an appropriate QC rule Three controls are used to monitor haemoglobin. After 20 measurements collected over a period of 20 days when the system was stable, the mean for the high control was calculated to be 18.1 g/dl and the SD was calculated to be 0.3 g/dl. The following package insert information was provided: Parameter Units Low Normal High Hgb g/dl 5.5 ± 0.4 13.4 ± 0.6 18.0± 0.8 Laboratory management is currently advocating upper management’s support to enroll in a proficiency testing program and to participate in an inter ‐laboratory peer comparison program with the manufacturer of the control materials. 17 Activity: Workstation Purpose What will you do? During the site visit from Working in groups of 6, you will: Activity 9, the QA officer  has asked for your further Review the information provided in the assistance. You have agreed Handout 2 and Worksheet 2 to help him develop a QC  Design an appropriate, implementable and strategy for the XYZ analyzer cost ‐effective QC strategy(ies) for all the workstation. analytes tested at the XYZ workstation.  Include the following with each QC strategy: What will you need?  Control rule(s), N, R, and supervisor chart • Handout 2: Monthly review frequency Review Log  Select a spokesperson • Worksheet 2: Candidate  QC Rules Participate in the class discussion • Flipchart paper Plenary Discussion  markers One group will be selected at random to • present their answers.  Have the spokesperson present a 3 ‐minute 40 minutes explanation of the group’s QC design. 18 6 HILS1758 Effective Date: 12/06/2016 ISO 15189 Standard: “The laboratory shall design internal quality control procedures that verify the attainment of the intended quality of results…” ISO 15189: 5.6.1 19 7 HILS1758 Effective Date: 12/06/2016