PEPFAR Quality Control and Method Validation Activity 12: Presentation Slides

Comparing your laboratory’s performance to your peers. ACTIVITY: Understanding Inter‐laboratory Comparison Programs 1 23 24 25 26 27 28 29 30 31 660.0 620.0 610.0 635.0 655.0 650.0 605.0 655.0 660.0 LS LS LS LS LS LS LS LS LS 23/5 24/5 25/5 26/5 27/5 28/5 29/5 30/5 31/5 MAY (current month) 2 710.0 +3 SD -------------------------- 685.0 + 2 SD C o n t 660.0 + 1 SD r o l 635.0 X V a l u 610.0 1 SD - e 585.0 2 SD - 560.0 -3 SD 23 24 25 26 27 28 29 30 31 Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 660.0 620.0 610.0 635.0 655.0 650.0 605.0 655.0 660.0 Value 610.0 615.0 635.0 660.0 590.0 635.0 680.0 655.0 660.0 610.0 625.0 LS LS LS LS LS LS LS LS LS Initials AM AM AM AM AM AM AM AM AM AM TK 23/5 24/5 25/5 26/5 27/5 28/5 29/5 30/5 31/5 Date 1/6 2/6 3/6 4/6 5/6 6/6 7/6 8/6 9/6 10/6 11/6 MAY (month ‐1) JUNE (current month)3 1 HILS1760 Effective Date: 12/06/2016 Cape Clinic Laboratory L-J Chart for XYZ Chemistry Analyzer Analyte: Glucose Control Material: Level 1 Units: mmol/L Lot #: 3-123 Exp Date: 19/4/20XX  assign 3.48 SD assign 0.05 From: June 1, 20XX Through: June 30, 20XX Target Value 3.39 TEa 6.90% 3.63 +3 SD C 3.58 + 2 SD o n t r 3.53 + 1 SD o l V 3.48 X a l u e 3.43 1 SD - 3.38 2 SD - 3.33 -3 SD 4 Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 External Quality Assessment Schemes (EQAS) • Monitor the accuracy of your method on an on‐going basis • Compare your results to other laboratories performing the same method for a particular analyte 5 Types of EQA Programs 1. Proficiency Testing (PT) Survey 2. Inter‐laboratory Comparison Program 3. Split‐sample analysis with a reference laboratory 6 2 HILS1760 Effective Date: 12/06/2016 ISO 15189:2012 ‐ 5.6.3.2 Alternative approaches Whenever an interlaboratory comparison is not available, the laboratory shall develop other approaches and provide objective evidence for determining the acceptability of examination results. Whenever possible, this mechanism shall utilize appropriate materials. NOTE Examples of such materials may include: certified reference materials; — samples previously examined; — material from cell or tissue repositories; — exchange of samples with other laboratories; — control materials that are tested daily in interlaboratory — comparison programmes. 7 ISO 15189:2012 ‐ 5.6.2.2 Quality control materials The laboratory shall use quality control materials that react to the examining system in a manner as close as possible to patient samples. Quality control materials shall be periodically examined with a frequency that is based on the stability of the procedure and the risk of harm to the patient from an erroneous result. NOTE 1 The laboratory should choose concentrations of control materials, wherever possible, especially at or near clinical decision values, which ensure the validity of decisions made. NOTE 2 Use of independent third party control materials should be considered, either instead of, or in addition to, any control materials supplied by the reagent or instrument manufacturer. 8 Enrollment QC Material 12345 Mean SD N Your Laboratory 100 5 33 9 3 HILS1760 Effective Date: 12/06/2016 Remove Outliers Calculate SDI and CVI Buret lanetc . Serum Beckman Coulter Unicel DxC 800 0:53 Method 0.5 0.36 0.5 33 1614 Peer 1ethod 0.27 -0.31 0.8 * phobs: 25 33 1295 1616 10 Inter-laboratory Comparison Programs 1. Your laboratory enrolls in a inter‐laboratory program offered by your QC manufacturer. 2. Your laboratory, along with other laboratories, analyze the same lot number of control materials for the month 3. Your laboratory submits your QC results through a QC data management program to a central facility. 4. The central facility examines the data for outliers and calculates the means and SDs for the peer group and all‐lab group, and SDI and CVI for your laboratory. 5. Your laboratory receives a report indicating your analytical performance. 11 Definitions of qualitative terms describing the performance characteristics of a measurement Accuracy of measurement ‐ Closeness of agreement between a quantity value obtained by measurement and the true value of the measurand Precision ‐ The closeness of agreement between independent test results obtained under stipulated conditions Trueness ‐ The closeness of agreement between the average value obtained from a large series of test results and an accepted reference value 12 4 HILS1760 Effective Date: 12/06/2016 Consensus Based Metrics Standard Deviation Index Coefficient of Variation (SDI) Index (CVI)  Peer‐based measure of  Peer‐based measure of bias imprecision  Describes how far our  Comparison of our mean is from the peer or laboratory's CV to the all lab methods’ mean peer or all‐lab CV  Also known as Coefficient of Variation Ratio (CVR) 13 Coefficient of Variation Index CVI Calculation Lab’s Monthly CV Peer Monthly CV Ideally, CVI ≤ 1.0, since your values are from a single lab, while the peer CV is from several laboratories. If CVI = 1.5 to 2.0, your lab is 50‐100% less precise than its peer group, usually requiring investigation. 14 Peer Group – comprised of other laboratories that use the same or similar instruments and methods. 1 LAB D LAB E LAB C LAB F LAB A Your Lab’s CV LAB G N = 8 LAB B Peer CV 15 5 HILS1760 Effective Date: 12/06/2016 All‐lab Group – all instruments or methods used by participating laboratories to measure an analyte All‐lab CV Peer Group D Peer Group A Peer Group F Peer Group C Peer Group E Your Lab’s Peer Group N = 8 Peer Group B Peer Group G 16 Activity: Calculating SDI and CVI Purpose What will you do? Become familiar with how to Using simple math and working in calculate SDI and CVI. Evaluate pairs, your method’s performance in  relationship to the presence or Use the information supplied on absence of SDI or CVI flags. Worksheet 1 to calculate your laboratory’s SDI and CVI.  Complete Worksheet 1 to evaluate What will you need? your method’s performance. • Worksheet 1: Calculating SDI  Verify your responses with the pair and CVI located closest to you. • Calculation Device (e.g. phone,  calculator) Resolve any discrepancies between the pairs. • Pencils  Participate in Worksheet 1’s class discussion. 17 20 minutes Platelet Count (cells x 109/L) Sysmex XE‐ Low Control Normal Control High Control 2100 Your Lab Mean 56 214 545 SD 6 17 37 CV 10.7 7.9 6.8 My Lab 10.7/11.3 = 0.9 (Peer) CVI 7.9/8.8= 0.9 6.8/8.1 =0.8 My Lab (56‐62)/7 = ‐0.9 (Peer) SDI (214‐215)/19 = ‐0.1 (545‐496)/40 = 1.2 Peer Group Mean 62 215 496 SD 7 19 40 CV 11.3 8.8 8.1 N 12 12 12 TE in units of cells x 109/L ǀ 56‐62ǀ + 1.65 * 6 = 15.9 ǀ214 ‐ 215ǀ + 1.65 * 17 = 29.1 ǀ545 ‐ 496ǀ + 1.65 * 37 = 110.1 TEA in units 62 * 0.25 = 15.5 215 * 0.25 = 53.8 496 * 0.25 = 124.0 of cells x 109/L TE < TEA NO YES YES 18 6 HILS1760 Effective Date: 12/06/2016 Effective SQC Program  IQC and EQAS programs compliment each other  IQC monitors the daily precision and accuracy of processes, personnel, and instruments  EQAS maintains long‐term accuracy  EQAS Benefits  Demonstration of technical competence  Identify potential areas for improvement  Increase confidence in the accuracy of the laboratory’s testing results  Verification of the effectiveness of training 19 Summary Statistic • Captures our method’s performance relative to some measure of quality or peer data. • Examples TE, when compared to TEA TE < TEA SDI Sigma-metric CVI SEC 20 4 Key Numbers Mean ‐ Verify the mean value used in the calculation reflects current method performance. SD ‐ Verify the SD value used in the calculation reflects current method performance. True Value ‐ Record the source of your true value for each control and verify its validity. Total Allowable Error (TEA) ‐ Record the source of your TEA for each control and verify its validity. Ensure the limit you selected is attainable and defensible. 21 7 HILS1760 Effective Date: 12/06/2016 L-J Chart for Unicel DxC Chemistry Analyzer Analyte: Alkaline Phosphatase Control Material: Level 1 Units: mmol/L Lot #: 166101 Exp Date: 31/1/2015  assign 114 SD assign 4.0 From: December 1, 2012 Through: December 31, 2012 Target Value 107.3 TEa 12.04% 126.0 +3 SD 122.0 + 2 SD -------- t- C 118.0 o + 1 SD n t r 114.0 X o l 110.0 1 SD - V a l 106.0 2 SD ----- - u e 102.0 -3 SD 22 Cape Clinic Laboratory L-J Chart for Unicel DxC Chemistry Analyzer Analyte: Alkaline Phosphatase Control Material: Level 2 Units: mmol/L Lot #: 166102 Exp Date: 31/1/2015  assign 485 SD assign 10.0 From: December 1, 2012 Through: December 31, 2012 Target Value 457.3 TEa 12.04% 515.0 +3 SD 505.0 + 2 SD C 495.0 o + 1 SD n t r 485.0 X o l 475.0 1 SD - V a l 465.0 2 SD - u e 455.0 -3 SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Information: Current data meet quality specifications: Level 1 ________ Level 2 ___________ Level 3 __________ The target values are valid: Level 1 ___√_____ Level 2 _____√______ Level 3 __________ The TEa limits are valid: Level 1 ____√____ Level 2 _____√______ Level 3 __________ The current actual mean and SDI indicate a positive ___√_____ or negative ________ bias compared to the target ___√_____ cumulative _√_ or peer mean, beginning this month ___ or gradually since Oct/2012. (based on the last page of the handout) The current actual SD and CVI indicate increased impression compared to the assigned to chart ________ cumulative ________ or peer ________ SD, , beginning this month ________ or gradually since ___ /___/___. Patient results may be erroneously High __√______ Low ________ Imprecise ________ Action: ___√_____ Verify validity of all statistics used to generate within-laboratory and peer data ____√____ Calculate mean and SD on recent data population Re-assign mean on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign SD on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign target values: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign TEa limits: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Select QC rules and strategy to maximize error detection ________ or minimize false rejection ________ ________ ___√_____ Initiate corrective action to reduce bias Initiate corrective action to reduce imprecision ________ Arrange instrument service from manufacturer ________ ___√_____ Temporarily discontinue reporting patient results 24 8 HILS1760 Effective Date: 12/06/2016 Performing Corrective Action (10.1, 10,2, & 10.3)  Assign who performs the investigation and by when  Determine the root cause and extent of the problem by examining documents and records and talking with personnel  Log an occurrence report if erroneous results have been released. Investigation MUST include why daily QC processes failed to alert us when method performance no longer me t quality specifications.  Propose and implement measures to address or alleviate the immediate problem  Communicate changes and document  Propose and implement long‐term strategy(ies) to prevent the problem from reoccurring  Communicate changes and document  Revisit long‐term strategy effectiveness (e.g. yearly QMS review meeting with laboratory management, internal audit) Document this entir 25  e process Activity: Who’s Talking Now Purpose What will you do? You will assist a site with investigating Working in groups of 4 ‐5, you will: inter ‐laboratory flags on their ion ‐  selective electrode (ISE) Use Tool 3 and Handout 3 to complete methodologies. You will focus your Worksheet 3. investigation solely on calcium to  Perform an investigation of calcium to determine the root cause and determine the root cause of the ISE propose corrective action for the ISE problem. problem.  Provide a post ‐it note to the facilitator if What will you need? you would like to interview a staff • Handout 3: Calcium Report member. You only have time to interview 2 staff members. • Worksheet 3: Calcium Summary Worksheet 4: Calcium Investigation  Complete Worksheet 4 • • Envelope containing Tool 3: Calcium  Select a spokesperson Investigation Records  The spokesperson will present his/her • 2 post ‐it notes group’s responses for Worksheet 4 to the • People to Interview folders class during the debrief. • Calculation Device (e.g. phone, calculator) 26 90 minutes QC Flags: TE > TEa __ √ SEc < 2.0 __ √ SDI > ± 2.0 __________ CVI > 1.0 ____ √ ______ _________ ___ Controls Affected: Level 1 √ Level 2 ______ √ Level 3 __________ Other ____________ ___ _____ _____ Data & Calculations Relating to Four Key Numbers (in units) Level 1 Level 2 Level 3 Target 2.08 2.97 TEa 0.075 (0.036 x 2.08) 0.107 (0.036 x 2.97) Mean Assigned to chart 2.08 2.98 N Value N Value N Value Current actual 57 2.08 60 2.96 Cumulative 1074 2.08 1149 2.98 Peer Group Method 5718 (95) 2.08 5635 (95) 2.97 All-group Method 6265 (105) 2.08 6183 (105) 2.97 SD Assigned to chart 0.20 0.30 N Value N Value N Value Current actual 57 0.105 60 0.180 Cumulative 1074 0.036 1149 0.054 Peer Group Method 5718 (95) 0.043 5635 (95) 0.047 All-group Method 6265 (105) 0.043 6183 (105) 0.049 TE on current data 0.173 (0.105 x 1.65) 0.307(0.01+ 0.180 x 1.65) SEc on current data Less than 0 Less than 0 27 9 HILS1760 Effective Date: 12/06/2016 Information: Current data meet quality specifications: Level 1 ________ Level 2 ___________ Level 3 __________ The target values are valid: Level 1 ___√_____ Level 2 _____√______ Level 3 __________ The TEa limits are valid: Level 1 ____√____ Level 2 _____√______ Level 3 __________ The current actual mean and SDI indicate a positive __ or negative ________ bias compared to the target ________ _____ _ cumulative __ or peer mean_____, beginning this month ___ or gradually since ___ /___/___. The current actual SD and CVI indicate increased imprecision compared to the assigned to chart ________ cumulative ___√_____ or peer ___√_____ SD, , beginning this month ________ or gradually since approximately 23/11/2012. Patient results may be erroneously High ________ Low ________ Imprecise ___√_____ Action: ___√_____ Verify validity of all statistics used to generate within-laboratory and peer data ____√____ Calculate mean and SD on recent data population Re-assign mean on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign SD on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign target values: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign TEa limits: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Select QC rules and strategy to maximize error detection ________ or minimize false rejection ________ ________ Initiate corrective action to reduce bias ________ ____√____ Initiate corrective action to reduce imprecision Arrange instrument service from manufacturer ________ ___√_____ Temporarily discontinue reporting patient results 28 10 HILS1760 Effective Date: 12/06/2016