PEPFAR Quality Control and Method Validation Activity 8: Presentation Slides

Finally, defining a significant change ACTIVITY: Putting the TEA into Quality 1 For TE, you will need 3 Key Numbers Mean ‐ fact Absolute Bias for SE S SD ‐ fact for RE True Value (Target Value) best estimate – 2 Activity: How Correct are the Test Results? Purpose What will you do? Based on the supplied QC chart, your Working in groups of 4, you will: group will determine if the method  Begin with Worksheet 1. can produce test results that are  clinically acceptable for the intended Participate in your group’s use of the test. discussion and reach a consensus.  Complete the appropriate section What will you need? on Worksheet 1 and proceed to • Worksheet 1: Intended Use the next worksheet as directed. • Worksheets 2A – 4: Intended Use  Continue working on your • Job Aid 1 directed worksheets until your • Calculation Device (e.g. phone, group reaches Job Aid 1. calculator) • Pencils 3 45 minutes 1 HILS1756 Effective Date: 12/06/2016 TE < TEA TE is the total TEA specifies the variation of our maximum acceptable value from the true variation from the value true value. 4 On‐going Evaluation of Your Quality Control Systems Gaussian is the Key to Quality Control Alerts us to changes in Monitors and evaluates accuracy and precision method performance NS EQA TEA MN Determine if the system is stable or undergoing a Defines an acceptable change by comparing today’s change vs. observed value with the past unacceptable change history of previous results (TE < TEA) For Quality, you will need 4 Key Numbers Mean ‐ fact SD ‐ fact True Value (Target Value) best estimate – Total Allowable Error (TEA) best estimate – 6 2 HILS1756 Effective Date: 12/06/2016 Name of the Laboratory Name and Identifier # of Instrument Analyte: Control Material: Units: Lot #: Exp Date:  assign SD assign From: Through: Target TEa +4 SD +3 SD C o + 2 SD n t r + 1 SD o l X V 1 SD - a l u 2 SD - e -3 SD +4 SD Run # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Date Value Initals Document # 7 “If you cannot specify the quality requirement, there is no point to quality control” Dr. James Westgard 8 Stockholm Hierarchy (of Desirability) Sources in descending order of priority: 1.Medical Decision Levels 2.Biological Variation 3.Proficiency Testing Guidelines 4.Proficiency Test Results 5.Tonk’s Rule 6.Current SD x 3 9 3 HILS1756 Effective Date: 12/06/2016 Biological Variation (BV) Within‐subject biological variation Also known as intra‐individual or CVI % 10 Biological Variation (BV) Between‐subject biological variation Also known as inter‐individual or CVG% 11 Biological Variation (BV) Three Levels of quality specifications  Desirable  Most frequently used level for BV  If the desirable level ‘s %TEA is unattainable, then use the minimum level  If the desirable level ‘s %TEA is too large or wide, then use the optimal level  Minimum Optimal Analyte Desirable Specification Minimum Specification Specification  Optimal CV(%) Bias (%) %TEa %TEa %TEa Glucose 2.9 2.2 6.9 11.8 3.9 Potassium 2.4 1.8 5.8 8.7 2.9 12 4 HILS1756 Effective Date: 12/06/2016 Proficiency Testing Guidelines Performance goals set by the organizers of external proficiency assessment programs (e.g. CLIA in the USA) or in ‐country regulatory bodies. CLIA’s analytical quality requirements are presented in four different ways, depending upon the analyte: 1. Percentage Chloride Target Value ± 5% 2. Absolute concentration limit Potassium Target Value ± 0.5 mmol/L 3. Distribution of a survey group TSH Target Value ± 3SD 4. Combination of requirements #1 and #2 (for lower concentrations) Glucose Target Value ± 0.3 mmol/L or ± 10%, whichever is greater 13 Step-wise Approach from a Leading Expert i 1.Gather mean & SD data 2.Estimate true value 3.Calculate Total Error 4.Research potential sources of TEa based on Stockholm hierarchy 5.Select best "attainable and defensible" TEa limit o Select Professional Expert Publications, if available o Select Biological variation ‐ if easily attainable AND less than PT limits o Select PT limits ‐ if available and BV is unattainable o Select Proficiency Testing (PT) Survey CV% x 3 only if the above sources are unavailable o Use Tonk's rule only if the above sources are unavailable o Select 3x your median historical internal SD if other sources unavailable 6.Choose a defensible TEa that allows a comfortable margin for error 14 i From Brooks, Zoe at http://www.awesome ‐numbers.org and sign up for a free account Laboratory Applications Comparing TE to TEA 1.Assess method performance 2.Check calibrations 3.Evaluate the acceptability of new reagent lot numbers 4.Evaluate changes to the analytical system 5.Determine the size of change that must be detected 6.Select appropriate QC rules 15 5 HILS1756 Effective Date: 12/06/2016 ¯ x Target Value Target ‐ TEA Target + TEA Lower TEA Upper TEA Limit Limit From which point do we calculate our total allowable error limits? 16 True Value Right  Target True Value  Best estimate of the sample’s true value  The right answer TEA TEA Lower Limit Upper Limit Wrong Wrong Close enough to true to be considered true 17 TEA is 9% ¯ Calculate the x error limits. Target Value Target ‐ TEA Target + TEA Lower TEA Upper TEA Limit Limit 91 96 100 109 mmol/L mmol/L mmol/L mmol/L 18 6 HILS1756 Effective Date: 12/06/2016 TEA is 10% ¯ Calculate the x error limits. Target Value Target ‐ TEA Target + TEA Lower TEA Upper TEA Limit Limit 180 195 200 220 mmol/L mmol/L mmol/L mmol/L 19 TEA is ± 4mmol/L ¯ Calculate the x error limits. Target Value Target ‐ TEA Target + TEA Lower TEA Upper TEA Limit Limit 126.0 130.0 131.0 134.0 mmol/L mmol/L mmol/L mmol/L 20 TEA is 10% ¯ x Calculate the Target error limits. Value Lower TEA Upper TEA Limit Limit 87 100 mmol/L 90 110 mmol/L mmol/L mmol/L 21 7 HILS1756 Effective Date: 12/06/2016 Sample (pmol/L) Mean SD %CV C-1 10.9 1.1 10.0% C-2 43.1 4.2 9.7% C-3 21.3 2.0 9.4% C-4 51.0 5.3 10.3% C-5 56.7 6.1 10.8% The CLIA TE A limit for free thyroxine (FT4) is Target ± 3SD What TE will you use with a target value of A 24.0 pmol/L? 2.0 pmol/L x 3 6 pmol/L = (6/24) x 100% =25% What are your lower and upper TE A limits? 24.0 – 6 = 18 24.0 + 6 = 30.0 22 Sample (cell/uL) Mean SD %CV C-1 786.4 32.2 4.1% C-2 56.0 2.8 5.0% C-3 491.3 27.5 5.6% C-4 103.4 6.0 5.8% C-5 292.8 14.1 4.8% You decide to use a past survey report to derive a TE % for A CD4 cell count since there is no other published source available. What TE % will you use? 5% x 3 = 15% A For a target of 100 cell/uL , what are your lower and upper TE A limits? 15% of 100 15 cells/uL; 85 and 115 cell/uL = 23 Activity: Calculating TE A Limits Purpose What will you do? To determine the error limits Using simple math and working in based on a provided quality pairs, requirement goal and target  Complete Worksheet 5 (25 minutes). value.  Verify your responses with the pair What will you need? located closest to you.  Resolve any discrepancies between • Worksheet 5: Calculating TE A the pairs. Error Limits  • Job Aid 2: Sample TE Participate in Worksheet 5’s class A Calculations discussion. • Calculating Device (e.g. calculator, phone) • Pencils 24 30 minutes 8 HILS1756 Effective Date: 12/06/2016 Worksheet 5: Calculating TEA Limits QC Material Target Value TEA in units Lower TEA Upper TEA Limit Limit Level I 100 mmol/L 5 mmol/L 95 mmol/L 105 mmol/L Chloride Level II 120 mmol/L 6 mmol/L 114 mmol/L 126 mmol/L Level I 3.5 mmol/L 0.5 mmol/L 3.0 mmol/L 4.0 mmol/L Potassium Level II 5.8 mmol/L 0.5 mmol/L 5.3 mmol/L 6.3 mmol/L Level I 0.9 mU/L 0.24 mU/L 0.66 mU/L 1.14 mU/L TSH Level II 4.5 mU/L 0.99 mU/L 3.51 mU/L 5.49 mU/L Level I 2.78 mmol/L 0.3 mmol/L 2.48 mmol/L 3.08 mmol/L Glucose Level II 5.72 mmol/L 0.57 mmol/L 5.15 mmol/L 6.29 mmol/L Level I 50 cell/uL 9 cell/uL 41 cell/uL 59 cell/uL CD4 Level II 200 cell/uL 36 cell/uL 164 cell/uL 236 cell/uL Level III 700 cell/uL 126 cell/uL 574 cell/uL 826 cell/uL 25 Activity: Changing Reagent Lot Numbers Purpose What will you do? TE calculations can be applied to Working in groups of 4, you will: new reagent lots to evaluate  acceptability before putting the Complete Worksheet 6 (30 minutes) new lot into use with patient  Select a spokesperson to present the testing. group’s answers.  Participate in the class discussion What will you need? Plenary Discussion • Worksheet 6: Reagent Lot Number Change  One group will be selected at random • Job Aid 2: Sample TEA to present their answers. Calculations  The populated chart will be projected • Calculating Device (e.g.  The spokesperson will address calculator, phone) Worksheet 6’s questions • Pencil • Glue stick • Scissor 40 minutes 26 Worksheet 6: Reagent Lot Number Change Current Mean 5.80 Current SD 0.10 Target Value 5.60 Bias 0.20 Absolute Bias 0.20 Current TE 0.37 TEA 0.39 Reagent Lot Number A Reagent Lot Number B New Reagent Lot Number Mean 5.73 5.84 Current SD 0.10 0.10 Target Value 5.60 5.60 Bias 0.13 0.24 Absolute Bias 0.13 0.24 New Reagent Lot Number TE 0.30 0.41 TEA 0.39 0.39 TE < TEA (yes or no) Yes No 27 9 HILS1756 Effective Date: 12/06/2016 X X 5.73 5.84 Reagent A Reagent B Lower TEA Limit Target Value 5.80 Upper TEA Limit 5.21 5.60 5.99 ISO 15189 Standard: “Performance specifications for each procedure used in an examination shall relate to the intended use of that procedure.” ISO 15189: 5.5 29 ISO 15189 in QC Workshop Language Standard: WHERE WE ARE for each procedure used in an examination shall relate to WHERE WE WANT TO BE for that procedure. 30 10 HILS1756 Effective Date: 12/06/2016