PEPFAR Quality Control and Method Validation Activity 5: Presentation Slides

Pitfalls encountered when assigned values do not reflect current observed values on our charts ACTIVITY: It Begins with the Right Chart 1 QC data points are a FACT If we do not like the value produced by the analytical system, then we must look to our processes, people, or tools. Facts are not ignored in a quality laboratory. 2 WHERE WE ARE - • Mean ( x ) X = the average of a set of values – primary indicator of accuracy n – measure of systematic error (error in a given – direction) • Standard deviation (SD) used to measure dispersion/scattering of a – group of values around a mean primary indicator of precision – measure of random error ( error in any – direction) 3 1 HILS1753 Effective Date: 12/07/2016 For Each QC Data Point We Need to Decide… SYSTEM IS STABLE SYSTEM IS UNSTABLE – SIGNIFICANT CHANGE NO RULE VIOLATION True Accept – system is stable and no QC False Accept – system is unstable and no rule violations occurred – report patients QC rule violations occurred –wrong results may be reported RULE VIOLATION OCCURRED ON L‐J CHART False Reject– system is stable and a QC True Reject – system is unstable and a QC rule violation occurred due to inherent rule violation occurred – begin investigation randomness‐ waste of time, effort, and $$ 4 and troubleshooting Label the Data Points Control Limit ‐ the defined limits or ranges expected due to random variation of the data ‐2SD +2SD Control Control points ‐‐ beyond those Limit Limit limits, some course of action involving investigation and troubleshooting are taken. a stable, single population = 5 Label the Data Points ‐2SD +2SD Control Control Limit Limit Original population when system was stable = New population as system undergoes a change = 6 2 HILS1753 Effective Date: 12/07/2016 1 SD Shift +4SD +3SD + 2SD + 1SD X -1SD 2SD - -3SD -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 7 2 SD Shift +4SD +3SD + 2SD + 1SD X -1SD 2SD - -3SD -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 8 Creating a L-J Chart 1)Label the chart 2)Label the X‐axis in terms of time period used or Run # 3)Scale the X‐axis into evenly sized increments numbering sequentially 4)Label the Y‐axis Control Value 5)Scale and label the Y‐axis from lowest to highest expected control values so that: 1) the mean is located at the center of your graph 2) the SD is the numerical value applied in an incremental fashion to determine ±1 SD, ±2 SD, ±3 SD, ±4 SD numerical values 6) Write the numerical values obtained for ±1 SD, ±2 SD, ±3 SD, ±4 SD next to the correct label on the chart. 7)Draw lines for mean and SDs 8)Begin plotting analyzed QC results. 9 3 HILS1753 Effective Date: 12/07/2016 Levey-Jennings Chart +3SD +2SD 95% 99% Mean +1SD 68% .1SD -2SD -3SD 10 1 On‐going Evaluation of Your Quality Control Systems Gaussian is the Key to Quality Control Alerts us to changes in Monitors and evaluates accuracy and precision method performance EQA TEA MN We use L‐J charts and rule violations to detect changes in accuracy and precision to alert us to a medically significant error. Activity: It Begins with the Right Chart– Part I Assigned = Observed Purpose What will you do? • To perform an overview Working in pairs, you will analysis of the method’s  Complete Worksheets 1, 2, and 3 (60 minutes) current performance in  order to determine if a Calculate the control limits and write the change in accuracy or numerical values on the Y‐axis precision has occurred.  Plot the missing points  Answer the questions What will you need?  Apply all the single rules we have learned Working within your group of 4‐5 participants • Worksheet 1: L‐J Chart 1  1 hour into the activity, your group will be • Worksheet 2: L‐J Chart 2 Worksheet 3: L‐J Chart 3 assigned a chart to review with the class •  • Pencils Using flipchart and markers, create the • Flipchart necessary charts and information (20 minutes)  Select a spokesperson • Markers  You will have 5 minutes to provide a review to the class 80 minutes 12 4 HILS1753 Effective Date: 12/07/2016 Cape Clinic Laboratory 1 Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 90 - SD assigned 3 102.0 +4SD 99.0 +3SD 96.0 + 2SD 93.0 + 1SD 90.0 X 87.0 -1SD 84.0 2SD - 81.0 -3SD 78.0 -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 97 90 91 95 90 85 91 94 89 91 85 89 Mean observed SD observed Median Mode (umol/L) (umol/L) (umol/L) (umol/L) Run # 1‐30 (stable population) 90.2 3.2 90 90 13 1 99.7% 95% 68% 2.5% 13.5% 34% 34% 13.5% 2.5% -4SD -3SD -2SD -1SD 1SD 2SD 3SD 4SD 14 Cape Clinic Laboratory 1 Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 90 SD assigned 3 102.0 +4SD 99.0 +3SD 96.0 + 2SD 93.0 + 1SD 90.0 X 87.0 -1SD 84.0 2SD - 81.0 -3SD 78.0 -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 97 90 91 95 90 85 91 94 89 91 85 89 15 5 HILS1753 Effective Date: 12/07/2016 2 Cape Clinic Laboratory Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 90 SD assigned 3 +4SD 102.0 +3SD 99.0 96.0 + 2SD + 1SD 93.0 X 90.0 -1SD 87.0 2SD - 84.0 -3SD 81.0 -4SD 78.0 Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 95 90 96 91 87 82 96 85 88 101 82 91 Mean observed SD observed (umol/L) (umol/L) Run # 1‐20 (stable population) 90.2 3.1 Run # 21‐30 (emerging change population) 89.9 6.3 Run # 1‐30 (mixed population) 90.1 4.3 16 2 A new population of data points is emerging due to a change in precision 17 2 Cape Clinic Laboratory Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 90 SD assigned 3 +4SD 102.0 +3SD 99.0 96.0 + 2SD + 1SD 93.0 X 90.0 -1SD 87.0 2SD - 84.0 -3SD 81.0 -4SD 78.0 Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 95 90 96 91 87 82 96 85 88 101 82 91 18 6 HILS1753 Effective Date: 12/07/2016 3 Cape Clinic Laboratory Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 90 SD assigned 3 102.0 +4SD 99.0 +3SD 96.0 + 2SD 93.0 + 1SD 90.0 X 87.0 -1SD 84.0 2SD - 81.0 -3SD 78.0 -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 85 90 93 88 87 89 96 92 95 88 90 89 87 92 90 94 88 90 84 85 86 81 86 92 89 86 82 89 86 Mean SD observed observed (umol/L) (umol/L) Run # 1‐20 (stable population) 90.0 3.2 Run # 21‐30 (emerging change population) 86.2 3.3 Run # 1‐30 (mixed population) 88.7 3.7 19 3 A new population of data points is emerging due to a SMALL change in accuracy 20 3 Cape Clinic Laboratory Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 90 SD assigned 3 102.0 +4SD 99.0 +3SD 96.0 + 2SD 93.0 + 1SD 90.0 X 87.0 -1SD 84.0 2SD - 81.0 -3SD 78.0 -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 85 90 93 88 87 89 96 92 95 88 90 89 87 92 90 94 88 90 84 85 86 81 86 92 89 86 82 89 86 21 7 HILS1753 Effective Date: 12/07/2016 Assigned ≠ Observed “Theoretically, laboratory QC is easy. Simply calculate the mean and SD for a set of data and use one or more QC rules to warn when data fall outside the acceptable range. Unfortunately, many pitfalls exist that may lead to incorrect conclusion about method performance.” Brooks, Zoe C. Performance‐Driven Quality Control, 2001, p 70. 22 Reasons for more data points to lie farther from the mean  A change in accuracy or precision • When a change occurs in the mean or SD, the control measurements after that change will not be part of the same population of measurements used to assign the mean and SD on the L‐J chart • More than 5% of the measurements will exceed ± 2SD  The mean and/or SD are not assigned correctly on the L‐J chart • Assigned values ≠ Observed values • More than 5% of the measurements will exceed ± 2SD Assigned Mean ≠ Observed Mean Rendering our QC rules ineffective. 24 8 HILS1753 Effective Date: 12/07/2016 ------ SD obs -- -- --- MEAN obs MEAN ASSIGNED Remember?? Control Rule Nomenclature A, A:L or A = number of control measurements involved L control limits often expressed as = the mean assigned to the L‐J chart ± a multiple of the SD 26 +3s +2s +1s Mean 135 rule Xassign -1s 1 -25 violation 2 -3s 3 1 2 3 4 5 6 7 8 9 10 27 9 HILS1753 Effective Date: 12/07/2016 Assigned SD ≠ Observed SD Rendering our QC rules ineffective. 28 A MEAN obs SD obs SD assigned Activity: It Begins with the Right Chart– Part II Assigned ≠ Observed Purpose What will you do? • To understand how our QC Working in pairs, you will rules are rendered ineffective  Complete Worksheets 4, 5, and 6 (60 minutes). when the wrong values are  assigned to the chart. Determine the assigned values and observed values for the mean and SD  Draw the Gaussian distributions on the chart What will you need?  Answer the questions • Worksheet 4: L‐J Chart 4 Working within your group of 4‐5 participants • Worksheet 5: L‐J Chart 5  60 minutes into the activity, your group will be • Worksheet 6: L‐J Chart 6 assigned a chart to review with the class • Pencils  Using flipchart and markers, create the necessary • Flipchart charts and information (20 minutes) • Markers  Select a spokesperson  You will have 5 minutes to provide a review to the class 80 minutes 30 10 HILS1753 Effective Date: 12/07/2016 Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 93 SD assigned 3 X observed 90 SD observed 3 105.0 +4SD 102.0 +3SD 99.0 + 2SD False Accept 96.0 + 1SD 93.0 X 90.0 True Accept -1SD 87.0 2SD - 84.0 -3SD 81.0 -4SD False Reject Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 97 90 91 95 90 85 91 94 89 91 85 89 It’s a Fact – this is This is WHERE WE WHERE WE ARE EXPECTED TO BE – based on the assigned values of the chart False False Rejects Accepts Mean Mean observed assigned 32 Cape Clinic Laboratory 4 Assigned 87 SDassigned 3 X observed 90 SD observed 3 alse Reject EI +250 21 +19 True Accept LI -250 False Accept 450 3 5 17 ......... Value 93 90 92 94 90 91| 90 94 85 97 91 91 33 11 HILS1753 Effective Date: 12/07/2016 4 False False Rejects Accepts True Accepts Mean Mean assigned observed 34 5 Cape Clinic Laboratory Creatinine (wo H.) L. J Chartfor CantrelXYZ(cap 38/11/XX) on Ilustra Chemistry Analyzer(serial#123) Xassigned 90 SD assigned 6 Xobserved 90 SDobserved 3 False Accept True Accept H1 45D ......... ........ 35D ......... ........................ 45D ................ ........ 2 3 5 1 21 22 33 44 7 31 ........ ....... 35 .................... 5 False False Accepts Accepts True Accepts WHERE WE ARE SD observed SD assigned 36 12 HILS1753 Effective Date: 12/07/2016 6 Cope Chui Laboratory Creatinine (unoH.) L. J Chart for CentralXYZ (ap 30/11/XX) on Illustra Chemistry Analyzer (serial#123) Xassigned 90 SD assigned 2 Xobserved 90_ 0 observed 3 LI +450 False Reject True Accept 450 3 12 21 72 23 26 ......... ........ ......... ......... ...... ....... .......... ....... ............... .......... 37 6 False False Rejects Rejects True Accepts SD observed SD assigned 38 Assigned vs. Observed – What is the difference? Mean assign SD assign Mean observed SD observed • May or may not be based on • Mathematically fact calculated using data • Values applied to the L‐J Chart points (facts) • Sources often chosen other than • Answers WHERE WE ARE currently observed performance in terms of accuracy and include: precision of the method • Past data points before a • Specific to your method’s change performance • Cumulative values 39 • Package insert values 13 HILS1753 Effective Date: 12/07/2016 When Do You Create a New LJ Chart? • Lot changes Controls (parallel testing) – Reagents – Reference solutions – • Calibration Expected (part of instrument maintenance – schedule) Unexpected (a problem during testing is – corrected) • Major component changed on the instrument • Change in the testing process, usually manufacturer‐directed. 40 How can we determine if a change is acceptable or not? Alerts us to changes in Monitors and evaluates accuracy and precision method performance TEa EQA Target TEa MN Total Error We us L‐J charts and rule Total Error (TE) < Total violations to alert us to a Allowable Error (TEA) medically significant error. Select appropriate rules 14 HILS1753 Effective Date: 12/07/2016