PEPFAR Quality Control and Method Validation Activity 7: Presentation Slides

Totally, WHERE ARE WE? ACTIVITY: Total Error (TE) 1 On‐going Evaluation of Your Quality Control Systems Gaussian is the Key to Quality Control Alerts us to changes in Monitors and evaluates accuracy and precision method performance NS EQA TEA MN To evaluate our method’s performance we must compare WHERE WE ARE to WHERE WE WANT TO BE WHERE WE ARE Accuracy Precision Mean ( x ) Standard deviation (SD) used to measure the average of a dispersion/scattering of a group of set of values values around a mean primary indicator primary indicator of precision of accuracy measure of random error (error in any direction) measure of Coefficient of Variation (CV) systematic error (error in a the great statistical equalizer given direction) ratio often expressed in % estimates the SD at the concentration level of interest 3 1 HILS1755 Effective Date: 12/06/2016 WHERE WE EXPECT TO BE Stable population Future points lie with the same probability of past results 4 Criteria Needed for SQC Gaussian Distribution • symmetrical • clusters around the mean • 68‐95‐99 rule Single Population Assigned = Observed 5 Random Error (change)  Inconsistent change in the analytical system  Error in any direction  A change in precision  Changes in SD and CV  Rules that look at the tails of the distribution 6 2 HILS1755 Effective Date: 12/06/2016 Systematic Error (change)  Consistent change in the analytical system  Error in a given direction  A change in accuracy  A shift in the observed mean value  A change in bias  Rules that look for consecutive control measurements exceeding the same control limit 7 Accuracy and Precision Inaccurate but Precise Inaccurate and Imprecise SD BIAS Accurate and Precise 8 WHERE WE ARE Bias = x -True Value ¯  Consistent change in the analytical system  Error in a given direction (negative or positive number) Absolute Bias =| x True Value| ¯ -  Used in calculations for TE  Positive number  Sometimes TE equations do not specify an absolute bias is being used. 9 3 HILS1755 Effective Date: 12/06/2016 Bias = x -True Value ¯ x True Value True Value of the XYZ QC material is 86 umol/L 86 umol/L 90 umol/L Bias of + 4 umol/L 10 Bias = x - True Value ¯ ¯ x x True Value True Value of the HWD QC material is 110 umol/L 103 umol/L 107 umol/L 110 umol/L Before Calibration Bias of ‐ 3 umol/L After Calibration Bias of ‐ 7 umol/L 11 Question??? True or False The mean assigned on the L‐J chart is the target value? 12 4 HILS1755 Effective Date: 12/06/2016 TE = SE + RE Total Systematic Random Error Error Error 13 ¯ TE = SE + RE x True Value TE is the total variation of our value from the true value Bias RE SE TE 14 ¯ TE decreases if x True • SE decreases Value • Bias decreases • Mean moves closer to the true value Bias RE SE TE 15 5 HILS1755 Effective Date: 12/06/2016 ¯ TE decreases if x True • RE decreases Value • Imprecision decreases • SD becomes smaller Bias RE SE TE 16 For TE, you will need 3 of the 4 SQC Key Numbers Mean ‐ fact Absolute Bias for SE S SD ‐ fact for RE True Value (Target Value) best estimate – 17 Activity: Total Error (TE) Purpose What will you do? To calculate the TE in units and Using simple math and working in • in percent. From the pairs, calculation, analyze and discuss  the information presented on Complete Worksheet 1 (20 minutes). each worksheet  Verify your responses with the pair located closest to you.  What will you need? Resolve any discrepancies between the pairs. • Worksheet 1: Calculate TE  Participate in Worksheet 1’s class • Worksheet 2: Impact of Bias discussion. and Imprecision on TE  • Calculation Device (e.g. phone, Complete Worksheet 2 in the same calculator) manner (20 minutes).  • Pencil Illustrate the TE for Worksheet 2.  Participate in Worksheet 2’s class discussion. 65 minutes 18 6 HILS1755 Effective Date: 12/06/2016 TE = SE + RE Total Systematic Random Error Error Error TE = l Bias l + Z * SD Z Factor  Units for measuring data in terms of SD units  Determine the portion of the population of data points to be included in the estimation of TE  Common multipliers with the SD are: 2 – commonly used for quick calculations 1.96 – includes 97.5% probability 1.65 – includes 95% probability 19 TE = SE + RE = Bias + 1.65 SD TE is the total variation from the target value for 95% of the sample population True or Target Value Bias (SE) 1.65 SD (RE) 5% -4SD -3SD -2SD -1SD 1SD 2SD 3SD 4SD XI k 1.65 SD 20 TE = SE + RE = Bias + 1.96 SD TE is the total variation from the target value for 97.5% of the sample population True or Target Value Bias (SE) 1.96 SD (RE) 2.5% .4SD -3SD -2SD -1SD X 1SD 2SD 3SD 4SD 1.96 SD 21 7 HILS1755 Effective Date: 12/06/2016 3 Key ¯ x Numbers True Mean 80 U/L Value True Value 100 U/L SD 5 U/L SE ?? RE ?? TE ?? 80 U/L 100 U/L RE (8.3 U/L) SE (20 U/L) TE (28.3 U/L) 22 Worksheet 1: Calculate TE Analyte Bias l Bias l TE in units %CV Abs Bias % TE in % (abs bias) ‐True Value | ‐True Value| Abs Bias + 1.65*SD (SD /x)*100% (Abs Bias /True Value) *100% Abs Bias % + 1.65* %CV WBC 18.0 – 18.2 = 0.2 0.2 + 1.65 *1.2 (1.2 /18.0)*100% (0.2 / 18.2)* 100% 1.1% + 1.65 *6.7% ‐0.2 = 2.2 = 6.7% =1.1% =12.2% Potassium 3.8 – 3.5 0.3 0.3 + 1.65 *0.1 (0.1 /3.8)*100% (0.3 / 3.5)* 100% 8.6% + 1.65 *2.6% = 0.3 =0.5 = 2.6% =8.6% =12.9% Creatinine 90 – 90 0 0 + 1.65 *4 (4 /90)*100% (0.0 / 90)* 100% 0% + 1.65 *4.4% = 0 = 6.6 = 4.4% =0% =7.3% Platelet 160 – 150 10 10 + 1.65 *7 (7 /160)*100% (10 / 150)* 100% 6.7% + 1.65 *4.4% = 10 = 21.6 = 4.4% =6.7% =14.0% Glucose 6.5 –6.7= 0.2 0.2 + 1.65 *0.2 (0.2 /6.5)*100% (0.2 / 6.7)* 100% 3.0% + 1.65 *3.1% ‐0.2 = 0.5 = 3.1% =3.0% =8.1% Calcium 2.26 – 2.25 0.01 0.01 + 1.65 *0.03 (0.03/2.26)*100% (0.01 / 2.25)* 100% 0.4% + 1.65 *1.3% = 0.01 =0.06 = 1.3% =0.4% =2.5% 23 Worksheet 2: Impact of Bias and Imprecision on TE Bias l Bias l TE in units TE in % %CV Abs Bias % TE in % (abs bias) Analyte ‐True Value | ‐True Value| Abs Bias + 1.65*SD (TE /True Value) (SD /x)*100% (Abs Bias /True Value) *100% Abs Bias % + 1.65* %CV *100% Lab A 150 – 150 0 0 + 1.65 *5 (8.3 / 150) (5 /150)*100% (0 / 150)* 100% 0% + 1.65 *3.3% = 0 = 8.3 *100% = 3.3% =0% =5.4% =5.5% Lab B 145 – 150 5 5 + 1.65 *5 (13.3 / 150) (5 /145)*100% (5 / 150)* 100% 3.3% + 1.65 *3.4% = ‐5 = 13.3 *100% = 3.4% =3.3% =8.9% =8.9% Lab C 150 – 150 0 0 + 1.65 *10 (16.5 / 150) (10 /150)*100% (0 / 150)* 100% 0% + 1.65 *6.7% = 0 = 16.5 *100% = 6.7% =0% =11.1% =11.0% Lab D 145 – 150 5 5 + 1.65 *10 (21.5 / 150) (10 /145)*100% (5 / 150)* 100% 3.3% + 1.65 *6.9% = ‐5 = 21.5 *100% = 6.9% =3.3% =14.7% =14.3 24 8 HILS1755 Effective Date: 12/06/2016 True _ True x value Laboratory A Value aboratory B 150 145 150 X RE RE SE RE TE TE TE True True Laboratory C value Laboratory D 150 45 150 RE X SE TE TE 25 TE WHERE ARE WE? cs - WHERE DO WE WANT TO BE? 26 9 HILS1755 Effective Date: 12/06/2016