PEPFAR Quality Control and Method Validation Activity 9: Presentation Slides

Quantifying wiggle room ACTIVITY: How Far Can Your Mean Shift? 1 ISO 15189 Standard: “Performance specifications for each procedure used in an examination shall relate to the intended use of that procedure.” ISO 15189: 5.5 2 ISO 15189 in QC Workshop Language Standard: WHERE WE ARE for each procedure used in an examination shall relate to WHERE WE WANT TO BE for that procedure. 3 1 HILS1757 Effective Date: 12/06/2016 Right TE < TEA Result True Value TEA TEA Lower Limit Upper Limit Wrong Wrong Result Result X Close enough to true to be considered true 4 RAY MOUR 5 Margin for Error = TEA ‐ TE ME Significant Medical Error 6 2 HILS1757 Effective Date: 12/06/2016 Critical Systematic Error (ΔSEc) ΔSEc =[(TEA‐ǀ Biasobsǀ)/SDobs] – z factor WHERE WE WHERE WE ARE WANT TO BE Z factor ‐ defines the tail of the Gaussian distribution that exceeds TEA limit Dr. Westgard uses a z factor of 1.65 7 Critical Systematic Error (ΔSEc) ΔSEc = [(TEA‐ǀ Biasǀ)/SD] – 1.65 TEA (Mean – Target Value) SD The The 4 Key Numbers in 1 Equation ation 8 ± 3 includes 99.7% of our data ± 2 includes 95% of our data ± 1 includes 68% of our data ± 1.65 includes 90% of our data 5% 1.65 SD 1.65 SD 5% -4SD -3SD -2SD -1SD 1SD 2SD 3SD 4SD -1.65 SD 1.65 SD 9 3 HILS1757 Effective Date: 12/06/2016 If our current mean shifts so that the 1.65 SD is at the error limit, how much of our data measurements will exceed the TEA limit? ¯ x SHIFT 1.65 SD Before change After change Lower Upper TEA Limit TEA Limit 10 What is the ΔSEc? Meanobs = 16 mmol/L Answer SDobs = 2 mmol/L 1.35 Target Value = 12 mmol/L TEA = 10 mmol/L ΔSEc = [(TEA‐ǀ Biasǀ)/SD] – 1.65 11  assign 16 mmol/L SD assign 2 mmol/L Target Value 12 mmol/L TEa 10 mmol/L 24 +4 SD 22 +3 SD Upper TEA Limit 20 + 2 SD ---------- 18 + 1 SD 1.35 16 X 14 1 SD - 12 2 SD Target Value - 10 -3 SD 8 -4 SD 6 -5 SD 4 -6 SD 2 -7 SD Lower TEA Limit 12 4 HILS1757 Effective Date: 12/06/2016 What is the ΔSEc? Meanobs = 15 mmol/L SDobs = 3 mmol/L Answer = 2.35 Target Value = 18 mmol/L TEA = 15 mmol/L ΔSEc = [(TEA‐ǀ Biasǀ)/SD] – 1.65 13  assign 15 mmol/L SD assign 3 mmol/L Target Value 18 mmol/L TEa 15 36 +6 SD 33 +5 SD Upper TEA Limit 37 +4 SD 24 +3 SD 21 + 2 SD 18 + 1 SD Target Value 15 X 12 1 SD - 2.35 9 2 SD - 6 -3 SD 3 -4 SD Lower TEA Limit | | | | | | | | | | | 14 What is the ΔSEc? Meanobs = 23 mmol/L SDobs = 3 mmol/L Answer = 3.35 Target Value = 28 mmol/L TEA = 20 mmol/L ΔSEc = [(TEA‐ǀ Biasǀ)/SD] – 1.65 15 5 HILS1757 Effective Date: 12/06/2016  assign 23 mmol/L SD assign 3 mmol/L Target Value 28 mmol/L TEa 20 50 +9 SD Upper TEA Limit 47 +8 SD 44 +7 SD 41 +6 SD 38 +5 SD 35 +4 SD 32 +3 SD 29 + 2 SD Target Value 26 + 1 SD 23 X 20 1 SD - 17 2 SD 3.35 - 14 -3 SD 11 -4 SD 8 -5 SD Lower TEA Limit 4 -6 SD 16 What is the ΔSEc? Meanobs = 55 mmol/L SDobs = 3 mmol/L Answer = 5.35 Target Value = 56 mmol/L TEA = 22 mmol/L ΔSEc = [(TEA‐ǀ Biasǀ)/SD] – 1.65 17  assign 55 mmol/L SD assign 3 mmol/L Target 56 mmol/L TEa 22 mmol/L 79 +8 SD Upper TEA Limit 76 +7 SD 73 +6 SD 70 +5 SD 67 +4 SD 64 +3 SD 61 +2 SD 58 + 1 SD Target Value 55 X 52 1 SD - 49 2 SD - 46 -3 SD 5.35 43 -4 SD 40 5 SD - 37 6 SD +- +- +-+ ------------ - 34 -7 SD Lower TEA Limit 31 -8 SD 28 -9 SD 18 6 HILS1757 Effective Date: 12/06/2016 Critical Systematic Error (ΔSEc) • Combines the 4 Key Quality Numbers into one statistic • Indicates how far the mean can shift before quality performance requirements are exceeded (maximum risk of 5% of a wrong test result with a z factor of 1.65) • Can be used to select appropriate control rules for a method 19 TE vs. ΔSEc • Indicates how far • Indicates how far the results are from mean can shift before the target value quality performance requirements are exceeded • Indicates how far • Indicates how far WHERE WE ARE from WHERE WE ARE from WHERE WE can shift before WANT TO BE crossing the TEA limits 20 Sigma-metric Sigma= [(TEA‐ǀ Biasobsǀ)/SDobs] ΔSEc = [(TEA‐ǀ Biasobsǀ)/SDobs] – 1.65 ΔSEc = Sigma – 1.65 Sigma = ΔSEc + 1.65 21 7 HILS1757 Effective Date: 12/06/2016 Sigma-metric • Combines the 4 Key Quality Numbers into one statistic • Measures the multiples of SD that fit within TE A limits • Benchmarks the performance of the measurement procedure in relationship to the quality required (i.e. Sigma Scale) • Can be used to select appropriate control rules for a method 22 Sigma-metrics Rule Selection Guide  6 sigma ‐ any QC will do (just don’t use 2 SD limits)  5 sigma ‐ single ‐rules such as 1:3s or 1:2.5s  4 sigma ‐ multirules  < 4 sigma ‐ multirules with look ‐back to previous runs, increase the number of controls analyzed  3 or less ‐ look for better analytical methods From www.westgard.com/links ‐to ‐india ‐part i ‐best ‐qc ‐practices ‐and ‐westgard ‐rules.htm. Last accessed on March 5, 2013. ‐ 23 Key Points Regarding Sigma  Relationship exists with SEc (SEc + 1.65) = Sigma – 1.65 or Sigma = SEc  Sigma ‐metric o 6 excellent – easy to control using SQC o 5 good o 4 fair o 3 poor – must improve method performance  Can identify and target laboratory resources on poorly controlled measurement procedures24 8 HILS1757 Effective Date: 12/06/2016 Activity: Calculating ΔSEc Purpose What will you do? Calculate the ΔSEc and Sigma Using simple math and working in ‐ metric using the four key numbers pairs, supplied in the table. Suggest a  Use the information supplied on control rule based on the provided Sigma ‐metric Rule Worksheet 1 to calculate the ΔSEc and Selection Guide. Sigma ‐metric.  Use the Sigma ‐metric Rule Selection What will you need? Guide provided next to the table to suggest an appropriate control rule. • Worksheet 1: Calculating SEc  and Sigma Verify your responses with the pair located closest to you. • Calculation Device (e.g. phone,  calculator) Resolve any discrepancies between the pairs. • Pencils  Participate in Worksheet 1’s class discussion. 25 20 minutes l Bias l TE in units TE A in units SEc Sigma QC Rules Based on the Sigma ‐metric [(TE A ‐ ǀBias ǀ)/SD] Analyte: | ‐True Value| Abs Bias + 1.65*SD (TE A %/100%)* Target Value [(TE A ‐ ǀBias ǀ)/SD] – 1.65 OR Platelet SEc + 1.65 Count (cell * 10 9/L) Lab 0 0 + 1.65 *3 20 [(20 ‐0)/3] ‐1.65= 6.7 Any QC such as 1:3.5s = 5.0 5.0 A Lab 5 5 + 1.65 *3 20 [(20 ‐5)/3] ‐1.65= 5 1:3s = 10.0 3.35 B Lab 0 0 + 1.65 *5 20 [(20 ‐0)/5] ‐1.65= 4 Multirule such as = 8.3 2.35 C 1:3s/2 of 3:2s/R:4s/3:1s N=3 Lab 2 2 + 1.65 *5 20 [(20 ‐2)/5] ‐1.65= 3.6 Multirule such as =10.3 1.95 D 1:3s/2 of 3:2s/R:4s/3:1s/12x N=6 Lab 5 5 + 1.65 *10 20 0 (TE> TE A) 1.65 No QC strategy is = 21.5 E available; stop reporting patients and begin corrective action immediately 26 Activity: QA Officer Needs Your Help! Purpose What will you do? You are visiting a site and will Working individually, you will: assist the QA Officer with  Review Handout 2 interpreting his statistics for  Participate in the plenary discussion the month of June 20XX and help him resolve any issues Working in groups of 4 ‐5, you will: encountered.  Investigate each problem revealed during the plenary discussion using records What will you need? supplied with Tool 2.  Complete Worksheet 2 for each problem • Worksheet 2: QC Investigation investigated (40 minutes/problem)  Worksheet 3: Case Scenario Complete Worksheet 3 for the case your • • Handout 2: Monthly Review group is assigned  Envelope containing Tool 2: Select a spokesperson • Investigation Records  Have the spokesperson present a 5 ‐minute • Calculation Device (e.g. presentation for your assigned case to the phone, calculator) class. • Pencils 27 125 minutes 9 HILS1757 Effective Date: 12/06/2016 Documentation of the Analytical System Every laboratorian who operates a method has the responsibility to record  any changes made to the system  control results obtained during operation  decisions on reporting or not reporting patient test results  corrective action taken To document effectively, we must record the information as events occur. 28 ISO 15189:2012 ‐ 4.9 Identification and control of nonconformities a) the responsibilities and authorities for handling nonconformities are designated; b) the immediate actions to be taken are defined; c) the extent of the nonconformity is determined; d) examinations are halted and reports withheld as necessary; e) the medical significance of any nonconforming examinations is considered and, where appropriate, the requesting clinician or authorized individual responsible for using the results is informed; f) the results of any nonconforming or potentially nonconforming examinations already released are recalled or appropriately identified, as necessary; g) the responsibility for authorization of the resumption of examinations is defined; h) each episode of nonconformity is documented and recorded, with these records being reviewed at regular specified intervals to detect trends and initiate corrective action. On‐going Evaluation of Your Quality Control Systems Gaussian is the Key to Quality Control Alerts us to changes in Monitors and evaluates accuracy and precision method performance Sorry Umsings Ough NS EQA TEa MN Determine if the system is stable or undergoing a Defines an acceptable change vs. change by comparing today’s unacceptable change (TE < TEA) observed value with the past history of previous results Use SEc or Sigma to benchmark methods and select appropriate control rules 30 10 HILS1757 Effective Date: 12/06/2016 ISO 15189 Standard: “The laboratory shall design internal quality control procedures that verify the attainment of the intended quality of results…” ISO 15189: 5.6.1 31 In QC Workshop Terms Standard: “The laboratory shall select appropriate control rules so that the laboratory’s internal quality alerts them before erroneous results are released.” 32 11 HILS1757 Effective Date: 12/06/2016