PEPFAR Quality Control and Method Validation Activity 14: Presentation Slides

Ensuring your method is acceptable for your patients ACTIVITY: Introduction to Method Evaluation 1 23 24 25 26 27 28 29 30 31 660.0 620.0 610.0 635.0 655.0 650.0 605.0 655.0 660.0 LS LS LS LS LS LS LS LS LS Method Evaluation Studies 23/5 24/5 25/5 26/5 27/5 28/5 29/5 30/5 31/5 MAY (curren t month) 2 710.0 +3 SD 685.0 + 2 SD C o n t 660.0 + 1 SD r o l 635.0 X V a l u 610.0 1 SD - e 585.0 2 SD - 560.0 -3 SD 23 24 25 26 27 28 29 30 31 Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 660.0 620.0 610.0 635.0 655.0 650.0 605.0 655.0 660.0 Value 610.0 615.0 635.0 660.0 590.0 635.0 680.0 655.0 660.0 610.0 625.0 LS LS LS LS LS LS LS LS LS Initials A M A M A M A M A M A M A M A M A M AM TK Method Evaluation Studies 23/5 24/5 25/5 26/5 27/5 28/5 29/5 30/5 31/5 Date 1/6 2/6 3/6 4/6 5/6 6/6 7/6 8/6 9/6 10/6 11/6 MAY (month ‐1) JUNE (current month) 3 1 HILS1762 Effective Date: 12/06/2016 Method On‐going monitoring with Evaluation IQC & EQA Studies First Determines patient Current • WHERE WE reported patient workload ARE • Assures us that WHERE WE ARE is WHERE WE WANT TO BE 4 ISO 15189:2012 ‐ 5.5.1.2 Verification of examination procedures …………. The performance claims for the examination procedure confirmed during the verification process shall be those relevant to the intended use of the examination results. 5 The Purpose of Method Evaluation Error Assessment Apply a clinical perspective -- set a target, an analytical goal, before you begin Perform experiments that gather representative data about a method's analytical performance Convert data into statistical estimates of errors Compare error estimates to specifications for medically allowable error for an objective assessment of the errors 6 2 HILS1762 Effective Date: 12/06/2016 Laboratory methods can be subject to errors in: Accuracy: results reported higher or lower than their true values Precision: results that vary significantly from the average ‐ some too high, some too low Reportable Range: inaccurate results reported above or below this range without special handling Specificity: higher or lower results due to interference by substances in patient samples Sensitivity: false negatives because methods fail to detect very low amounts of the analyte Reference Interval(s): even though results are accurate and precise, reported results will be clinically misleading if the reference interval does not match your population Brooks, Zoe C. (2013) from Awesome Numbers 2 Evaluating Error On ‐line Training Program 7 1 ‐ Method Evaluation Validation Verification • Confirmation of known • Establishment of performance parameters performance parameters by defining and published by the characterizing the magnitude of the developers of the method. analytical error present • Performed by the • Performed by the developers of the method laboratory as part of the (e.g. manufacturers) first step in QC of the • Requires extensive studies system to reveal and assess the • Requires studies to error present confirm manufacturer's claims 8 ISO 15189:2012 ‐ 5.5 Examination processes 5.5.1 Selection, verification and validation of examination procedures 5.5.1.1 General The laboratory shall select examination procedures which have been validated for their intended use. …….. The specified requirements (performance specifications) for each examination procedure shall relate to the intended use of that examination. 9 3 HILS1762 Effective Date: 12/06/2016 Non‐standard method Laboratory developed test (LDT) Standard method used outside of intended scope by manufacturer ( i.e. urine sample with serum validated method) Modified validated method VALIDATION ‐ determination of performance characteristics, once developed Experiments to be performed Linearity Precision Interference Recovery Comparison of Method Detection Limit Establishment of Reference Interval 10 Existing method with defined performance Existing method used after repair of a major component Existing method relocated to another part of the facility FDA‐approved method or instrument VERIFICATION ‐ confirmation of performance characteristics previously determined by the manufacturer during validation (much simpler and streamlined process than validation) Experiments to be performed Linearity Precision Comparison of Method Detection Limit (when applicable) Transference of Reference Interval 11 Why is verification important? Even though manufacturers test their methods extensively, there are many factors in an individual laboratory that can affect the actual precision and accuracy (WHERE WE ARE) of the method. • Different climate conditions • Different shipping and storage conditions that may affect reagents and materials • Changes in instrument or system components • Different skill level of the operators 12 4 HILS1762 Effective Date: 12/06/2016 Process for Introducing a New Method at your Site •Pre‐purchase Assessment 1 •Installation and Calibration 2 •Familiarization Period 3 •Method Validation Plan 4 •Introduction into Routine Service 5 13 · Pre-purchase Assessment Pre‐selection is often based on no more than a superficial impression of potential suitability that may have been gained from commercial advertising, published evaluations, or the previous experiences of colleagues. White, G.H. & Fraser, C.G. (1984). The evaluation kit for clinical chemistry: A practical guide for the evaluation of methods, instruments, and reagent kits. Journal of Automatic Chemistry, 6(3), 122‐141. 14 1 . Pre-purchase Assessment Darfarmanon charmetariction Performance characteristics Factors that demonstrate how well a method performs • Reportable Range • Precision • Accuracy • Interference • Detection Limit • Reference Interval Analytical method validation mainly focuses on evaluation of performance characteristics. During this step, these characteristics should be considered and expected quality specifications defined in the purchase contract. 15 5 HILS1762 Effective Date: 12/06/2016 EQA Summary Report for Chloride Acceptable Chloride N X SD Range CV% Abbott Architect c4000 118 97.5 0.9 92 ‐ 103 0.92 AWDT Clinical Chem. Systems 316 96.9 1.4 92 ‐ 102 1.44 Beckman AU Chem Syst 317 96.3 1.2 91 ‐ 102 1.25 Beckman Synchron CX 75 98.6 2.5 93 ‐ 104 2.54 Beckman UniCel DxC 600, 800 328 96.7 1.4 91 ‐ 102 1.45 Horiba Med. (ABX) Pentra 400 83 96.3 2.3 91 ‐ 102 2.39 Medica Easylyte 29 96.8 1.7 91 ‐ 102 1.76 OCD Vitros 250 ‐ 950 520 95.8 1.5 91 ‐ 101 1.57 OCD Vitros 5, 1 FS 143 96.1 1.4 91 ‐ 101 1.46 OCD Vitros 5600 130 96.1 1.4 91 ‐ 101 1.46 Poly‐Chem 15 93.5 2.2 88 ‐ 99 2.35 Roche Cobas c 501 113 93 1.4 88 ‐ 98 1.51 16 •Installation and Calibration 2 Candidate instrument selected and purchased becomes the test method in your laboratory once it arrives. • Do not unpack the items. Let the service engineer unpack and set up the method. • Service engineer will perform installation system checks and initial calibration. • Retain all installation and calibration paperwork Set up the method in the physical location identified for routine operations. 17 Familiarization Period 3 The time period when the analytical staff learns how to operate the instrument or new method. You need to know first how to operate the instrument before you can perform experiments that will determine the amount of error (SE and RE) present in the new method. At this point in the process, ensure that all required ancillary equipment ( e.g. centrifuge, pipettes) is functioning in an acceptable manner. 18 6 HILS1762 Effective Date: 12/06/2016 •Method Validation Plan 4 • Define quality requirement 1 • Select the appropriate types of experiments to reveal analytical errors 2 • Collect experimental data 3 • Use statistical tools on the data to estimate size of analytical errors 4 • Compare the observed errors with the defined allowable error 5 • Judge the acceptability of observed performance characteristics 6 • If method is acceptable, then perform reference range experiment 7 From Westgard.com Website. The Experimental Plan for Method Validation. Last accessed May 1, 2013 19 1 •Define quality requirement TE < TEA Meet or exceed manufacturer’s performance specifications and/or TE = bias + 3SD < CLIA TEA (%TE = %bias + 3CV% < CLIA %TEA) 20 •Select the appropriate types of 2 experiments to reveal analytical errors Types of analytical errors  Random Error (RE)  Systematic Error (SE)  Constant SE  Proportional SE 21 7 HILS1762 Effective Date: 12/06/2016 Method Validation/Verification, TE < TEA To reveal the Linearity Studies TE present Determines Random Error Systematic Error reportable range Precision Studies Comparison of Method Studies SD, %CV Bias, %Bias or the Difference between Methods 22 Error Assessment  Random Error (RE)  Systematic Error (SE)  Constant SE  Proportional SE 23 Random Error‐ scattering of points Constant RE around the regression line SE Constant SE  the whole line is shifted up and ALL results are increased by the same amount Test Method Result (Y)  Remains the Y‐intercept same over a range of concentrations Comparative Method Result (X)  Increases by an absolute value 24 8 HILS1762 Effective Date: 12/06/2016 Random Error‐ scattering of points around the regression line Proportional SE  Magnitude of the error increases as the test results get RE higher  Changes as Test Method Result (Y) concentration changes Slope Increases by a Proportional SE  percentage Comparative Method Result (X) 25 •Select the appropriate types of 2 experiments to reveal analytical errors Select the appropriate types of experiments for AST method evaluation  Linearity experiment– determine the working range of the new method  Precision experiment –RE  Comparative of Method experiment –SE If experiments are acceptable, then the reference interval experiment is performed 26 Example Validation Plan for Vitros 250 Chemistry Analyzer Validation Overview Precision 2. Accuracy Plan 4. Analytical measurement range (AMR) and Clinical reportable range (CRR) 3. Linearity 5. Sensitivity 6. Specificity 7. Reference Range 8. Method Approval Plan: The validation will be conducted on the Vitros 250 analyzer (serial number 25012919) for the following analytes and methods: Albumin, Alkaline Phosphatase, ALT, Amylase, AST, BUN, Calcium, Chloride, Cholesterol, CK, CO2. Creatinine, Direct Bilirubin, Glucose, HDL Cholesterol, Lactate, Lipase, Phosphorous, Potassium, Sodium, Total Bilirubin, Total Protein, Triglycerides, Uric Acid 1. Precision a. Precision is reproducibility- the agreement of the measurements of replicate runs of the same sample. It is the process of determining the range of random error. The precision is measured in terms of coefficient of variation b. Random Error will be evaluated by running between day and within day (CV) precision using normal and abnormal control samples, Between-day will be tested by running each sample once per day for 20 days or 4 samples per day for 5 days. Within day will be tested by running each sample 20 times in the replicates. one day. The mean, standard deviation (SD) and CV will be calculated of c. Acceptability criteria: The % CV for each assay is expected to be equal to or less than the manufacturer's performance specifications for precision, In the event that an assay does not perform as expected, the %CV will be compared to the allowable random error (33% of SMILE Total Allowable Error Limits for between day and 25% of SMILE Total Allowable Error Limits for 27 within day) Refer to SMILE Chemistry TE Limits table (Appendix 11 9 HILS1762 Effective Date: 12/06/2016 Documentation 1. Summarize Data Validation Summary Report ABC Lab Vitros 950 Purposes Validation Description of Equipment Process: Equipment Ps Serial Number: 995435 cess: Vatros 990 Chemistry Analyzer Location: ABC Lab, City-Township, Country Date: 18 June - 20 August 2008 FDA Approval Status: Approved Procedure: Refer to the ABC Lab V Plan for Vitros 95º Chemistry Analyzer Results: All raw data reports and statistical analysis can be found in the Vitros 950 Chemistry Validation binder. refer to tab A 2. Organize & File: • Expected Results Observed Results Plan Analyse Between Day Manufacturers Precision X3% of CLIA Nonil Control Aon Control Acceptability • Report Acceptable • Raw Data Acceptable Expected Results Olserved Results Analyse Within Run Precision 25% of CLIA ma Control Aun Control ALY CV% Acceptable 28 •Collect experimental data 3 Role Plays Group 1 – Linearity Experiment Group 2 –Precision Experiment Group 3 ‐ Comparison of Methods Experiment 4 •Use statistical tools on the data to estimate size of analytical errors •Compare the observed errors with the defined 5 allowable error •Judge the acceptability of observed 6 performance characteristics Plenary Following each role play, the class will perform Steps 4, 5, 6. 29 •If method is acceptable, then perform 7 reference interval experiment Role Play Group 4 –Reference Interval Experiment Plenary Evaluate the data and determine acceptability 30 10 HILS1762 Effective Date: 12/06/2016 Activity: Collect Experimental Data Role Play Purpose What will you do? Learn how to collect the Working in your assigned group, you will: necessary experimental data for  Design a 10‐15 minute role play to the following method validation demonstrate how to collect the data experiments: as specified in Handout 3 for your  Linearity Experiment group’s assigned experiment.  Replication Experiments  Determine the actors and their roles  Comparison of Methods Experiment  Create the needed props  Transference of Reference  Address all procedural steps outlined Intervals in the Experiment Handout regarding What will you need? sample type, preparation, and testing.  • Handout 3: AST Validation Plan Practice performing the role play at • Experiment Handout ‐ guideline least one time. specific to your assigned  Perform the role play for the class experiment when called upon by the facilitator. • Materials to create the necessary props for the role play • Workstation set‐up table 90 minutes 31 Linearity Experiment • Validates that a test continues to work properly throughout its entire reportable range • First experiment to be performed because Linearity experiment defines the usable analytical – range (reportable range) Valid data (within the reportable range) can be – collected for the other experiments to determine the amount of error present 32 Method Validation/Verification, TE < TE A To reveal the TE present Linearity Studies Random Error Systematic Error Confirms reportable range Precision Studies Comparison of Method Studies SD, %CV Bias, %Bias or the Difference between Methods 33 11 HILS1762 Effective Date: 12/06/2016 Calibrator Set Points Linearity Experiment QC Points ............ > Upper Limit of Upper Limit Reportable of Range???? Reportable Range Set‐point Actual Value Zero‐point Assigned Value 34 What is the relationship between solutions? POOL POOL POOL POOL POOL 1 2 3 4 5 LOW HIGH A series of known relationships established by dilution 0 0.5 1 1.5 2 35 Slope Ideally, the slope is equal to 1.0 Acceptable Range Guideline: 0.9‐1.1 lf the slope is outside the acceptable range, examine the results of the highest standard first. It is possible that the test is nonlinear at its highest value. University of California, Santa Cruz. (2012, May). Linearity Testing (Reportable Range) Calibration 36 Verification. Retrieved April 5, 2013, from http:shs‐manual.ucsc.edu/policy/linearity‐testing‐reportable‐ range‐calibration‐verification 12 HILS1762 Effective Date: 12/06/2016 Intercept Ideally, the Y‐intercept is equal to zero. For enzyme determinations and other assays with results in high numerical values, the Y—intercept may be much higher with no clinical significance. The Y—intercept for assays with low numerical values should be 0.0 ± 1.0. University of California, Santa Cruz. (2012, May). Linearity Testing (Reportable Range) Calibration37 Verification. Retrieved April 5, 2013, from http:shs‐manual.ucsc.edu/policy/linearity‐testing‐reportable‐ range‐calibration‐verification Raw data collected Linearity from Experiment experiment Amount Line = mX +b of SE present at this data Predicted Y point using linear Actual Value regression Assigned Value 38 Raw data collected Outliers from experiment ... Actual Value Assigned Value 39 13 HILS1762 Effective Date: 12/06/2016 Raw data collected Outliers vs Nonlinearity from experiment Best‐fit line drawn adhering to the Actual Value lower points Assigned Value 40 Analytical Measurement Range(AMR)‐ the range of analyte values that a method can directly measure on the specimen without any dilution or any other pretreatment not part of the usual assay process; also known as the reportable range Clinical Reportable Range (CRR)‐ the range of analyte values that a method can measure, allowing for specimen dilution or other pretreatment used to extend the direct AMR For AST on the Cobas c501, the dilution used on‐board is 1:10 41 Handout 4: AST Validation Summary, page 3 Allowable Linear Regression Statistics Systematic Analyte Error Linear Range Verified Evaluation Slope Intercept 50% of TEa Ex:ALT 0.970 0.282 10% 2.5-770 Linear AST 0.99 5.60 10% 6 - 704 Linear Mfg’s AMR Low Value High Analyte Reportable Verified Value Verified Range Dilutions CRR Ex:ALT 5-700 U/L 2.5 770 5-700 1:10 5-7000 AST 5 - 700 U/L 6 704 6 - 700 1:10 6 - 7000 42 14 HILS1762 Effective Date: 12/06/2016 Precision (Replication) Experiment Within ‐run Precision Between ‐run Precision • The closeness of agreement • The closeness of agreement between results of test between results of test performed on the same performed on the same method items, by the same method items in a single analyst, on the same laboratory but over an instrument, under the same extended period of time conditions in the same • Intermediate Precision, Total location and repeated over a Precision, Within short period of time ‐laboratory • Analyze a sample once each • Repeatability day for 20 days • Analyze a sample 20 times in • one analytical run Reflects the effects of long term variation due to day ‐to ‐ • Reflects best ‐case scenario for day changes in operating precision conditions 43 Precision Experiment First, always determine if the within ‐run precision is acceptable.  If the within ‐run precision (best case scenario) is not acceptable, then the random error must be reduced or the method should be abandoned.  If the within ‐run precision is acceptable, then determine the between ‐run precision The goal is to have a imprecision less than 1/3 or 1/4 the total allowable error. 44 Handout 4: AST Validation Summary, page 1 Observed Results Expected Results Between Day Analyte Manufacturer’s Normal Abn Acceptability Precision 33% of CLIA Control Control CV% CV% Ex:ALT 3.3% 6.6% 3.8% 4.3% Acceptable AST 1.3% 6.6% 3.1% 1.8% Acceptable Observed Results Expected Results Within Run Analyte Normal Abn Acceptability Manufacturer’s 25% of Control Control Precision CLIA CV% CV% Ex:ALT 2.6% 5% 2.6% 4.3% Acceptable AST 0.8% 5% 2.3% 1.3% Acceptable 45 15 HILS1762 Effective Date: 12/06/2016 Ensuring your method is acceptable for your patients ACTIVITY: Introduction to Method Evaluation (Day 2) 46 Process for Introducing a New Method at your Site •Pre‐purchase Assessment 1 •Installation and Calibration 2 •Familiarization Period 3 •Method Validation Plan 4 •Introduction into Routine Service 5 47 •Method Validation Plan 4 • Define quality requirement 1 • Select the appropriate types of experiments to reveal analytical errors 2 • Collect experimental data 3 • Use statistical tools on the data to estimate size of analytical errors 4 • Compare the observed errors with the defined allowable error 5 • Judge the acceptability of observed performance characteristics 6 • If method is acceptable, then perform reference range experiment 7 From Westgard.com Website. The Experimental Plan for Method Validation. Last accessed May 1, 2013 48 16 HILS1762 Effective Date: 12/06/2016 •Select the appropriate types of 2 experiments to reveal analytical errors Select the appropriate types of experiments for AST method verification  Linearity experiment– determine the working range of the new method  Precision experiments –RE  Comparative of Method experiment –SE If MV experiments are acceptable, then the reference interval experiment is performed 49 •Collect experimental data 3 Role Plays Group 1 – Linearity Experiment Group 2 –Precision Experiment Group 3 ‐ Comparison of Methods Experiment 4 •Use statistical tools on the data to estimate size of analytical errors •Compare the observed errors with the defined 5 allowable error •Judge the acceptability of observed 6 performance characteristics Plenary Following each role play, the class will perform Steps 4, 5, 6. 50 •If method is acceptable, then perform 7 reference interval experiment Role Play Group 4 –Reference Interval Experiment Plenary Evaluate the data and determine acceptability 51 17 HILS1762 Effective Date: 12/06/2016 Accuracy Determined by one of the following: • Assaying materials with assigned or known values • Comparing patient specimen results with an accurate method in long ‐ standing use • Comparing results across laboratories by splitting specimens with another laboratory 52 Comparison of Methods Experiment • The experiment is performed to estimate the amount of systematic error • The laboratory must demonstrate that a new method performs comparably to or better than the old method of testing. 53 AST Performed on the XYZ Analyzer • Used for the current patient testing • Operating within pre‐defined TEA • Passed the previous three proficiency tests • Compares well with the peer group of the inter‐ laboratory report on a monthly basis • SEc has remained relatively stable for the last 6 months • Calibration verification was acceptable will be used as the comparative method (X) 54 18 HILS1762 Effective Date: 12/06/2016 Comparison of Methods Experiment Y Axis – New Method (i.e. Cobas c501 Analyzer) Ideally, X=Y where - - - -------- the slope is 1.00 and Line of Identity the Y ‐intercept is 0.0 Y X Axis –Reference or Old = mX + b Y = 1.00 X +0.0 Method (i.e. XYZ Analyzer) Y=X 55 Random Error ‐ scattering of points Constant SE around the regression line Constant SE  ALL test method results change by the same amount  Remains the Test Method Result same over a range of Y ‐intercept concentrations  Increases by an absolute value Comparative Method Result (X) 56 Random Error ‐ scattering of points around the regression line Proportional SE  Magnitude of the error increases as the test results get higher  Changes as Test Method Result concentration changes Slope Increases by Proportional SE  a percentage Comparative Method Result (X) 57 19 HILS1762 Effective Date: 12/06/2016 Linear Regression Y = mX +b Y= 0.991X +1.22U/L Slope = 0.991 Cobas c501 (Y) Y‐intercept = 1.22 Hitachi 917(X) 58 What is Linear Regression? • Linear regression is a statistical procedure designed to show a linear relationship between two variables (i.e. X and Y) • Based on the equation for a straight line Y = mX +b • Regression is used to show a relationship between the X and Y values, as well as to predict the Y‐values (Y’) that correspond to critical X‐values (Xc) 59 Predict the Y-values (Y’) that correspond to critical X-values (Xc) The SE at a given medical decision concentration (Xc) can be determined as follows: Y’ = mXc + b SE = Y’ ‐ Xc 60 20 HILS1762 Effective Date: 12/06/2016 Y mX +b = Y= 0.991X +1.22U/L At a medical concentration of 30 U/L (X ) c What is the Yc’?? Slope = 0.991 Cobas c501 (Y) Y ‐intercept = 1.22 Hitachi 917(X) 61 To Use Linear Regression to Estimate SE • The relationship between X and Y must be linear • Visually check your comparison plot • Outliers, especially at the ends, can significantly affect the calculations • Visually inspect your data using the comparison or difference plots • Identify and remove outlier points • A sufficient range of data must be collected to clearly define the regression line • 50% of your samples selected for the experiment should be outside the reference interval 62 Correlation Coefficient (r) • Defines the strength of a relationship between two variables • Set ‐up on an r ‐scale that ranges from +1.00 to ‐1.00 • A +1.00 means a perfect positive relationship • X increases, Y always increases • A 0.00 means there is NO relationship at all between X and Y; • all the points will scatter randomly • Y may increase, decrease, or remain the same as X • Useful for assessing whether the range of data is wide enough to provide good estimates of the slope and Y ‐intercept 63 21 HILS1762 Effective Date: 12/06/2016 Which Regression Line is the Correct One? Unable to clearly define the regression line because the range of collected data is too narrow. r will be low Test Method Result Comparative Method Result (X) 64 Regression line can clearly be defined when the range of collected data encompasses as much of the AMR as possible. r will be high Test Method Result Comparative Method Result (X) 65 Random Error scattering of ‐ points around the regression line Sy/x • The regression statistic that describes the standard deviation about the regression line. • It quantifies the scatter of points Test Method Result (Y) about the regression Comparative Method Result (X) line. • Reflects the RE from both methods 66 22 HILS1762 Effective Date: 12/06/2016 Handout 4: AST Validation Summary, page 2 Correlation Total Coefficient Linear Regression Acceptability Analyte Allowable (r) Statistics For Linear Error Expected Regression >0.975 Slope Intercept Ex.ALT 0.5U/L or 0.997 1.002 2.19 Acceptable 20% AST 20% 0.9998 0.9818 2.8531 Acceptable Analyte Xc Linear Regression Bias at Xc %Bias at Equation Xc Ex.ALT 40 U/L Y= 1.002 X + 2.19 2.3 U/L 5.7% AST 35 U/L Y = 0.9818 X + 2.8531 2.2 U/L 6.3% AST 120 U/L Y = 0.9818 X + 2.8531 0.7 U/L 0.6% 67 •Collect experimental data 3 •Use statistical tools on the data to estimate size 4 of analytical errors •Compare the observed errors with the defined 5 allowable error •Judge the acceptability of observed 6 performance characteristics 68 Handout 4: AST Validation Summary, page 5 Concentration AST %TEA at Long-term %Bias or Total Sigma Acceptability of Xc concentration Precision %difference Error Metric of Xc with comparative method 35 U/L 20% 3.1% 6.3% 15.6% 4.4 acceptable 120 U/L 20% 1.8% 0.6% 6% 10.8 acceptable 69 23 HILS1762 Effective Date: 12/06/2016 •Judge the acceptability of observed 6 performance characteristics •If method is acceptable, then 7 perform reference interval experiment 70 Reference Interval • Describes the variations of a measurement or value in healthy individuals. • Serves as a basis for a provider to interpret a set of results for a particular patient. • Sometimes called a “reference range” or “normal range” (old term) 71 Why Perform Reference Interval Evaluation? Even though results are accurate and precise, reported results will be clinically misleading if the reference interval does not match your population When to Perform Reference Interval Evaluation • New analyte • Change instruments or methods • Change sample collection systems • Reformulation of reagent 72 24 HILS1762 Effective Date: 12/06/2016 Reference Interval Experiment • Establish a reference interval for a measurement procedure and reference population. • 120 samples for each partition • Verify a reference interval for a measurement procedure and reference population. • 20 samples for each partition • Transfer a reference interval from a comparable measurement procedure. • Use linear regression equation • Limited to one time transference 73 Creating the Questionnaire Exclusion Criteria Partition Criteria • Details about the candidate • Characteristics of the reference individual that, if selected reference present, serve to keep that individual that divide the individual from being reference sample into included in the reference significant subclasses data collection. • Examples Gender • Examples – Alcohol consumption Age – – Recent illness Race – – Pregnancy Stage of menstrual cycle – – Recent Surgery Stage of pregnancy – – Recent blood transfusion Blood group – – Abnormal blood pressure Ethnic background – – 74 Questionnaire • Should include contact information in case the experiment reveals some potential abnormalities • The laboratory should have a mechanism in place for medical review and confidential notification • May be accompanied by a consent form • Example: “Cape Clinic Laboratory personnel are allowed to obtain specimens and use the laboratory values and questionnaire information for the determination of reference intervals. All information will be held in a confidential manner. Notification of the result will only occur if the laboratory director deems the result requires further follow ‐up.” 75 25 HILS1762 Effective Date: 12/06/2016 Partition 1; Male Partition 2; Female Run # Date Collect Analysis Initials Result Date Collect Analysis Initials Result Time Time Time Time 1 27-Nov 13 0800 0900 AM 27-Nov 13 0930 1000 AM 12 2 27-Nov 13 0910 1000 AM 27-Nov 13 1030 1100 AM 22 27-Nov 13 0930 1100 AM 28-Nov 13 0900 1000 LS 6 E 27-Nov 13 1000 1100 AM 28-Nov 13 0930 1000 LS 18 27-Nov 13 1015 1100 AM 28-Nov 13 1100 1200 LS 25 6 27-Nov 13 1030 1100 AM 28-Nov 13 1115 1200 LS 15 7 27-Nov 13 1045 1100 AM 29-Nov 13 0930 1000 TZ 21 28.Nov 0900 0900 1000 LS 29-Nov 13 1000 1100 TZ 19 28-Nov 13 0915 1000 LS 29-Nov 13 9-Nov 13 1015 1100 1100 TZ 28 11 28-Nov 13 0930 1000 LS 1030 1100 TZ 12 28-Nov 13 1015 1200 LS 9-Nov 13 1045 1100 TZ 22 28-Nov 13 1030 1200 LS 9-Nov 13 1200 1400 TZ 20 28-Nov 13 1100 1200 LS 12 9-Nov 13 1230 1400 TZ 25 29-Nov 13 1300 1400 TZ O-Nov 13 1300 1500 LS 16 29-Nov 13 1315 1400 TZ 30-Nov 13 O-Nov 13 1315 1500 LS 21 29-Nov 13 1330 1400 TZ 30-Nov 13 1330 1500 LS 23 30-Nov 13 1400 1500 LS 28 01-Dec 13 0800 0900 AM 23 Nov 13 1045 1200 1000 AM 7 1100 AM 23 Reference Interval is verifica, needs to be re-evaluated. Reference Interval is verified>> needs to be re-evaluated. Analysis of transference performed by Date: Analyte Adult Reference Ranges Reference Range Cited % Verified Males = 38 U/L XYZ Manufacturer Range (Expected 290%) 95% AST Females $32 U/L 100% 76 Handout 4: AST Validation Summary, page 5 Adult Analyte Reference Reference % Verified Ranges Range Cited (Expected ≥90%) Males ≤ 38 XYZ 95% U/L Manufacturer Females ≤32 Range 100% AST U/L 77 Documentation of Laboratory Director’s Approval of the New Method Method Approval Approved / Not Approved If not approved, provide recommendations/corrective actions below. Laboratory Director: _______________________________ Date: _________________ Insert Lab director name here Prepared by: __________________________________ Date: ___________________ Insert name and title here 78 26 HILS1762 Effective Date: 12/06/2016 •Method Validation Plan 4 • Define quality requirement 1 • Select the appropriate types of experiments to reveal analytical errors 2 • Collect experimental data 3 • Use statistical tools on the data to estimate size of analytical errors 4 • Compare the observed errors with the defined allowable error 5 • Judge the acceptability of observed performance characteristics 6 • If method is acceptable, then perform reference range experiment 7 From Westgard.com Website. The Experimental Plan for Method Validation. Last accessed May 1, 2013 79 Process for Introducing a New Method at your Site • Pre‐purchase Assessment 1 •Installation and Calibration 2 •Familiarization Period 3 •Method Validation Plan 4 •Introduction into 5 Routine Service 80 •Introduction into Routine Service 5  Select QC procedures  Write operating procedures  Train analytical staff  Notify your customers if sample collection requirements or reference interval have changed  Introduce method into service (“Go‐live date”)  Monitor routine performance 81 27 HILS1762 Effective Date: 12/06/2016 ISO 15189:2012 ‐ 5.5 Examination processes 5.5.1 Selection, verification and validation of examination procedures 5.5.1.1 General The laboratory shall select examination procedures which have been validated for their intended use. …….. The specified requirements (performance specifications) for each examination procedure shall relate to the intended use of that examination. 82 Gaussian Distribution is the Key to Statistical Quality Control 83 28 HILS1762 Effective Date: 12/06/2016