PEPFAR Quality Control and Method Validation Participants Manual

On the Road to Laboratory Improvement QC WORKSHOP Taking a turn towards QC QC Workshop Participant’s Manual HILS1746 Effective Date: 12/06/2016 QC WORKSHOP Purpose The purpose of the QC workshop is to provide participants with the needed training and tools to evaluate and monitor their quantitative tests. Through the use of interactive exercises and an assigned improvement project, participants will learn how to translate their new-found knowledge into practical applications for their site. In this workshop, participants will focus on:  Understanding principles of Statistical QC.  Understanding the information conveyed by statistical data.  Applying QC foundational principles at the site to make appropriate decisions and take appropriate actions.  Establishing patient-focused strategies using pre-defined quality performance for the test and method.  Using on-going QC monitoring of internal and external QC information to evaluate method’s current performance.  Utilizing data obtained from initial method evaluation experimental techniques to judge the acceptability of a quantitative test. Target Audience Participants should be responsible for establishing and implementing QC procedures at a site. This may include, but not limited to:  Laboratory managers, QA officers, and section supervisors managing IQC and EQC systems in their laboratory.  Technical staff who are assigned to manage specific quantitative instruments.  Mentors responsible for assisting sites with QC implementation of current methods.  SLMTA coordinators providing adjunct programs to assist laboratories during their post SLMTA phase.  Instructors responsible for teaching basic QC. Performance Outcome With satisfactory participation in the training and successful implementation of laboratory improvement project(s), a participant’s laboratory should be able to establish, implement and sustain a QC program. Using quality indicators specific to IQC and EQC such as the number of quantitative tests performed with a defined QC strategy, sponsors of this workshop will be able to measure improvement at the site. Checklist Items Supported by this Workshop This workshop supports the requirements for the following items from the WHO-AFRO Laboratory Accreditation Preparedness Checklist: 1.3, 1.5, 1.6, 1.7, 1.10, 2.1, 2.2, 2.3, 3.3, 3.4, 3.8, 3.9, 4.1, 4.4, 5.1, 5.2, 5.3, 5.6, 5.9, 5.10, 5.11, 5.12, 5.13, 5.15, 7.1, 7.5, 7.7, 7.8, 7.9, 7.10, 8.7, 8.8, 8.9, 8.10, 8.12, 8.13, 8.14, 9.3, 10.1, 10.2, 10.3, 10.4, 10.5, 11.2, 11.3, 11.4, 11.5, 12.4 HILS1746 Effective Date: 12/06/2016 What’s in this Workshop? ACTIVITY PURPOSE DURATION TITLE Participants are provided with an overview of how this Workshop workshop integrates quality control (QC) into their laboratory to Introduction support their accreditation efforts. Additionally, workshop 30 minutes expectations are presented to participants. In this activity, the pre-requisite reading’s Final Examination on basic QC and Measures of Central Tendency worksheet will be Let’s Examine reviewed with participants. Tools to calculate mean, standard 1 hour the Basics deviation (SD), and coefficient of variation (CV) will be 40 minutes introduced and used throughout the workshop. Stable analytical systems will produce the same Gaussian distribution of data. When a system undergoes a change, an Gaussian is the unexpected data point will be produced. In this activity, 2 hour Key participants will demonstrate the foundational principles of 10 minutes statistical QC (SQC). Match It Up: A QC rules alert laboratorians to a change in the analytical Rule Violation system (measurement procedure). In this activity, participants 2 hours Game match charts to the correct QC rule violation and identify the 30 minutes type of error. Front-line workers assess whether or not an analytical run is acceptable. QC protocols developed by laboratory The Front-line management must standardize the decision criteria used by the Worker workers. In this activity, participants are provided with QC 1 hour charts and a Westgard Multirule algorithm and must apply the 30 minutes decision-making process to determine acceptability of the analytical run. Using the QC tool, L-J Chart, one can visually inspect a quantitative method’s current accuracy and precision to quickly It Begins with assess if a change is occurring within the system. In this 5 hours the Right Chart activity, participants learn how to correctly create this tool, how 45 minutes to visually assess it, and how to avoid pitfalls commonly encountered with its set-up. HILS1746 Effective Date: 12/06/2016 ACTIVITY PURPOSE DURATION TITLE Only when the system is known to be stable can data be Parallel collected to calculate the observed mean and SD of a new lot Testing: number of QC material. In this activity, participants create a 3 hours Making History parallel testing SOP document based on their observation of a role-played laboratory scenario. % CV is introduced and used in the SD computation of short shelf-life control materials. Total error (TE) combines bias and imprecision to quantify the largest variation from the true (target) value. In this activity, participants are introduced to bias and true value. Using the Total Error (TE) information supplied by 3 key numbers, participants calculate 2 hours the TE in units and percent. Additionally, participants explore 50 minutes the impact that bias and imprecision can have on TE and relate site-specific processes that affect TE to checklist items. To control quality, one must first determine what quality is needed for a test. In this activity, participants will explore what Putting the TEA information the fourth key number must provide to define the quality required. Additionally, participants will use a laboratory 3 hours into Quality scenario to assess if the method remains acceptable for the 25 minutes intended clinical use of the test result. Benchmarking measurement procedures using TEA and the current method’s performance allows a laboratory to identify which methods are meeting quality specifications and which How Far Can are not. Your Mean In this activity, Sigma-metric and Critical Systematic Error 4 hours Shift? (ΔSEc) are introduced. Participants will review a site’s monthly 25 minutes summary report and perform an investigational analysis using instrument records and QC charts. Laboratories must select appropriate QC rules based on the quality required for a test and the method’s observed accuracy 3 hours How to Select and precision. In this activity, participants are introduced to the Control Rules Sigma-metrics QC Selection Tool, available at Westgard.com. 35 minutes Using a case scenario, participants will select appropriate QC rules and design a QC strategy for a workstation. External Quality Assessment Schemes (EQAS) enable the How Proficient laboratory to assess on-going accuracy by comparing its Are We? method’s performance to external sources. In this activity, 2 hours participants are introduced to standard deviation index (SDI) and z-score and will review a proficiency test (PT) report. HILS1746 Effective Date: 12/06/2016 ACTIVITY PURPOSE DURATION TITLE Laboratories using the same measurement procedure and analyzing the same control material will generate similar means and SDs. This information is a valuable source to determine Understanding the target value of the QC material for a laboratory and Inter-laboratory compare their method’s performance to their peers. 4 hours Comparison In this activity, participants are introduced to coefficient of 10 minutes Reports variation index (CVI). Based on the information supplied from the inter-laboratory comparison report, participants will investigate a QC problem. The knowledge gained from a workshop becomes effective Improvement when the improvement is applied at the site for better patient 2 hour Project care. In this activity, participants will map the process needed 45 minutes Assignment to perform an improvement project (IP) assignment. In this small-group learning activity, each site receives one-on- IP Master one coaching in turn to develop an individualized 1 hour Class implementable plan for the assigned improvement project. HILS1746 Effective Date: 12/06/2016 ACTIVITY PURPOSE DURATION TITLE Before a method can be placed into routine service, it must be evaluated to ensure that the measurement procedure meets defined criteria, such as sensitivity, specificity, precision, accuracy, and linearity. In this activity, participants will perform method verification experiments and assess data to determine if the method is acceptable or not. Introduction activity includes the following experiments: Linearity –- The linearity experiment determines the reportable range of the method. Linearity is important because it validates that a test continues to work properly throughout the entire reportable range. Test specimens with analyte concentrations spanning the entire manufacturer’s reportable range are used to verify the manufacturer’s linearity claims. In this activity, participants will perform and analyze a linearity experiment. Precision –- The precision experiment, also known as a replication experiment, is performed to estimate the Introduction to imprecision or random error of a measurement procedure. Method 10 hours Evaluation Data collected from the within-run and between-run precision experiments is used to quantify the amount of random error present in the method. In this activity, participants will perform and analyze the within-run and between-run precision experiments. Comparison of Methods -- The comparison of methods experiment is performed to estimate the systematic error present in a new method. In this experiment, verification of accuracy can be accomplished by comparing split- sample results obtained from a method with clinically valid results. In this activity, participants will determine the systematic error present and calculate the total error for a test method. Reference Interval -- Providers can reach clinically misleading interpretations if the reference interval is inappropriate for the test. In this experiment, participants learn how to transfer a reference interval to the new method. TOTAL WORKSHOP TIME: 51 hrs 10 min HILS1746 Effective Date: 12/06/2016 Resources Advance for Administrators of the Laboratory. www.advanceweb.com/labmanager [Website]. Brooks, Zoe C. (2001). Performance-driven quality control. Washington DC: AACC Press. Brooks, Zoe. (2003). Quality control – from data to decisions. Basic concepts. [Course work] Worthington, Ontario, Canada: Zoe Brooks Quality Consulting. Brooks, Zoe. www.awesome-numbers.org [Website]. CLSC 5213 Clinical Laboratory Data Analysis. [Course work]. University of Medicine & Dentistry of New Jersey: School of Health Related Professions. CLSI. (2011). Laboratory quality control based on risk management; approved guideline (Vol. 31, No. 18). CLSI document EP23-A. Wayne: Clinical and Laboratory Standards Institute. CLSI. (2006). Statistical quality control for quantitative measurement procedures: Principles and Definitions; approved guideline – third edition. (Vol. 26, No. 25). CLSI document C24--A3. Wayne: Clinical and Laboratory Standards Institute. Cooper, Greg. (2008). Basic lessons in laboratory quality control. Bio-Rad Laboratories, Inc. Current Laboratory Practice Series (2009) Laboratory Quality Management System Training Toolkit [CD-ROM]. World Health Organization Fraser, Callum G. (2001). Biological variation: from principles to practice. Washington DC: AACC Press. pSMILE.org [Website] Westgard, James O. (2007). Assuring the right quality right. Madison: Westgard QC, Inc. Westgard, James O. (2000). Basic planning for quality. Madison: Westgard QC, Inc. Westgard, James O. (2003). Basic method validation (2nd ed.). Madison: Westgard QC, Inc. Westgard, James O. (2008). Basic method validation (3rd ed.). Madison: Westgard QC, Inc. Westgard, James O. (2007). Basic QC practices (2nd ed.). Madison: Westgard QC, Inc. Westgard, James O. (2010). Basic QC practices (3nd ed.). Madison: Westgard QC, Inc. Westgard QC. www.westgard.com [Website] HILS1746 Effective Date: 12/06/2016 TABLE OF CONTENTS Activity 1A: Workshop Introduction Page 1 Activity 1B: Let’s Exam the Basics Page 8 Activity 2: Gaussian is the Key Page 11 Activity 3: Match It Up: A Rule Violation Game Page 15 Activity 4: The Front-line Worker Page 22 Activity 5: It Begins with the Right Chart Page 24 Activity 6: Parallel Testing: Making History Page 40 Activity 7 Total Error (TE) Page 45 Activity 8: Putting the TEA into Quality Page 52 Activity 9: How Far Can Your Mean Shift? Page 61 Activity 10: How to Select Control Rules Page 67 Activity 11: How Proficient Are We? Page 75 Activity 12: Understanding Inter-laboratory Comparison Programs Page 83 Activity 13: Improvement Project Assignment Page 92 Activity 14: Introduction to Method Validation Page 107 14A: Linearity Experiment Page 113 14B: Precision Experiment Page 118 14C: Comparison of Methods Experiment Page 123 14D: Reference Interval Experiment Page 131 HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 1A Workshop Introduction PURPOSE: Participants are provided with an overview of how this workshop integrates quality control (QC) into their laboratory to support their accreditation efforts. Additionally, workshop expectations are presented to participants. This workshop supports the following laboratory management tasks and SLIPTA checklist items Management Tasks 1.4 Assess personnel competency against standards and determine corrective action and training needs 1.5 Conduct weekly staff meetings to coordinate activities, review lab operations, reward success, celebrate accomplishments, and resolve issues 1.6 Meet with staff individually to communicate expectations, provide feedback, coaching, or on-the-job training to ensure competency and productivity 1.7 Provide/coordinate new-hire orientation and training to staff 1.8 Maintain and update personnel records (training, certification, competency assessment) 1.9 Create a work plan and budget based on personnel, test, facility, and equipment needs 1.10 Create/review/forward reports on lab operations to upper management 1.12 Develop and implement lab improvement plans based on best practices and feedback from staff, patients, customers, quality indicators, and external assessment 1.13 Communicate to upper management regarding personnel, facility, and operational needs 2.4 Ensure appropriate physical work environment for testing 2.7 Ensure reagents and chemicals are stored properly 3.4 Enforce good stock management practices (proper storage, stock cycling, inspection of incoming orders, etc.) 3.5 Inspect quality of existing inventory and dispose of expired test kits, reagents, supplies and equipment according to policy 4.1 Accurately evaluate needs for equipment, supplies and reagents taking into consideration past patterns, present trends, and future plans 5.2 Ensure proper preventive maintenance (i.e., cleaning, proper shutdown) on instruments when used 5.3 Perform and record troubleshooting on malfunctioning equipment 5.4 Review and sign maintenance logs to ensure regular preventive maintenance and timely repairs 5.5 Take corrective actions or issue repair orders and record all issues 5.6 Follow up on all corrective action – see if equipment is properly functioning, observe for trends or determine training needs 5.7 Communicate to upper management equipment specifications and maintenance needs 6.1 Ensure that the Quality Manual with quality assurance policies and procedures is accessible to and reviewed by all staff 6.2 Ensure that QC material is tested according to SOP 6.3 Establish acceptable ranges for control material 6.4 Validate new equipment, reagents, and supplies 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.7 Review records of environmental checks & QC trends to assess impact on testing and take corrective action 6.8 Review occurrence log for patterns/trends and take corrective action HILS1746 Effective Date: 12/06/2016 6.9 Monitor reagent performance 6.10 Customize site-specific SOPs as needed 6.11 Ensure that SOP are read and understood by staff 6.12 Enroll in EQA program, monitor results, and take corrective actions 6.13 Periodically observe/assess accuracy of staff performance and take corrective action 7.3 Enforce good specimen handling and processing practices 8.1 Monitor testing to ensure SOPs are followed and tests are performed and reported properly and promptly 8.3 Review test records and findings promptly to ensure accuracy and timely release of test results 8.4 Validate assigned tests and specific abnormal results 9.3 Consult with clients regarding specimen quality, test results and findings in a professional manner and ensure each issue is resolved promptly and documented appropriately 10.1 Maintain a library of documents (policies, guidelines, SOPs, references, etc.); review and update annually 10.2 Maintain integrity, organization, and confidentiality of records (client test results, specimen transfer logs, maintenance logs, inventory logs, etc.) 10.3 Assure proper record retention, rotation to storage, and disposal according to protocol Checklist Items 1.3 Document and Information Control System Does the laboratory have a system in place to control all documents and information from internal and Laboratory Strengthening Checklist external sources? 1.5 Laboratory Policies and Standard Operating Procedures Are policies and/or standard operating procedures (SOPs) for laboratory functions, technical and managerial procedures current, available and approved by authorized personnel? 1.6 Policy and SOPs Accessibility Are policies and SOPs easily accessible/available to all staff and written in a language commonly understood by respective staff? 1.7 Policies and SOPs Communication Is there documented evidence that all relevant policies and SOPs have been communicated to and are understood and implemented by all staff as related to their responsibilities? 1.10 Data Files Are test results, technical and quality records, invalid or discontinued policies and procedures archived for a specified time period in accordance with national/international guidelines? 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? 2.2 Management Review Does the laboratory management perform a review of the quality system at a management review meeting at least annually? 2.3 Are findings and actions from MR communicated to the relevant staff? 3.3 Laboratory Director Is the laboratory directed by a person(s) with the competency, delegated responsibility to perform? 3.4 Quality Management System Oversight Is there a quality officer/manager with delegated responsibility to oversee compliance with the quality management system? 3.7 Laboratory Staff Training Is there a system for training? 3.8 Staff Competency Assessment and Retraining Is there a system for competency assessment? 3.9 Staff meetings Are staff meetings held regularly? 4.1 Advice and Training by Qualified Staff Do staff members with appropriate professional qualifications provide clients with advice and/or training regarding required types of samples, choice of examinations, repeat frequency, and interpretation of results? HILS1746 Effective Date: 12/06/2016 4.4 Communication Policy on Delays in Service Is timely, documented notification provided to customers when the laboratory experiences delays or interruptions in testing (due to equipment failure, stock outs, staff levels, etc.) or finds it necessary to change examination procedures and when testing resumes? 5.1 Adherence to Proper Equipment Protocol Is equipment installed and placed as specified in the operator’s manuals and uniquely labelled or marked? 5.2 Are equipment operated by trained, competent and authorized personnel? 5.3 Equipment and Method Validation/Verification and Documentation Are all equipment and methods validated/verified on-site upon installation and before use and is documented evidence available? 5.6 Equipment Maintenance Records Is relevant equipment service information readily available in the laboratory? 5.9 Equipment Calibration and Metrological Traceability Protocol 5.10 Equipment Preventive Maintenance Is routine user preventive maintenance performed on all equipment and recorded according to manufacturer’s minimum requirements? 5.11 Equipment Service Maintenance Is equipment routinely serviced according to schedule as per the minimum manufacturer recommendations by qualified and competent personnel and is this information documented in appropriate logs? 5.12 Equipment Malfunction - Response and Documentation Is equipment malfunction resolved by the effectiveness of the corrective action program and the associated root cause analysis? 5.13 Equipment Repair Monitoring and Documentation 5.15 Manufacturer’s Operator Manual Are the manufacturer’s operator manuals readily available to testing staff and, available in the language understood by staff? 7.1 Inventory and Budgeting System Is there a system for accurately forecasting needs for supplies and reagents? 7.5 Budgetary Projections Are budgetary projections based on personnel, test, facility and equipment needs, and quality assurance procedures and materials? 7.7 Laboratory Inventory System 7.8 Storage Area Are storage areas set up and monitored appropriately? 7.9 Inventory Organization and Wastage Minimization Is First-Expiration-First- Out (FEFO) practiced? 7.10 Product Expiration Are all reagents/test kits in use (and in stock) currently within the manufacturer-assigned expiration or within stability? 8.7 Documentation of Examination Procedures Are examination procedures documented in a language commonly understood by all staff and available in appropriate locations? 8.8 Reagents Acceptance Testing Is each new reagent preparation, new lot number, new shipment of reagents or consumables verified before use and documented? 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)? 8.12 Are environmental conditions checked and reviewed accurately? Are the following environmental conditions checked and recorded daily? 8.13. Have acceptable ranges been defined for all temperature- dependent equipment with procedures and documentation of action taken in response to out of range temperatures? 8.14. Does the laboratory participate in inter-laboratory comparison program or alternative assessment systems for all tests? 9.3 Analytic System/Method Tracing When more than one instrument is in use for the same test, are test results traceable to the equipment used for testing? HILS1746 Effective Date: 12/06/2016 10.1 Are all identified nonconforming activities/ work identified and documented adequately? 10.2 Root Cause Analysis Is documented root cause analysis performed for non-conforming work before corrective actions are implemented? 10.3 Is corrective action performed and documented for non-conforming work? 10.4 Are implemented corrective actions monitored and reviewed for their effectiveness before closure/clearance? 10.5 Preventive Actions Are documented preventive actions implemented and monitored for their effectiveness? 11.2 Quality Management System Improvement Measures Does the laboratory identify and undertake continual quality improvement projects? 11.3 Communication System on Laboratory Operations Does the laboratory communicate with upper management regularly regarding needs for continual improvement? 11.4 Are quality indicators (TAT, rejected specimens, stock-outs, etc.) selected and tracked? 11.5 Is the outcome of the review of quality indicators used to improve lab performance? 12.4 Is the physical work environment appropriate for testing?  KEY MESSAGES Can they:  The workshop targets primarily quantitative QC and  Understand how the QC method validation. workshop supports the SLMTA  Completion of nightly homework serves as the VISA for process? entering the classroom on the following day.  Understand the workshop’s Linkages between QC workshop training and the scope targeting quantitative  checklist items will be made throughout the workshop. QC?  Understand expectations during  Where are we, where do we expect to be, and where do we want to be are three questions that will be used and after the workshop? throughout the workshop to assist with data analysis  ACTIVITY OBJECTIVES and evaluation. MET? HILS1746 Effective Date: 12/06/2016 WHERE WE ARE Accuracy Precision WHERE WE EXPECT TO BE HILS1746 Effective Date: 12/06/2016 WHERE WE WANT TO BE Criteria Needed for Effective SQC HILS1746 Effective Date: 12/06/2016 Σxi ¯ x = n SD = √ Σ(xi - x ) 2 ¯ n-1 CV% = (SD / x ) * 100% ¯ Mean – Target z-score SD ¯ Bias = x -True Value ¯ Absolute Bias =| x -True Value| % bias = ( bias/target value) * 100% ¯ TE = | x -True Value| + (z factor * SD) = | bias| + (z factor*SD) ¯ | x -True Value| + 2 SD ease for computation ¯ | x -True Value| + 1.96 SD 97.5% of the population of data points included in the estimation of total error ¯ | x -True Value| + 1.65 SD 95% of the population of data points included in the estimation of total error % TE = % bias +( z factor * CV%) ≅ (TE in units / Target Value in units) * 100% % bias + 2CV% ease for computation % bias + 1.96 CV% 97.5% of the population of data points included in the estimation of total error % bias + 1.65 CV% 95% of the population of data points included in the estimation of total error TE < TEA Sigma = [(TEa - |biasobs|)/SDobs] ΔSEc = [(TEa - |biasobs|)/SDobs] - z factor = Sigma – z factor Sigma – 1.65 = ΔSEc value used by Dr. Westgard where 5% of the population of data points exceed TEA limits Sigma= ΔSEc + 1.65 SDI= (x¯ lab – x ¯ group)/ SDgroup CVI (CVR) = within lab CV/peer group CV HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 1B Let’s Examine the Basics PURPOSE: In this activity, the pre-requisite reading’s Final Examination on basic QC and Measures of Central Tendency worksheet will be reviewed with participants. Tools to calculate mean, standard deviation (SD), and coefficient of variation (CV) will be introduced and used throughout the workshop. This activity supports the following laboratory management tasks and SLIPTA checklist items Management Tasks 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.7 Review records of environmental checks & QC trends to assess impact on testing and take corrective action 6.12 Enroll in EQA program, monitor results, and take corrective actions 7.3 Enforce good specimen handling and processing practices Checklist Items 7.5 Budgetary Projections Are budgetary projections based on personnel, test, facility and equipment needs, and quality assurance procedures and Laboratory Strengthening Checklist materials? 8.9 Quality Control Is internal quality control performed, documented, and ================= verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)? 8.14Does the laboratory participate in inter-laboratory comparison program or alternative assessment systems for all tests?  KEY MESSAGES Can they: The mean, SD, and %CV are foundational statistics  Calculate the mean, SD and  used with QC %CV from a given data set? Given a data set, the mean, SD, and %CV can be  Utilize the tools introduced to  calculated. obtain a mean, SD, and %CV  Calculators, spreadsheets, and online resources are from their site’s data set? available tools sites can use with their QC data.  Calculate the measures of central tendency?  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 A B 1 4.23 A formula always starts with an equal 2 4.23 sign (=). 3 4.23 For mean, type in AVERAGE( first cell 4 4.23 number :last cell number ), then press 5 4.27 enter 6 4.31 For SD, type in STDEV.S( first cell 7 4.36 4.36 number :last cell number ), then press 8 enter. In earlier version of Excel, the 9 4.36 4.40 function may be listed as STDEV. 10 11 4.44 For %CV, type in (cell number 12 4.48 containing the SD /cell number 13 4.48 containing the mean)*100, then 14 4.53 15 4.57 press enter 16 4.57 4.61 The Excel displays the formula’s 17 4.61 result in the cell 18 19 4.66 20 4.70 =AVERAGE(A1:A21) 21 4.83 22 23 4.45 0.1749 =STDEV.S(A1:A21) 24 25 3.93 =(A24/A23)*100 HILS1746 Effective Date: 12/06/2016 For the following data sets, calculate the mean (X¯ ), standard deviation (SD), and percent coefficient of variation (%CV): 1) {5.1, 5.3, 4.9, 5.1, 5.4} (X¯ ) = ___________________ SD = ___________________ %CV = ___________________ 2) {2.13, 2.09, 2.10, 2.11, 2.15} (X¯ ) = ___________________ SD = ___________________ %CV = ____________________ 3) {36.83, 35.79, 37.01, 35.72, 36.29, 36.33, 36.54, 36.48, 36.91, 35.87} (X¯ ) = ____________________ SD = ____________________ %CV = ____________________ HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 2 Gaussian is the Key PURPOSE: Stable analytical systems will produce the same Gaussian distribution of data. When a system undergoes a change, an unexpected data point will be produced. In this activity, participants will demonstrate the foundational principles of statistical QC (SQC). This activity supports the following laboratory management tasks and SLIPTA checklist items Management Tasks 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.7 Review records of environmental checks & QC trends to assess impact on testing and take corrective action Checklist Items 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? Laboratory Strengthening Checklist ----------- 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)?  KEY MESSAGES Can they: The accuracy and precision of an analytical system  List the performance  describes WHERE WE ARE. specifications that describe  WHERE WE ARE and WHERE WE EXPECT TO BE is WHERE WE ARE with an the basis of SQC. analytical system?  If WHERE WE EXPECT TO BE is not WHERE WE Define SQC and how it applies  ARE, then a change to the analytical system has to monitoring their analytical occurred. system?  Define the criteria required for SQC to monitor change?  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 Worksheet 1: Definition of Terms Quiz203 Time: 5 minutes Instructions: Match each term (in the left column) with the appropriate definition item (in the right column). Write the letter of the appropriate definition in the space next to the term. Term Definition 1. Accuracy A. Amount of scatter or variability __ _ around the mean, measured in units and percent - 2. Precision B. The average of our data or the __ _ best estimate of the true value produced by our laboratory 3. Imprecision C. The specific substance you are __ _ interested in measuring 4. Inherent Imprecision D. Difference between the average __ _ measured value (the mean) and the true value 5. Single Population of Data E. The closeness of agreement __ _ between results in a set of replicate measurements; typically expressed in terms of imprecision 6. Mean F. Normal distribution of data that is __ _ symmetrical and clusters around the mean 7. Bias G. Quantifies the variation or __ _ dispersion of values measured in units 8. Analyte H. Closeness between the measured __ _ value for an analyte and the true value for that analyte 9. Standard Deviation (SD) I. Amount of imprecision __ _ encountered when the analytical system is stable due to slight changes in temperature, voltage, reagent or sample pipetting - 10 Gaussian Distribution J. Random points exhibiting a normal __ _ distribution with a specific mean . and SD HILS1746 Effective Date: 12/06/2016 M e a s u r e m e n t Y-axis X-axis (time of collection) HILS1746 Effective Date: 12/06/2016 Levey-Jennings Graph 203.0 202.5 202.0 201.5 201.0 200,5 200,0 EQA 199.5 199,0 198.5 T 198,0 TEa Target TEa 0 20 30 40 50 Total Error s an acceptable change vs. acceptable change (TE < TEA) mines how far the mean can sift fore erroneous results are reported c and Sigma-metric) appropriate control rules based on e SEc or Sigma for that method HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 3 Match It Up: A Rule Violation Game PURPOSE: QC rules alert laboratorians to a change in the analytical system (measurement procedure). In this activity, participants match charts to the correct QC rule violation and identify the type of error. This activity supports the following laboratory management tasks and SLIPTA checklist items. Management Tasks 1.4 Assess personnel competency against standards and determine corrective action and training needs 2.7 Ensure reagents and chemicals are stored properly 3.4 Enforce good stock management practices (proper storage, stock cycling, inspection of incoming orders, etc.) 3.5 Inspect quality of existing inventory and dispose of expired test kits, reagents, supplies and equipment according to policy 5.2 Ensure proper preventive maintenance (i.e., cleaning, proper shutdown) on instruments when used 5.3 Perform and record troubleshooting on malfunctioning equipment 5.4 Review and sign maintenance logs to ensure regular preventive maintenance and timely repairs 5.6 Follow up on all corrective action – see if equipment is properly functioning, observe for trends or determine training needs 6.2 Ensure that QC material is tested according to SOP 6.4 Validate new equipment, reagents, and supplies 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.7 Review records of environmental checks & QC trends to assess impact on testing and take corrective action 6.9 Monitor reagent performance 6.13 Periodically observe/assess accuracy of staff performance and take corrective action Checklist Items 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? Laboratory Strengthening Checklist 3.7 Laboratory Staff Training Is there a system for training? 3.8 Staff Competency Assessment and Retraining Is there a system for ====== ========= competency assessment? 5.1 Adherence to Proper Equipment Protocol Is equipment installed and placed as specified in the operator’s manuals and uniquely labeled or marked? 5.6 Equipment Maintenance Records Is relevant equipment service information readily available in the laboratory? 5.10 Equipment Preventive Maintenance Is routine user preventive maintenance performed on all equipment and recorded according to manufacturer’s minimum requirements? 5.11 Equipment Service Maintenance Is equipment routinely serviced according to schedule as per the minimum manufacturer recommendations by qualified and competent personnel and is this information documented in appropriate logs? 5.12 Equipment Malfunction - Response and Documentation Is equipment malfunction resolved by the effectiveness of the corrective action program and the associated root cause analysis? HILS1746 Effective Date: 12/06/2016 7.10 Product Expiration Are all reagents/test kits in use (and in stock) currently within the manufacturer-assigned expiration or within stability? 8.7 Documentation of Examination Procedures Are examination procedures documented in a language commonly understood by all staff and available in appropriate locations? 8.8 Reagents Acceptance Testing Is each new reagent preparation, new lot number, new shipment of reagents or consumables verified before use and documented? 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)?  KEY MESSAGES Can they: Results of QC samples are analyzed to assess and  Analyze L-J charts and  alert us to changes in accuracy and precision (method histograms and identify the rule performance). violation?  The nomenclature for rule violations includes the  Determine if the rule indicates a number of data points involved and the applied control systematic or random error? limits usually expressed as the mean ± a multiple of the  Understand how SE and RE SD. assists with troubleshooting?  Different rules indicate if the change is random or  ACTIVITY OBJECTIVES systematic. This information is used to assist with MET? troubleshooting an out-of-control run. HILS1746 Effective Date: 12/06/2016 99.7% 95% 68% 2.5% 13.5% 34% 34% 13.5% 2.5% -4SD -3SD -2SD -1SD X 1SD 2SD 3SD 4SD Handout 1: .ell Curve with 5istribution HILS1746 Effective Date: 12/06/2016 +3 SD 2.5% + 2 SD 13.5% + 1 SD 34% 99.7% 95% 68% X 34% 1 SD - 13.5% 2 SD - 2.5% -3 SD ...... DAY or RUN 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Handout 2: L-J Chart with 5istribution HILS1746 Effective Date: 12/06/2016 Rule Violation Answer Sheet306 Part I Part II (SE or RE) 1:2s or 1 ____ ____ ___ 2s 2:2s or 2 ____ ____ ___ 2s 1:3s or 1 ____ ____ ___ 3s 4:1s or 4 ____ ____ ___ 1s 2 of 3:2s or 2 of 3 ____ ____ ___ 2s 3:1s or 3 ____ ____ ___ 1s 10x ____ ____ ___ 7T ____ ____ ___ R:4s or R ____ ____ ____ ___ 4s HILS1746 Effective Date: 12/06/2016 SYSTEMATIC ERRORS (SE) RANDOM ERRORS (RE) O   Change in reagent lot Improperly mixed/dissolved Change in calibrator lot reagent   Wrong calibrator values  Air bubbles in reagents and reagent lines, sampling or reagent  Improperly prepared reagents Deterioration of reagents or syringes   control material Pipette tips not fitting properly  Inappropriate storage of  A clogged pipettor (clot) reagents   Pipette maladjustments or Imprecise pipettor  misalignment Unstable temperature and Change in temperature of incubation   incubators and reaction blocks Unstable power supply  Failing light source Poor operator technique   Defective disposable consumables  Changes in procedure from one operator to another Sample evaporation  Outdated reagents Improper mixing of processed Inadequately cleaned samples  glassware Incorrect reconstitution of the control material HILS1746 Effective Date: 12/06/2016 Troubleshooting Out-of- Control Runs Step 1. What rule has been violated? Step 2. Is it a systematic or random error? Step 3. What is the potential cause of the error? Step 4. If more than one analyte is out-of- control, do the analytes have any common factors during testing? Step 5. Did we change something recently? Step 6. Once we resolved the problem, did we document the problem and its resolution? HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 4 The Front-line Worker PURPOSE: Front-line workers assess whether or not an analytical run is acceptable. QC protocols developed by laboratory management must standardize the decision criteria used by the workers. In this activity, participants are provided with QC charts and a Westgard Multirule algorithm and must apply the decision-making process to determine acceptability of the analytical run. This activity supports the following laboratory management tasks and SLIPTA checklist items. Management Tasks 1.4 Assess personnel competency against standards and determine corrective action and training needs 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.13 Periodically observe/assess accuracy of staff performance and take corrective action Checklist Items 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? Laboratory Strengthening Checklist 3.7 Laboratory Staff Training Is there a system for training? 3.8 Staff Competency Assessment and Retraining Is there a system for competency assessment? 8.7 Documentation of Examination Procedures Are examination procedures documented in a language commonly understood by all staff and available in appropriate locations? 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)?  KEY MESSAGES Can they: Multirules provide a high level of error detection  Determine whether an analytical run is  capability while at the same time keeping false rejection acceptable or not based upon a given rates low. QC protocol?  Recognize how the 1:2s warning rule  Laboratory management must provide clear instructions to the front-line workers regarding QC decision criteria. can be used as a scanning tool? The 1:2s warning rule is a scanning tool that can save  Understand when a multirule should be  time and effort in manual QC applications. applied to a measurement procedure?  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 A QC protocol (plan) describes how the analytical staff routinely performs QC and responds to QC data. 1. Define the statistical QC rule to be applied and its frequency. 2. Define how the control materials will be analyzed. 3. Define how to interpret the rules. 4. Define what troubleshooting actions should be taken, including documentation. Example QC Protocol for XYZ Analyzer Statistical QC Rules 1:3s ALT, AST, Bili-Tot, Creat 1:2.5s TP, CK 1:3s/2:2s/R:4s/4:1s Glucose Analysis of Control Analyze one sample of the Precinorm Normal and one sample of the Material Precinorm Abnormal controls for a total of 2 control measurements in each analytical run Interpretation of Results 1) Scan the chart for any measurement greater than ± 2SD. If no measurement is found, then accept the run and report patient results. 2) If a measurement is found, inspect the control data using the specified QC rejection rule for the analyte. a. Within current run inspection • apply the 1:3s from each material • apply the 1:2.5s from each material • apply the 2:2s and R:4s rules across materials b. Across-run inspection • apply the 2:2s rule within each material across the last two runs • apply the 4:1s rule within each material across the last 4 runs • apply the 4:1s from the last two runs and the two measurements on each material 3) If none of the rules from Step 2 are violated when applying the specified rejection rule for the analyte, accept the run and report patient test results. If a rule for an analyte is violated in Step 2, reject the run and do not report patient test results for that analyte. Troubleshooting Actions 1) Circle the analytical run number to indicate the run was rejected 2) Complete a Daily QC Investigation Report to identify the type of error occurring and the possible causes for the error. 3) Refer to the troubleshooting section of the operator’s manual as needed. 4) Take corrective action based on the investigational report. 5) File the investigational report in the instrument’s log under the corrective action section. HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 5 It Begins with the Right Chart PURPOSE: Using the QC tool, L-J Chart, one can visually inspect a quantitative method’s current accuracy and precision to quickly assess if a change is occurring within the system. In this activity, participants learn how to correctly create this tool, how to visually assess it, and how to avoid pitfalls commonly encountered with its set-up. This activity supports the following laboratory management tasks and SLIPTA checklist items. Management Tasks 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.7 Review records of environmental checks & QC trends to assess impact on testing and take corrective action 6.9 Monitor reagent performance Checklist Items 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? Laboratory Strengthening Checklist 5.10 Equipment Preventive Maintenance Is routine user preventive maintenance performed on all equipment and recorded according to manufacturer’s ======= ============ minimum requirements? 8.8 Reagents Acceptance Testing Is each new reagent preparation, new lot number, new shipment of reagents or consumables verified before use and documented? 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)?  KEY MESSAGES Can they:  Create a properly prepared L-J  One of the most widely used tools with QC is the L-J chart. This tool allows one to visually assess for a chart? change in a method’s current performance.  Visually assess a chart to see if When L-J charts are regularly examined, a small there has been a change in the  change may be recognized long before a QC rule method’s performance? violation occurs.  Recognize common pitfalls If the assigned values used on the chart do not reflect when the assigned values do  the observed values of the current performance, then not equal the observed values the QC rules are unable to function properly to detect of a stable system? errors. The data will no longer appear normally  ACTIVITY OBJECTIVES distributed. MET? HILS1746 Effective Date: 12/06/2016 For Each QC Data Point We Need to Decide… SYSTEa IS STABLE SYSTEa IS UNSTABLE – SIDNICICANT CHANDE Nh RULE VIhLATIhN RULE VIhLATIhN hCCURRED hN L-W CHART HILS1746 Effective Date: 12/06/2016 Creating a L-J Chart 1.Label the chart • Name of the laboratory • Name of the Instrument and Identifier # NMme of the IMNorMtory • Name of Test NMme Mnd Identifier # of Instrument • Units AnMlyte: Control MMteriMl: Units: • Name & expiration date of the control material Iot #: Exp DMte:  Mssign SD Mssign • Assigned mean and standard deviation used on the chart From: Through: TMrget TEM • Acceptable control limits +4 SD • Time period covered by the chart 2.Label the X-axis in terms of time period used or run +3 SD 3. Scale the X-axis into evenly sized increments numbering sequentially C o + 2 SD 4. Label the Y-axis Control Value and for X, ±1 SD, ±2 n SD, ±3 SD, ±4 SD t 5.Scale the Y-axis from lowest to highest expected r + 1 SD control values as follows using the assigned mean and o standard deviation (SD) so that the mean is located at l X the center of your graph: • Subtract the SD from the mean; this is the -1SD V 1 SD - M • Add the SD to the mean; this is the +1SD l • Multiply the SD by 2, and then subtract that value u 2 SD - from the mean. This is the -2SD e • Multiply the SD by 2 and then add that value to the -3 SD mean. This is the +2SD • Repeat this process for ± 3SD, ± 4SD using a factor of 3 and 4 respectively. +4 SD 6. Write the values obtained for X, ±1 SD, ±2 SD, ±3 SD, Run # 1 2 3 4 D 6 7 8 9 10 11 12 13 14 1D 16 17 18 19 20 ±4 SD next to the correct label on the chart. DMte VMlue 7. Draw lines for mean and SDs InitMls 8. Begin plotting analyzed QC results. Document # HILS1746 Effective Date: 12/06/2016 Cape Clinic Laboratory 1 Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 90 SD assigned 3 +4SD +3SD + 2SD -- + 1SD X -1SD 2SD - -3SD -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 97 90 91 95 90 85 91 94 89 91 85 89 Directions: Using the assigned mean and SD provided, calculate the control limits. On the Y-axis, write those numerical values on the appropriate line. Plot data points 21to 30. Visually assess the completed L-J chart before answering the questions. HILS1746 Effective Date: 12/06/2016 1 Based on the visual pattern and not QC rules, 1) Do the data in runs 21 to 30 show a change in accuracy? YES / NO 2) Do the data in runs 21 to 30 show a change in precision? YES / NO Complete the following for data runs 21-30 only 68% of the data points lie between ________ and ________ . 95% of the data points lie between ________ and ________ . What QC Rule(s) are violated on the graph? Run Number Rule(s) Violated What Type of Error (SE or RE) Do the rules violated support your visual assessment? On the back of this page, draw the Gaussian curve of the population(s) represented by this L- J chart. HILS1746 Effective Date: 12/06/2016 2 Cape Clinic Laboratory Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 90 SD assigned 3 +4SD +3SD + 2SD + 1SD X -1SD 2SD - -3SD -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 95 90 96 91 87 82 96 85 88 101 82 91 .......... Directions: Using the assigned mean and SD provided, calculate the control limits. On the Y-axis, write those numerical values on the appropriate line. Plot data points 21to 30. Visually assess the completed L-J chart before answering the questions. HILS1746 Effective Date: 12/06/2016 2 Based on the visual pattern and not QC rules, 3) Do the data in runs 21 to 30 show a change in accuracy? YES / NO 4) Do the data in runs 21 to 30 show a change in precision? YES / NO Complete the following for data runs 21-30 only 68% of the data points lie between ________ and ________ . 95% of the data points lie between ________ and ________ . What QC Rule(s) are violated on the graph? Run Number Rule(s) Violated What Type of Error (SE or RE) Do the rules violated support your visual assessment? On the back of this page, draw the Gaussian curve of the population(s) represented by this L- J chart. HILS1746 Effective Date: 12/06/2016 3 Cape Clinic Laboratory Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 90 SD assigned 3 +4SD +3SD ............. + 2SD .............. + 1SD X -1SD 2SD - -3SD -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 85 90 93 88 87 89 96 92 95 88 90 89 87 92 90 94 88 90 84 85 86 81 86 92 89 86 82 89 86 .................. ........... Directions: Using the assigned mean and SD provided, calculate the control limits. On the Y-axis, write those numerical values on the appropriate line. Plot data points 21 to 30. Visually assess the completed L-J chart before answering the questions. HILS1746 Effective Date: 12/06/2016 3 Based on the visual pattern and not QC rules, 5) Do the data in runs 21 to 30 show a change in accuracy? YES / NO 6) Do the data in runs 21 to 30 show a change in precision? YES / NO Complete the following for data runs 21-30 only 68% of the data points lie between ________ and ________ . 95% of the data points lie between ________ and ________ . What QC Rule(s) are violated on the graph? Run Number Rule(s) Violated What Type of Error (SE or RE) Do the rules violated support your visual assessment? On the back of this page, draw the Gaussian curve of the population(s) represented by this L- J chart. HILS1746 Effective Date: 12/06/2016 Directions: Based on the information provided by the L-J chart, determine the assigned mean and assigned SD in umol/L. Based on your visual examination of the chart, approximate the observed mean and observed SD in umol/L. Fill in the blank spaces with your responses. Draw the Gaussian curve using the assigned mean and assigned SD on the L-J chart. Draw another Gaussian curve using the observed mean and observed SD. Label the following sections of the curves as: true accept, false reject, false accept. Refer to Job Aid 1: Possible Outcomes for assistance. Cape Clinic Laboratory 4 Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned SD assigned X observed SD observed 99.0 +4SD 96.0 +3SD 93.0 + 2SD .......... 90.0 + 1SD .......................... .............. 87.0 X 84.0 -1SD 81.0 2SD - 78.0 -3SD 75.0 -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 97 90 91 95 90 85 91 94 89 91 85 89 ............. HILS1746 Effective Date: 12/06/2016 4 Using the assigned mean, what are the values (umol/L) for ± 2SD control limits? Using the observed mean, what are the values (umol/L) for ± 2SD ? We expect that 68% of the measurements lie between -1SD and +1SD control limits when the assigned = observed on the chart. However on this chart where the Mean assigned ≠ Mean observed, 68% of the data points lie between what control limits? 95% of the data points lie between what control limits? What QC Rule(s) are violated on the chart when the assigned mean ≠observed mean Run Number Rule(s) Violated Interpret the Control Measurement (true accept, false reject, or false accept) The data points used in Chart #4 are identical to the data points used in Chart #1 (a stable population normally distributed). When compared to Chart #1, there is an increase / decrease in rule violations and this is because…… Circle one If the next measurement (Run #31) is 81 umol/L, what type of outcome would be given to that value? Refer to Job Aid 1: Possible Outcomes, if needed. Draw the Gaussian curves of the assigned and observed populations on the back of the worksheet. Label the following sections of the curves as: true accept, false reject, false accept. HILS1746 Effective Date: 12/06/2016 Directions: Based on the information provided by the L-J chart, determine the assigned mean and assigned SD in umol/L. Based on your visual examination of the chart, approximate the observed mean and observed SD in umol/L. Fill in the blank spaces with your responses. Draw the Gaussian curve using the assigned mean and assigned SD on the L-J chart. Draw another Gaussian curve using the observed mean and observed SD. Label the following sections of the curves as true accept and false accept. Refer to Job Aid 1: Possible Outcomes for assistance. 5 Cape Clinic Laboratory Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned SD assigned X observed SD observed 114.0 +4SD 108.0 +3SD 102.0 + 2SD -- - - - - 96.0 + 1SD 90.0 X .......... 84.0 -1SD 78.0 2SD - 72.0 -3SD 66.0 -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 97 90 91 95 90 85 91 94 89 91 85 89 . HILS1746 Effective Date: 12/06/2016 5 Using the assigned SD, what are the values (umol/L) for ± 2SD control limits? Using the observed SD, what are the values (umol/L) for ± 2SD ? We expect that 68% of the measurements lie between -1SD and +1SD control limits when the assigned = observed on the chart. However on this chart where the SD assigned ≠ SD observed, 68% of the data points lie between what control limits? 95% of the data points lie between what control limits? What QC Rule(s) are violated on the chart when the assigned SD ≠observed SD Run Number Rule(s) Violated Interpret the Control Measurement (true accept, false reject, or false accept The data points used in Chart #5 are identical to the data points used in Chart #1 (a stable population normally distributed). When compared to Chart #1, there is an increase / decrease in rule violations and this is because…… Circle one If the next measurement (Run #31) is 78 umol/L, what type of outcome would be given to that value? Refer to Job Aid 1: Possible Outcomes, if needed. Using your SDobserved, how many multiples of SD does the 78 umol/L control measurement lie from the meanobserved? Draw the Gaussian curves of the assigned and observed populations on the back of the worksheet. Label the following sections of the curves as true accept and false accept. HILS1746 Effective Date: 12/06/2016 Directions: Based on the information provided by the L-J chart, determine the assigned mean and assigned SD in umol/L. Based on your visual examination of the chart, approximate the observed mean and observed SD in umol/L. Fill in the blank spaces with your responses. Draw the Gaussian curve using the assigned mean and assigned SD on the L-J chart. Draw another Gaussian curve using the observed mean and observed SD. Label the following sections of the curves as true accept and false reject. Refer to Job Aid 1: Possible Outcomes for assistance. 6 Cape Clinic Laboratory Creatinine (umol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned SD assigned X observed SD observed 98.0 +4SD 96.0 +3SD 94.0 + 2SD 92.0 + 1SD 90.0 X 88.0 -1SD ............. 86.0 2SD ............ - 84.0 -3SD 82.0 -4SD Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 93 84 90 93 88 86 88 95 92 94 88 90 89 87 91 90 94 88 97 90 91 95 90 85 91 94 89 91 85 89 ........ HILS1746 Effective Date: 12/06/2016 6 Using the assigned SD, what are the values (umol/L) for ± 2SD control limits? Using the observed SD, what are the values (umol/L) for ± 2SD ? We expect that 68% of the measurements lie between -1SD and +1SD control limits when the assigned = observed on the chart. However on this chart where the SD assigned ≠ SD observed, 68% of the data points lie between what control limits? 95% of the data points lie between what control limits? What QC Rule(s) are violated on the chart when the assigned SD ≠observed SD Run Number Rule(s) Violated Interpret the Control Measurement (true accept, false reject, or false accept The data points used in Chart #6 are identical to the data points used in Chart #1 (a stable population normally distributed). When compared to Chart #1, there is an increase / decrease in rule violations and this is because…… Circle one Draw the Gaussian curves of the assigned and observed populations on the back of the worksheet. Label the following sections of the curves as true accept and false rejects. HILS1746 Effective Date: 12/06/2016 Assigned vs. Observed – What is the difference? SD observed cally sing data s) HERE WE ARE accuracy and the method your method’s e HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 6 Parallel Testing: Making History PURPOSE: Only when the system is known to be stable can data be collected to calculate the observed mean and SD of a new lot number of QC material. In this activity, participants create a parallel testing SOP document based on their observation of a role-played laboratory scenario. % CV is introduced and used in the SD computation of short shelf-life control materials. This activity supports the following laboratory management tasks and SLIPTA checklist items. Management Tasks 6.1 Ensure that the Quality Manual with quality assurance policies and procedures is accessible to and reviewed by all staff 6.2 Ensure that QC material is tested according to SOP 6.3 Establish acceptable ranges for control material 6.4 Validate new equipment, reagents, and supplies 6.5 Track test performance (e.g., Levy-Jennings chart) for trends Checklist Items 1.5 Laboratory Policies and Standard Operating Procedures Are policies and/or standard operating procedures (SOPs) for laboratory functions, technical and Laboratory Strengthening Checklist managerial procedures current, available and approved by authorized personnel? 1.6 Policy and SOPs Accessibility Are policies and SOPs easily accessible/available to all staff and written in a language commonly understood by respective staff? 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? 8.8 Reagents Acceptance Testing Is each new reagent preparation, new lot number, new shipment of reagents or consumables verified before use and documented? 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)?  KEY MESSAGES Can they:  Understand the importance of  New lot numbers of control material should be tested in parallel with the currently in-use control material. parallel testing?  Parallel testing is only performed when the instrument  Identify the steps to perform is operationally stable. parallel testing at their site?  Package inserts serve as a guideline only and should  Calculate the %CV? not be used when creating the L-J chart for the new  ACTIVITY OBJECTIVES control. MET?  %CV tells us WHERE WE ARE in terms of precision. HILS1746 Effective Date: 12/06/2016 We expect QC materials to provide information about what is occurring with the measurement procedure. In other words, we expect the performance of the QC materials to mirror the same effects as what is occurring to our patient samples. To do this, QC materials should: [i] 1. Mimic the matrix and viscosity of the patient samples being tested a. Matrix—the base from which control materials are prepared in addition to the preservatives added for stability b. Matrix effect – the influence of the control material’s matrix, other than the concentration of the analyte, on the measurement procedure to produce differing results when compared to other methods while still producing consistent results on patient samples 2. Be both physically and chemically sensitive to changes in the measurement procedure as patient samples 3. Contain concentrations of analytes at or near medical decision points 4. Be available in one lot number that is stable for an extended period of time 5. Be available at different concentration levels to assess the measuring range of the method 6. Remain stable before and after opening a vial as indicated by the manufacturer 7. Produce minimal vial-to-vial variability [i] Brooks, Zoe (2003). Quality Control – from Data to Decisions. Basic Concepts (Section 2, Topic 2) In addition to the above stated qualities, other considerations should be weighed for you specific site, such as: 1. Use of lyophilized (freeze-dried) controls a. Usually less costly per box than liquid b. Require a special diluent or deionized Type I water c. Require availability of clean Class A Volumetric pipets and pipetting bulbs d. Require staff that is capable of i. Accurately pipetting manually ii. Strictly adhering to reconstitution and mixing instructions provided by the manufacturer e. May experience more vial-to-vial variability (increase imprecision) especially if improper handling and reconstitution occurs f. Frequently has a shorter opened vial expiry interval iii. May result in discarding unused portion (hidden cost consideration) 2. Use of liquid controls a. Usually more costly per box than lyophilized b. Eliminates many of the handling and reconstitution errors c. Influence of matrix effect may be greater with the method you use d. Frequently has a longer opened vial expiry interval i. May discard less or none of the product if consumed within opened expiry date 3. Frequency of lot number changes a. Performing parallel testing takes time and money (costs of performing testing on QC materials) b. With each QC material lot number change, lose access to summary or cumulative data HILS1746 Effective Date: 12/06/2016 c. Recommend to purchase a year supply of the same lot number, when possible i. Desired expiration date should be specified at time of purchase ii. Storage issues iii. Difficulties encounter with setting up a standing order with the vendor 4. Vendor considerations a. Availability of an inter-laboratory comparison program b. Provide troubleshooting support c. Ability to accommodate standing orders d. Ability to sequester specified lot number and automatically ship and bill as outlined in the purchase agreement HILS1746 Effective Date: 12/06/2016 Heading Purpose Answers the question: Title of Write a title that clearly defines Procedure the content of the SOP. State the reasons for performing Test Summary the test. Why do we perform parallel testing Outline the scientific principle Principle involved in the process. How does parallel testing work? Reagents, List by category any equipment, Materials, & reagents, or materials required What items do we need to perform Equipment for following the SOP. parallel testing at our site? Give the instructional steps, in Procedure order, required to complete the procedure How do we perform parallel testing? HILS1746 Effective Date: 12/06/2016 Applying %CV – The Great Equalizer Using %CV at the cross-over between 2 different lot #s of control material when materials have a short out-of date shelf life 1) Perform cross-over when the system is stable. If the system is unstable, then you should investigate and troubleshoot the system and not perform parallel testing. 2) Calculate the % CV of the current lot number (currently in-use) by using the Mean obs and SD obs. Mean obs SD obs %CV 100 umol/L 5 umol/L 3) Calculate the Mean obs using the new lot number by analyzing the QC material 8-10 times. Confirm that the values obtained fall within the initial range using the package insert as a guideline only. Run # Value (umol/L) Run # Value (umol/L) 1 115 6 116 2 117 7 114 3 114 8 117 4 116 9 115 5 115 10 115 Mean obs new lot # = 4) Using the %CV from the current method and the Mean obs from the new lot number (assuming the system has remained stable with the same amount of inherent randomness), calculate the new lot number’s SD. CV% current lot # = (SD / x obs new lot #) × 100% ¯ Solving for SD (CV% current lot # ÷ 100%) × x obs new lot # = SD new lot # ¯ SD new lot # = 5) Calculate your ranges for ± 1SD, ± 2 SD, ± 3 SD, ± 4 SD. To avoid errors with rounding, perform the rounding off at the conclusion ± 1SD ± 2 SD ± 3 SD ± 4 SD Range 6) Verify the mean and SD on the new lot number when data from a longer period of stable operation becomes available. HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 7 Total Error (TE) PURPOSE: Total error (TE) combines bias and imprecision to quantify the largest variation from the true (target) value. In this activity, participants are introduced to bias and true value. Using the information supplied by 3 key numbers, participants calculate the TE in units and percent. Additionally, participants explore the impact that bias and imprecision can have on TE and relate site-specific processes that affect TE to checklist items. This activity supports the following laboratory management tasks and SLIPTA checklist items. Management Tasks 1.4 Assess personnel competency against standards and determine corrective action and training needs 1.6 Meet with staff individually to communicate expectations, provide feedback, coaching, or on-the-job training to ensure competency and productivity 1.7 Provide/coordinate new-hire orientation and training to staff 2.4 Ensure appropriate physical work environment for testing 3.4 Enforce good stock management practices (proper storage, stock cycling, inspection of incoming orders, etc.) 3.5 Inspect quality of existing inventory and dispose of expired test kits, reagents, supplies and equipment according to policy 5.2 Ensure proper preventive maintenance (i.e., cleaning, proper shutdown) on instruments when used 5.3 Perform and record troubleshooting on malfunctioning equipment 5.4 Review and sign maintenance logs to ensure regular preventive maintenance and timely repairs 5.5 Take corrective actions or issue repair orders and record all issues 5.6 Follow up on all corrective action – see if equipment is properly functioning, observe for trends or determine training needs 6.1 Ensure that the Quality Manual with quality assurance policies and procedures is accessible to and reviewed by all staff 6.2 Ensure that QC material is tested according to SOP 6.3 Establish acceptable ranges for control material 6.4 Validate new equipment, reagents, and supplies 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.7 Review records of environmental checks & QC trends to assess impact on testing and take corrective action 6.8 Review occurrence log for patterns/trends and take corrective action 6.9 Monitor reagent performance 6.10 Customize site-specific SOPs as needed 6.11 Ensure that SOP are read and understood by staff 6.12 Enroll in EQA program, monitor results, and take corrective actions 6.13 Periodically observe/assess accuracy of staff performance and take corrective action 10.1 Maintain a library of documents (policies, guidelines, SOPs, references, etc.); review and update annually 10.3 Assure proper record retention, rotation to storage, and disposal according to protocol HILS1746 Effective Date: 12/06/2016 Checklist Items 1.3 Document and Information Control System Does the laboratory have a system in place to control all documents and information from internal and Laboratory Strengthening Checklist external sources? 1.6 Policy and SOPs Accessibility Are policies and SOPs easily accessible/available to all staff and written in a language commonly understood by respective staff? 1.7 Policies and SOPs Communication Is there documented evidence that all relevant policies and SOPs have been communicated to and are understood and implemented by all staff as related to their responsibilities? 1.9 Discontinued Policies and SOPs Are invalid or discontinued policies and procedures clearly marked / identified and removed from use and one copy retained for reference purposes? 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? 2.2 Management Review Does the laboratory management perform a review of the quality system at a management review meeting at least annually? 3.7 Laboratory Staff Training Is there a system for training? 3.8 Staff Competency Assessment and retraining Is there a system for competency assessment? 5.6 Equipment Maintenance Records Is relevant equipment service information readily available in the laboratory? 5.9 Equipment Calibration and Metrological Traceability Protocol 5.10 Equipment Preventive Maintenance Is routine user preventive maintenance performed on all equipment and recorded according to manufacturer’s minimum requirements? 5.11 Equipment Service Maintenance Is equipment routinely serviced according to schedule as per the minimum manufacturer recommendations by qualified and competent personnel and is this information documented in appropriate logs? 5.13 Equipment Repair Monitoring and Documentation 5.15 Manufacturer’s Operator Manual Are the manufacturer’s operator manuals readily available to testing staff and, available in the language understood by staff? 7.7 Laboratory Inventory System 7.8 Storage Area Are storage areas set up and monitored appropriately? 7.9 Inventory Organization and Wastage Minimization Is First-Expiration-First- Out (FEFO) practiced? 7.10 Product Expiration Are all reagents/test kits in use (and in stock) currently within the manufacturer-assigned expiration or within stability? 8.7 Documentation of Examination Procedures Are examination procedures documented in a language commonly understood by all staff and available in appropriate locations? 8.8 Reagents Acceptance Testing Is each new reagent preparation, new lot number, new shipment of reagents or consumables verified before use and documented? 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)? 8.12 Are environmental conditions checked and reviewed accurately? 8.13. Have acceptable ranges been defined for all temperature- dependent equipment with procedures and documentation of action taken in response to out of range temperatures? 8.14. Does the laboratory participate in inter-laboratory comparison program or alternative assessment systems for all tests? 9.3 Analytic System/Method Tracing When more than one instrument is in use for the same test, are test results traceable to the equipment used for testing? 12.4 Is the physical work environment appropriate for testing? HILS1746 Effective Date: 12/06/2016  KEY MESSAGES Can they: 3 key numbers are required to calculate TE – mean,  Calculate bias and absolute  SD, and true value. bias?  Calculate TE in units and  The target value is our best estimate using sources that each has its own limitation. percent?  TE is the total variation of our value from the true value.  Identify ways to reduce bias and imprecision at their site?  The checklist can serve as a guide to target areas to reduce or eliminate bias and reduce imprecision.  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 How to Calculate TE I. Need 3 key numbers II. Selecting the True Value (Target Value) a. Mean a. The value published in the control’s package insert for that lot 1) A fact calculated from observed data number points 1) Limitations 2) Will change due to a) May not be stated in insert 1) recalibration b) May be based on insufficient data performed by manufacturer 2) Introduction of reagent or calibrator b. The value obtained from an interlaboratory comparison lot number changes program 3) Changes to the instrumentation 1) Based on your peer group performing the same method b. SD for the test 1) A fact calculated from observed data 2) Limitations points a) Peer group may be too small to accurately reflect a true value 2) Certain amount of imprecision is expected; inherent with the analytical system b) Outlier laboratories in a small peer group will statistically affect the calculations c. True Value – target value c) An interlaboratory program may not be available by 1) Used with the mean to determine bias the manufacturer 2) Selected from the most appropriate source c. Cumulative mean from your laboratory’s data when the or combination of sources available for system has been stable the test 1) Limitations 3) Remains constant for the life of the a) Difficult to correctly determine accuracy from specific lot number of QC material; it is the internal sources alone true value after all b) The stable system you thought you had may actually 4) The value used in comparison with the have inherent problems and constant bias information gained after a change has c) Even though the data has been shown to be occurred to monitor the analytical system consistent from previous results over an extended time period, you may actually be consistently wrong. d) Method validation or verification studies were never performed to evaluate accuracy HILS1746 Effective Date: 12/06/2016 Worksheet 1: Calculate TE Analyte Mean True Value SD (target) WBC, Total Count (cell * 109 /L) 18.0 18.2 1.2 Potassium (mmol/L) 3.8 3.5 0.1 Creatinine (umol/L) 90 90 4 Platelet Count (cell * 109 /L) 160 150 7 Glucose (mmol/L) 6.5 6.7 0.2 Calcium (mmol/L) 2.26 2.25 0.03 HILS1746 Effective Date: 12/06/2016 Worksheet 1: Calculate TE Directions: Using the key numbers supplied on the previous page, complete the following table. The first two rows have been populated as a guide. Analyte Bias l Bias l TE in units %CV Abs Bias % TE in % (abs bias) x -True Value | x -True Value| Abs Bias + (1.65*SD) (SD /X)*100% (Abs Bias /True Value) *100% Abs Bias % + (1.65* %CV) ¯ ¯ WBC 18.0 – 18.2 = 0.2 0.2 + 1.65 *1.2 (1.2 (0.2 / 18.2)* 100% 1.1% + 1.65 *6.7% -0.2 2.2 /18.0)*100% =12.2% = =1.1% 6.7% = Potassium 3.8 – 3.5 0.3 0.3 + 1.65 *0.1 (0.1/ 3.8)*100% (0.3/ 3.5) * 100% 8.6% + 1.65 *2.6% = 0.3 =0.5 2.6% 8.6% 12.9% = = = Creatinine Platelet Glucose Calcium HILS1746 Effective Date: 12/06/2016 Worksheet 2: Impact of Bias and Imprecision on TE Directions: Use the following key numbers to complete the table below. Please note there are 2ways to calculate %TE. The second way is introduced in this worksheet. On the back of this worksheet, illustrate the method performance of each laboratory. Include the target value, the Gaussian curve, an arrow representing SE (if applicable), an arrow representing RE, and an arrow representing TE. Analyte: Platelet Count (cell * 109 /L) Mean True Value SD (target) Lab A 150 150 5 Lab B 145 150 5 Lab C 150 150 10 Lab D 145 150 10 Analyte Bias l Bias l TE in units TE in % %CV Abs Bias % TE in % (abs bias) ¯ ¯ x -True Value | x -True Value| Abs Bias + (1.65*SD) (TE /True Value) (SD /X)*100% (Abs Bias /True Value) Abs Bias % + (1.65* %CV) *100% *100% Lab A (8.3 / 150) *100% =5.5% Lab B Lab C Lab D HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 8 Putting the TEA into Quality PURPOSE: To control quality, one must first determine what quality is needed for a test. In this activity, participants will explore what information the fourth key number must provide to define the quality required. Additionally, participants will use a laboratory scenario to assess if the method remains acceptable for the intended clinical use of the test result. This activity supports the following laboratory management tasks and SLIPTA checklist items. Management Tasks 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.7 Review records of environmental checks & QC trends to assess impact on testing and take corrective action Checklist Items 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? Laboratory Strengthening Checklist 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)?  KEY MESSAGES Can they: TEA defines the quality specifications specific to the  Identify the four key numbers  analyte being tested to make the analytical result needed for quality? clinically meaningful.  Calculate error limits if given the QC statistics can effectively reflect method performance Target Value and TEA?  if acceptable performance limits are defined and  Compare TE to TEA to applied. determine if the method still  For clinically meaningful results, the TE should be less performs within acceptable than the TEA for that analyte. limits?  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 Sample Calculations to Determine your TEA Limits When TEA is provided in percent(%) Using peer group’s median %CV x 3 (PT Test Results) Using the Target Value (in units), multiply the value by the % TEA to • While ignoring the sample concentrations, list your PT peer convert into units. TEA in units = (TEA %/100%)* Target Value group’s %CV in increasing order • Select the median %CV Lower TEA Limits = Target Value - TEA in units • Upper TEA Limits = Target Value + TEA in units To calculate % TEA, multiply the median %CV by 3 ( % TEA = median %CV * 3) • For example, a QC control for glucose with a target value of 5.6 Follow When TEA is provided in percent(%) instructions to mmol/L, the acceptable performance for glucose is 6.9% using the TEA determine your lower and upper TEA limits. from the desirable biological variation. For example, using results from a PT survey as follows: TEA in units = (6.9%/100%)* 5.6mmol/L = 0.4 mmol/L Sample Mean SD %CV Lower TEA Limits = 5.6 mmol/L – 0.4 mmol/L = 5.2 mmol/L C-3 1.2 0.08 6.7% Upper TEA Limits = 5.6 mmol/L + 0.4 mmol/L = 6.0 mmol/L C-5 4.2 0..33 7.9% C-2 1.0 0.08 8.0% median When TEA is provided in units (e.g. CLIA) For example, a QC control for glucose with a target value of 2.2 C-1 0.6 0.05 8.3% mmol/L, CLIA’s acceptable performance for glucose is Target Value ± C-4 3.0 0.26 8.7% 0.3mmol/L or 10% whichever value is greater. % TEA =8.0% * 3 = 24% TEA in units = 0.3 mmol/L Lower TEA Limits = 2.2 mmol/L – 0.3 mmol/L = 1.9 mmol/L Upper TEA Limits = 2.2 mmol/L + 0.3 mmol/L = 2.5 mmol/L Applying Tonk’s Rule [(reference range span)/4/mean of range]* 100% In this example, the concentration limits of 0.3 mmol/L are greater than the percentage value limits of 10%. (10% of 2.2 is 0.2 For example, the CD4 count reference range is 500 mmol/L). Therefore, for this QC control, the concentration acceptable cells/mm3 to 1000 cells/mm3 performance would be applied. % TEA = [(1000 – 500)/4/750]*100% = 16.7% When TEA is provided in ± 3SD Using a survey sample report, select a concentration close to the True Value of your control material. Multiply the SD by 3. HILS1746 Effective Date: 12/06/2016 Lower Upper TEA Target TEA Limit Value Limit Target Value Target Value TEA + TEA - Directions: For each problem, calculate the lower and upper TEA limits with the information provided. Complete the diagrams. 1. Percentage (%) The CLIA TEA for chloride is Target Value ± 5%. QC Target TEA in units Lower TEA Upper TEA Material Value Limit Limit Level I 100 mmol/L Chloride Level II 120 mmol/L Level I Lower Target Uppe TE, Limit Value TE Limit Level II Lower Target Upper TE, Umi Value TE, Limit HILS1746 Effective Date: 12/06/2016 2. Absolute concentration limit The CLIA TEA for potassium is Target Value ± 0.5 mmol/L. QC Target TEA in units Lower TEA Upper TEA Material Value Limit Limit Level I 3.5 mmol/L Potassium Level II 5.8 mmol/L Level l T Lower Target Upper TE, Limit Value TE, Limit Level II Lower L Target Upper TE, Limit Value TE, Limit 3. Distribution of a survey group The CLIA TEA for TSH is Target Value ± 3SD. Results from most recent TSH survey. Values are reported in mU/L unless otherwise specified Sample Mean SD %CV C-1 0.6 0.05 8.3% C-2 1.0 0.08 8.0% C-3 1.2 0.08 6.7% C-4 3.0 0.26 8.7% C-5 4.2 0.33 7.9% HILS1746 Effective Date: 12/06/2016 QC Target TEA in units Lower TEA Upper TEA Material Value Limit Limit Level I 0.90 mU/L TSH Level II 4.50 mU/L Level I Lower Target Upper TE, Limit Value TE, Limit Level II Lower Target Uppe TE, Limit Value TE, Limit 4. Combination of requirements, percentage and absolute concentration units (for lower concentrations) The CLIA TEA for glucose is Target Value ± 0.3 mmol/L or ± 10%, whichever is greater. QC Target TEA in Lower TEA Upper TEA Material Value units Limit Limit Level I 2.78 mmol/L Glucose Level II 5.72 mmol/L Level I Lower L Target Upper TE, Limit Value TE, Limit Level II Lower Target Upper TE, Limit Value TE, Limit HILS1746 Effective Date: 12/06/2016 5. There is no published TEA limit, you decide to use a recent survey report for CD4 and multiply the peer group median %CV by a factor of 3. Results from most recent CD4 survey. Values are reported in cell/uL unless otherwise specified. Sample Mean SD %CV C-1 287.5 21.0 7.3% C-2 850.1 41.6 4.9% C-3 61.0 4.1 6.7% C-4 771.4 40.1 5.2% C-5 162.0 9.7 6.0% QC Target TEA in Lower TEA Upper TEA Material Value units Limit Limit Level I 50 cell/uL CD4 Level II 200 cell/uL Level III 700 cell/uL Level l T Lower Target Upper TEA Limit Value TE, Limit Level II Lower Target Upper TE, Umit Value TE, Limit Level III T Lower Target Upper TE, Limit Value TE. Limit HILS1746 Effective Date: 12/06/2016 By comparing the Total Error (TE) to the Total Allowable Error (TEA), we can assess whether or not a new lot number of reagent provides clinically acceptable results. In this exercise you will be evaluating two different lot numbers and answering questions pertaining to your data. Cape Clinic Laboratory Glucose (mmol/L) L-J Chart for Control XYZ (exp 30/11/XX) on Illustra Chemistry Analyzer (serial # 123) X assigned 5.80 mmol/L SD assigned 0.10 mmol/L Target 5.60 mmol/L TEA 6.9% 6.20 +4SD 6.10 +3SD ............. 6.00 + 2SD - -- - - 5.90 + 1SD 5.80 X 5.70 -1SD 5.60 2SD --- - 5.50 -3SD ................... ........................... 5.40 -4SD Day 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Value 5.90 5.60 5.80 5.90 5.75 5.68 5.70 5.97 5.80 5.92 5.75 5.80 5.78 5.70 5.82 5.80 5.93 5.75 6.00 5.80 5.75 5.95 5.65 5.80 5.91 5.75 5.81 5.80 5.64 5.78 Values (mmol/L) obtained with Control XYZ using the New Lot Number of Reagents Reagent Lot 5.73 5.74 5.75 5.73 5.71 5.73 5.72 5.73 Number A Reagent Lot 5.83 5.84 5.84 5.83 5.84 5.84 5.85 5.85 Number B HILS1746 Effective Date: 12/06/2016 Complete the table using the information from the previous page. Current Mean Current SD Target Value Bias Absolute Bias Current TE TEA Reagent Lot Number A Reagent Lot Number B New Reagent Lot Number Mean Current SD Target Value Bias Absolute Bias New Reagent Lot Number TE TEA TE < TEA (yes or no) By comparing the TE to the target value and TEA limit, we can conclude Reagent A is ____________________ . By comparing the TE to the target value and TEA limit, we can conclude Reagent B is ____________________ . Reagent B demonstrated a smaller shift from the current mean than Reagent A. Does this information support your conclusions regarding acceptability? Explain your answer. HILS1746 Effective Date: 12/06/2016 Demonstrate your conclusions in the diagram below. Cut out and paste the Gaussian curves into the diagram with relationship to TEA limits. Illustrate the reagents’ TE using arrows. T Lower L Target Upper TE. Limit Value TEA Limit Reagent A Reagent B HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 9 How Far Can Your Mean Shift? PURPOSE: Benchmarking measurement procedures using TEA and the current method’s performance allows a laboratory to identify which methods are meeting quality specifications and which are not. In this activity, Sigma-metric and Critical Systematic Error (ΔSEc) are introduced. Participants will review a site’s monthly summary report and perform an investigational analysis using instrument records and QC charts. This activity supports the following laboratory management tasks and SLIPTA checklist items Management Tasks 1.7 Provide/coordinate new-hire orientation and training to staff 1.12 Develop and implement lab improvement plans based on best practices and feedback from staff, patients, customers, quality indicators, and external assessment 3.4 Enforce good stock management practices (proper storage, stock cycling, inspection of incoming orders, etc.) 5.2 Ensure proper preventive maintenance (i.e., cleaning, proper shutdown) on instruments when used 5.3 Perform and record troubleshooting on malfunctioning equipment 5.4 Review and sign maintenance logs to ensure regular preventive maintenance and timely repairs 5.5 Take corrective actions or issue repair orders and record all issues 5.6 Follow up on all corrective action – see if equipment is properly functioning, observe for trends or determine training needs 6.2 Ensure that QC material is tested according to SOP 6.3 Establish acceptable ranges for control material 6.4 Validate new equipment, reagents, and supplies 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.7 Review records of environmental checks & QC trends to assess impact on testing and take corrective action 6.9 Monitor reagent performance 6.13 Periodically observe/assess accuracy of staff performance and take corrective action 8.1 Monitor testing to ensure SOPs are followed and tests are performed and reported properly and promptly Checklist Items 1.5 Laboratory Policies and Standard Operating Procedures Are policies and/or standard operating procedures (SOPs) for laboratory functions, technical Laboratory Strengthening Checklist and managerial procedures current, available and approved by authorized personnel? 1.6 Policy and SOPs Accessibility Are policies and SOPs easily accessible/available to all staff and written in a language commonly understood by respective staff? 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? 3.8 Staff Competency Assessment and retraining Is there a system for competency assessment? 5.10 Equipment Preventive Maintenance Is routine user preventive maintenance performed on all equipment and recorded according to manufacturer’s minimum requirements? 5.12 Equipment Malfunction - Response and Documentation Is equipment malfunction resolved by the effectiveness of the corrective action program and the associated root cause analysis? HILS1746 Effective Date: 12/06/2016 8.7 Documentation of Examination Procedures Are examination procedures documented in a language commonly understood by all staff and available in appropriate locations? 8.8 Reagents Acceptance Testing Is each new reagent preparation, new lot number, new shipment of reagents or consumables verified before use and documented? 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)? 8.12 Are environmental conditions checked and reviewed accurately? Are the following environmental conditions checked and recorded daily? 10.1 Are all identified nonconforming activities/ work identified and documented adequately? 10.2 Root Cause Analysis Is documented root cause analysis performed for non-conforming work before corrective actions are implemented? 10.3 Is corrective action performed and documented for non-conforming work? 10.4 Are implemented corrective actions monitored and reviewed for their effectiveness before closure/clearance? 11.2 Quality Management System Improvement Measures Does the laboratory identify and undertake continual quality improvement projects? 11.3 Communication System on Laboratory Operations Does the laboratory communicate with upper management regularly regarding needs for continual improvement? 11.4 Are quality indicators (TAT, rejected specimens, stock-outs, etc.) selected and tracked? 11.5 Is the outcome of the review of quality indicators used to improve lab performance?  KEY MESSAGES Can they: SEc indicates how far the mean can shift before quality  Describe how SEc and Sigma  performance requirements are exceeded. indicate method performance Benchmarking methods using a Sigma scale allows a relative to quality performance  laboratory to use their resources wisely. goals?   Good recordkeeping is essential in resolving QC Calculate SEc and Sigma? problems.  Use information gleaned from records to troubleshoot a QC issue?  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 Worksheet 1: Calculating SEc and Sigma Directions: Use the following key numbers to complete the table below. Each site analyzes three controls (low, normal, and high) for each analytical run. Analyte: Platelet Mean True Value SD % TEA Sigma-metric Rule Selection Guide Count (cell * 109 /L) (target)  6 sigma - any QC will do (just don’t use 2 SD limits)  Lab A 150 150 3 13.4% 5 sigma - single-rules such as 1:3s or 1:2.5s  4 sigma - multirules  Lab B 145 150 3 13.4% < 4 sigma - multirules with look-back to previous runs, increase the number of controls analyzed  Lab C 150 150 5 13.4% 3 or less - look for better analytical methods Lab D 148 150 5 13.4% Lab E 145 150 10 13.4% Analyte: l Bias l TE in units TEA in units SEc Sigma QC Rules Based on the Platelet [(TEA - ǀBiasǀ)/SD] Sigma-metric Count | x -True Value| Abs Bias + (1.65*SD) (TEA %/100%)* Target Value [(TEA - ǀBiasǀ)/SD] – 1.65 OR ¯ (cell * SEc + 1.65 109 /L) Lab A Lab B Lab C Lab D Lab E HILS1746 Effective Date: 12/06/2016 AWEsome Numbers Inc. OptimiZed Q.C. Processes - Sigma Patient-Focused Laboratory Quality OptimiZed Q.C. Process Sigma Sigma OC QC Rule Sample Front Line Supervisor Chart Investigation From To Strategy Frequency Chart Review Review and Quality Improvement Action 6,01 999.0 1 1-3.55 Routine ix /Week 1x / 4 Weeks None 5.51 6.00 2 1-3.55 Routine 1x / Week 1x / 4 Weeks None 5.01 5.50 3 1-3.05 Routine 1x Week 1x /4 Weeks None 4.50 5.00 4 Routine 1x / Week 1x/ 4 Weeks None 4.01 4.50 5 1-2.05 Incr Freq 1 1x Week 1x 2Weeks 3.51 4.00 6 1-2.0s Incr Freq 2 2x / Week 1x Week If Margin for Error is 3.01 3.50 7 1-2.0s Incr Freq 3 2x / Week 1x /Week lower/worse than peers 2.51 3.00 1-2.0s Incr Freq 4 1x/Day 2x / Week Yes! Make it better. 2.00 2.50 x Stop! NO NOT report any results until acceptable quality is verified (Total Error < TEa) Investigate, take action and repeat prior patients as stated in Process # Q-111 This is ONE possible QC strategy plan. Approve or modify this before implementing! 30 Levi:Lev2 Combinations - & Tests in 6 Labs Process to select Q.C. Strategy 1. Define the true value for each sample as per lab policy/hierarchy Compare Sigma Lev 1 Lev 2 2. Set the TEa limit for each sample as per lab policy hierarchy 14 Lev 1 3.1 3. Gather recent mean & SD of all Q.C. Samples 12 Lev 2 6.9 4. Calculate Sigma for each sample 10 5. Select Q.C. Strategy for each sample based on Sigma Value B 6 Level 2 Why use separate strategies at each level? Change can occur at one level and not the other 2 Check the graph. Sigma of Lev 2 is seldom the same as Lev 1 No single strategy works for a test where Lev 1 Sigma is 2.1 and Lev 2 is 6.9 5 10 How can you implement separate strategies at each level? Level 1 It's not difficult. See details in FAQs at www.awesome-numbers.org_ (c) Zoe C. Brooks 1995 -2013 [email protected] www.awesome-numbers.org. HILS1746 Effective Date: 12/06/2016 SEc Monthly Review Log June May QC Target TE (in TEA (in 20XX 20XX Instrument Analyte Material Units Mean SD Value units) units) TEA% SEc SEc XYZ Analyzer ALT Level 1 U/L 27.8 1.3 27.6 2.3 7.3 26.3 3.8 3.9 Level 3 U/L 200.0 9.8 198.0 18.2 52.1 26.3 3.5 3.5 XYZ Analyzer AST Level 1 U/L 48.0 1.1 47.0 2.8 7.1 15.2 3.9 4.0 Level 3 U/L 274.0 7.0 271.0 14.6 41.2 15.2 3.8 4 XYZ Analyzer Bili-Total Level 1 umol/L 10.4 0.6 10.5 1.1 3.3 31.1 3.6 3.6 Level 3 umol/L 130.0 4.8 126.0 11.9 39.2 31.1 5.7 5.8 XYZ Analyzer Calcium Level 1 mmol/L 1.43 0.01 1.40 0.05 0.05 3.60 0.39 1.40 Level 3 mmol/L 3.21 0.04 3.19 0.09 0.11 3.60 0.72 2.20 XYZ Analyzer CK Level 1 U/L 800.0 5.0 81.0 727.3 24.5 30.3 -140.5 3.1 Level 3 U/L 635.0 25.0 638.0 44.3 193.3 30.3 6.0 6.0 XYZ Analyzer Creatinine Level 1 umol/L 44.4 0.7 44.5 1.3 4.0 8.9 3.9 3.9 Level 3 umol/L 565.0 8.0 559.0 19.2 49.8 8.9 3.8 3.9 XYZ Analyzer Glucose Level 1 mmol/L 3.44 0.06 3.39 0.15 0.23 6.90 1.42 1.23 Level 3 mmol/L 20.39 0.39 20.20 0.83 1.39 6.90 1.44 1.33 XYZ Analyzer Potassium Level 1 mmol/L 2.60 0.10 2.59 0.18 0.23 8.70 0.50 1.04 Level 3 mmol/L 7.50 0.31 7.60 0.61 0.66 8.70 0.16 1.04 XYZ Analyzer Sodium Level 1 mmol/L 116.0 0.5 116.0 0.8 1.5 1.3 1.4 2.4 Level 3 mmol/L 157.0 0.8 157.0 1.3 2.0 1.3 0.9 1.8 XYZ Analyzer TP Level 1 g/L 42.0 0.2 41.6 0.7 1.4 3.4 3.4 3.6 Level 3 g/L 71.8 0.5 71.8 0.8 2.4 3.4 3.2 3.3 HILS1746 Effective Date: 12/06/2016 Daily QC Investigation Worksheet* Analyte Under Investigation__________________ Instrument Date ___ /___/___ _________________ Controls Affected: Level 1 ________ Level 2 ___________ Level 3 __________ Other ____________ QC Flags: 1:2S 1:2.5S 1:3S 1:4S (Random or Systematic) 2:2S 4:1S 10x 3:1s (Systematic) R:4S (Random) Other: The change began: suddenly____ or gradually____ at run#____ on date ___ /___/___. Recent data are distributed: above____ below____ or evenly about____ the mean. The % of data prior to the change within ±1 SD is: approximately 68%____ < 68%____ > 68%____ A new lot number of reagent was started at run # ____ on /___/___ ___ A new bottle of reagent was started at run # ____ on /___/___ ___ A new lot number of calibrator was started at run #____on /___/___ ___ A new bottle of calibrator was started at run # ____ on /___/___ ___ A new box of controls was started at run # ____ on /___/___ ___ A new bottle of control was started at run # ____ on /___/___ ___ Instrument maintenance occurred at run # ____ on /___/___ ___ A change in the analytical process occurred at run # ____ on /___/___ ___ Other test(s) affected: ______________________________ The common Instrument Wavelength Reagent Calibrator denominator is: Dispenser Test Principle Supply Shipment Other: Other instrument(s) affected: ______________________________ The common Reagent Calibrator Parts/Maintenance/Update Process or people denominator is: Other Information: The QC chart mean is assigned: Correctly____ Too high____ Too Low____ The QC chart SD is assigned: Correctly____ Too high____ Too Low____ Probable error type is: Systematic____ Random____ Either Random or Systematic____ Patient results may be erroneously: High____ Low____ Imprecise____ The change in method accuracy____ or precision____ coincides with a change in: Reagent____ Calibration____ Instrumentation____ Controls____ Process____ Action: Repeat Controls____(and All patient samples____or Borderline patient samples____) with: Fresh Reagent New Lot of Fresh Calibrator New Lot of Fresh Controls Alternate Controls Reagent Calibrator Clean or maintain instrument____ Arrange instrument service from manufacturer____ Consult with supervisor, director, or technical specialist____ Temporarily discontinue reporting patient results_____ Refer patient samples to another laboratory____ Assess current mean and SD against target and TEa____ If change not corrected with action above and TETEA TE>TEA 50% 4 2 1:3s/2:2s/R:4s/4:1s/8x 35% 4 1 1:3s 2:2s/R:4s/4:1s x 25% 4 1 1:2.5s x 2 1 1:2.5s x x x x x 78% x 2 1 1:3s/2:2s/R:4s x x x x x 68% x 2 1 1:3s x x x 80% x 88% 2 1 1:3.5s Based on the information contained in the table, assist the QA Officer with developing a QC strategy for this workstation. Write your response. HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 11 How Proficient Are We? PURPOSE: External Quality Assessment Schemes (EQAS) enable the laboratory to assess on-going accuracy by comparing its method’s performance to external sources. In this activity, participants are introduced to standard deviation index (SDI) and z-score and will review a proficiency test (PT) report. This activity supports the following laboratory management tasks and SLIPTA checklist items Management Tasks 1.12 Develop and implement lab improvement plans based on best practices and feedback from staff, patients, customers, quality indicators, and external assessment 6.12 Enroll in EQA program, monitor results, and take corrective actions 10.3 Assure proper record retention, rotation to storage, and disposal according to protocol Checklist Items 1.10 Data Files Are test results, technical and quality records, invalid or discontinued policies and procedures archived for a specified time period in Laboratory Strengthening Checklist accordance with national/international guidelines? 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? 8.14. Does the laboratory participate in inter-laboratory comparison program or alternative assessment systems for all tests? 10.1 Are all identified nonconforming activities/ work identified and documented adequately? 10.2 Root Cause Analysis Is documented root cause analysis performed for non-conforming work before corrective actions are implemented? 10.3 Is corrective action performed and documented for non-conforming work? 10.4 Are implemented corrective actions monitored and reviewed for their effectiveness before closure/clearance? 10.5 Preventive Actions Are documented preventive actions implemented and monitored for their effectiveness?  KEY MESSAGES Can they: Proficiency testing (PT) programs provide a way to  Recognize the common  verify long-term accuracy of a method. elements found in a PT report  SDI is a quick indicator of how well our mean compares and their significance? to the method peer mean.  Calculate SDI and use that  All PT failures must be investigated and documented by information to assess the the laboratory. accuracy of the method?  Recognize that PT results, both acceptable and unacceptable, should be evaluated?  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 General Guidelines for Proficiency Testing Laboratory proficiency testing (PT) is an essential element of laboratory quality assurance. Proficiency testing is an independent and unbiased assessment that evaluates the laboratory’s ability to produce correct answers. Proficiency testing provides an assessment of the validity of testing in your laboratory. Handling Your PT Survey Pre-analytical Note the date of receipt for your shipment  Immediately inspect and reconcile the contents of your shipment with the accompanying  paperwork Are all required specimens available?  Is the quality and appearance of the specimens acceptable?  o Store the shipment properly o Note due date of results o Reconstitute specimens with volumetric pipettes and correct diluent o Mix samples well before analyzing Analytical Analyze specimens at correct temperature. If shipment was stored in the refrigerator,  specimens may need to come to room temperature before testing. Always refer to your survey instructions for storage and specimen handling.  Analyze PT specimens in the same fashion as patient specimens.  Do not refer any PT samples to another laboratory, even if your instrument is non-  functioning or is part of your testing algorithm. Rotate testing responsibility for PT specimens between all laboratory personnel that are  routinely performing the analysis in your laboratory. Perform PT analysis well before due date of results.  Post-Analytical Assure that your laboratory’s results are reported according to the PT provider’s  instructions. Ensure the proper method and instrument code are recorded for each test so that you are  part of the correct peer group. If test not performed is the correct answer because of equipment issues, then indicate this  on the form. If the result obtained requires additional testing per your laboratory’s algorithm, then  indicate on the form to be sent to a reference laboratory or further testing required, but do not actually send the PT sample to another laboratory. Review results for clerical errors on answer sheet, including decimal point placement.  Retain a copy of answer sheet for your records. Attach all raw data and the instrument  print-out to the answer sheet. If possible, retain specimens in freezer for confirmatory testing if needed.  If you use the PT sample materials to cross-check other instrument or methods, or as part of  your competency training program, then be absolutely sure the PT results are submitted to the PT provider before starting these activities. HILS1746 Effective Date: 12/06/2016 Receipt of Results Review your results with your peer grouping.  Investigate all unacceptable grades.  Have the Laboratory Director and Supervisor review, and sign and date results.  Review results with testing personnel. Retain a copy for competency assessment and place  into personnel record. Investigate any failed responses and complete an EQA Failure Checklist assessment.  Follow-up with remedial actions if indicated.  HILS1746 Effective Date: 12/06/2016 Proficiency Testing – a laboratory testing program in which samples from a common pool are periodically sent to members of a group of laboratories for analysis, following which each laboratory’s results are compared to those of other laboratories and/or to an assigned value and are subsequently reported to the participating laboratory. N – the number of participating laboratories’ data used in the calculation of the group’s SD and/or mean. Mean (targets values) are derived from all-participant mean values calculated by a robust statistical technique. In some cases, however, it is recognized that method-, reagent-, and/or instrument-specific targets may be required and that peer-group-specific targets are used where appropriate. Acceptable Range: Represents limits established using criteria specified by the PT provider, such as CLIA, allowing for rounding to appropriate significant digits. Results falling within this range are scored as acceptable. Any result exceeding these limits is considered unacceptable. SDI (Standard Deviation Index, or a.k.a. z-score) – a calculated value that indicates the number of SD units of each result from the group’s mean; describes the bias of a method in units of SD. SDI is expressed as either a positive or negative value, indicating whether your result is above or below the group’s mean. For PT testing, the SDI indicates the relationship between the result you obtained from the PT sample and the expected result determined by the PT provider. SDI = (your result – expected result)/group SD SDI Interpretation 0 Your laboratory’s mean value is the same as the group (no bias) ± 1.0 Acceptable performance when compared to your group ± 1.0 to 1.5 Problem may exist, laboratory should investigate problem with bias ± 2.0 or Your result falls among the laboratories with the poorest performance greater Troubleshoot bias problem and perform corrective action Guidelines for evaluating PT testing results when 5 samples are analyzed for each analyte 1. If 2 or more of the 5 SDI results exceed ± 1.0 SDI, then further investigation is warranted. 2. If the average of the 5 SDI results is more than ± 1.5, then significant bias (SE) is present and calibration data should be reviewed to determine if a shift has occurred. 3. If a single SDI is greater than ± 3.0, there is a high probability of random error (RE). 4. If the range of SDI values between the highest and lowest PT results exceeds ± 4 SDI, random error is a possibility, and the method should be evaluated for sources of imprecision. HILS1746 Effective Date: 12/06/2016 Directions: Calculate the SDI and Limits of Acceptability. Based on the acceptable range, determine your grade for the survey sample. The first two rows are completed for you. CLIA acceptable performance for free thyroxine is Target Value ± 3SD of the PT group. Test Evaluation and Comparative Method Statistics Unit of Measure Limits of Acceptability Peer Group Sample Your S.D.I Result Mean SD N Lower Upper Your Grade Free Thyroxine (ng/dl) CH-01 4. 5 4.64 0.26 45 (4.5-4.64)/0.26 = 4.64 – 3(0.26) 4.64 + 3(0.26) acceptable 0.5 =3.86 5.42 = - Siemens Advia Centaur, CH-02 2.4 2.66 0.19 59 (2.4 – 2.66)/0.19 2.66 – 3(0.19) 2.66 + 3(0.19) acceptable XP -1.4 2.09 =3.23 = = Siemens Advia Centaur CH-03 1.0 1.13 0.08 45 rgt CH-04 1.8 2.07 0.14 33 CH-05 2.2 2.37 0.12 45 The survey sample CH-01 is above / below the group’s mean (target value). Your result is _______ group SDs from the peer group’s mean. Circle one The survey sample CH-02 is above / below the group’s mean(target value). Your result is _______ group SDs from the peer group’s mean. Circle one The survey sample CH-03 is above / below the group’s mean (target value). Your result is _______ group SDs from the peer group’s mean. Circle one The survey sample CH-04 is above / below the group’s mean (target value). Your result is _______ group SDs from the peer group’s mean. Circle one The survey sample CH-05 is above / below the group’s mean (target value). Your result is ________ group SDs from the peer group’s mean. Circle one SDI enables your laboratory to assess your performance in the context of peer laboratories. Even though the performance is satisfactory for this analyte, your assessment of free thyroxine testing may indicate a problem with (_____ accuracy) (____ precision) for your laboratory. check all that apply HILS1746 Effective Date: 12/06/2016 Peer Mean Lower Limit Upper Limit CH-02 CH-01 - 3SD -2SD -1SD OSD +1SD +2SD +3SD Using the SDI calculated for each survey sample, draw a circle to indicate the location of your survey samples in relationship to the peer group’s mean. CH-01 and CH02 are done for you. HILS1746 Effective Date: 12/06/2016 Proficiency Testing Failure Checklist Survey Name: Clinical Specialty: Specimens: Date: Problem Description: Assessment Review PT Report Reviewed for Clerical Errors: Evaluation results match your copy of submitted results Yes No N/A Wrong Data Entered Yes No N/A Wrong Units Reported Yes No N/A Incorrect instrument or methodology indicated Yes No N/A Sample Handling: Unexpected delays in receiving survey Yes No N/A Kit contents correct and in acceptable condition Yes No N/A Testing performed within suggested instructional time guidelines Yes No N/A Specimens stored at correct temperature between receipt and analysis Yes No N/A Specimen analyzed at correct temperature Yes No N/A Sample mixed properly before testing Yes No N/A Sample diluted properly Yes No N/A Special Handling instructions were followed Yes No N/A Testing Procedure: Testing Personnel competent to perform analysis Yes No N/A Manufacturer's package insert available and followed Yes No N/A Testing procedure properly followed Yes No N/A Kit components replaced from other kits Yes No N/A Sample mix-up Yes No N/A Samples demonstrate a matrix effect Yes No N/A Instrument recently calibrated or due for calibration Yes No N/A Instrument maintenance up-to-date Yes No N/A New lot number of reagents or calibrators used Yes No N/A Reagents within expiration date Yes No N/A Results reported within linearity Yes No N/A QC within established range Yes No N/A QC demonstrates an even distribution around the mean Yes No N/A QC results show a shift, trend, or bias Yes No N/A Manufacturer consulted Yes No N/A Sample Results: A single sample fails on several analytes Yes No N/A All samples failed for the analyte Yes No N/A Previous survey results for the analyte demonstrate a problem emerging Yes No N/A PT material reassayed Yes No N/A Date of Repeat Testing Specimen Analyte Reported Result Repeated Result Intended Number Results /Peer Group Proficiency Testing Failure Checklist (continued) HILS1746 Effective Date: 12/06/2016 Survey Name: Clinical Specialty: Specimens: Date: Investigation: Conclusion: Corrective Action Taken: Name Laboratory Director Review HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 12 Understanding Inter-laboratory Comparison Programs PURPOSE: Laboratories using the same measurement procedure and analyzing the same control material will generate similar means and SDs. This information is a valuable source to determine the target value of the QC material for a laboratory and compare their method’s performance to their peers. In this activity, participants are introduced to coefficient of variation index (CVI). Based on the information supplied from the inter-laboratory comparison report, participants will investigate a QC problem. This activity supports the following laboratory management tasks and accreditation preparedness checklist items Management Tasks 1.12 Develop and implement lab improvement plans based on best practices and feedback from staff, patients, customers, quality indicators, and external assessment 6.12 Enroll in EQA program, monitor results, and take corrective actions Checklist Items 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? Laboratory Strengthening Checklist 8.14. Does the laboratory participate in inter-laboratory comparison program or alternative assessment systems for all tests? 10.1 Are all identified nonconforming activities/ work identified and documented adequately? 10.2 Root Cause Analysis Is documented root cause analysis performed for non-conforming work before corrective actions are implemented? 10.3 Is corrective action performed and documented for non-conforming work? 10.4 Are implemented corrective actions monitored and reviewed for their effectiveness before closure/clearance?  KEY MESSAGES Can they: Inter-laboratory peer comparisons can help identify  Calculate the SDI and CVI?  method and PT problems.  Verify the 4 key numbers used The peer mean is an excellent source of the true to monitor and evaluate  (target) value for each control. performance?  Perform an investigational  Coefficient of Variation Index is a peer-based metric of imprecision. analysis regarding a summary statistic flag?  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 Internal QC – a single laboratory analyzing controls and making immediate internal decisions on the acceptance or rejection of analytical runs. IQC provides a good estimate of the imprecision of the method by reviewing SD and %CV. External QC – a program that involves many laboratories analyzing the same samples (control materials or PT samples). This allows individual laboratories to compare their performance to that of a peer group. EQC is useful for estimating the bias or inaccuracy of a method. Comparison of EQA Programs Proficiency Testing (PT) Inter-laboratory Comparison Description Many laboratories analyzing PT samples from the same Many laboratories analyzing the same lot number of control pool submit their results for each sample to the PT provider material submit monthly summary statistics (Mean, SD, number of control measurements) to a central data base Data Submitted Results obtained from testing provided PT samples. Monthly summary statistics calculated from routine QC testing • Must be analyzed in the same manner as patient testing. as per your laboratory’s QC protocol schedule • Receive and analyze 1-5 samples per discipline. • Samples may require special handling and reconstitution. Frequency 2-4 events a year -refer to PT testing schedule monthly Values of Not known at time of testing Known at time of testing - the new control measurements Sample should show the same distribution as the past control measurements if the system is stable. Provider Accredited PT provider Manufacturers of QC materials Information Describes your performance relative to your peers. • Captures a snapshot of your test performance for a given • provided by the analyte at a specific point in time • Excellent source to determine Target Value for each service • Grades your performance control • Provides some idea about the amount of bias present in the analytical system. Cost Additional costs to participate in a PT provider’s testing Usually no additional cost to participate; service provided with scheme the purchase of QC materials. TAT for Report The time between receipt of PT samples for testing and Many manufacturers provide on-line services for up-to-date Receipt returned report for review may be a few months. information Pitfall If report TAT is extensive it may not allow laboratories to Manufacturer of QC materials used for the method does not detect and correct in a timely manner problems that affect provide this service quality Compliance Required by many regulatory and accrediting bodies • Optional, but highly recommended if service is offered • Participation may fulfill regulatory and accreditation requirements for the analyte in some countries. HILS1746 Effective Date: 12/06/2016 When starting your review, begin by asking yourself the following questions: What mean value should my laboratory obtain for this control? What SD or CV do most laboratories achieve with this method? SDI (Standard Deviation Index, a.k.a. z-score) – the calculated value that indicates the number of SD units of each result from the group’s mean; describes the bias of a method in units of SD. SDI is expressed as either a positive or negative value, indicating whether your result is above or below the group’s mean. For inter-laboratory comparison reports, the SDI is a statistic that indicates the relationship between your mean value and the results reported by your group. SDI = (lab mean – group mean)/group SD CVI (Coefficient of Variation Index, a.k.a CVR) – the comparison or ratio of the laboratory’s CV to the group’s CV. CVI = lab CV/group CV N – the number of participating laboratories identified in the report SDI CVI 0 Your laboratory’s mean value is < 1.0 Indicates for a specific control the the same as the group, no bias imprecision is less than the group’s average imprecision ± 1.0 Acceptable performance when 1.0 Indicates for a specific control the compared to your group imprecision is the same as the group’s average imprecision ± 1.0 to 1.5 Problem may exist, laboratory > 1.0 Indicates for a specific control the should investigate problem with imprecision is greater than the bias group’s average imprecision ± 2.0 or greater Troubleshoot bias problem and > 1.5 Troubleshoot imprecision problem perform corrective action and perform corrective action In addition to the SDI and CVI, inter-laboratory reports often include: 1. Your month’s mean, SD, CV, and number of measurements involved. 2. Cumulative or lot-to-date (LTD) laboratory mean, SD, CV, and the number of measurements involved. 3. This month’s group mean, SD, CV, and number of measurements involved. 4. Cumulative or lot-to-date (LTD) group mean, SD, CV, and the number of measurements involved. HILS1746 Effective Date: 12/06/2016 1. Calculate the CVI for each control. 2. Calculate the SDI for each control. Platelet Count (cells x 109 /L) Sysmex XE- Low Control Normal Control High Control 2100 Your Lab Mean 56 214 545 SD 6 17 37 CV 10.7 7.9 6.8 My Lab (Peer) CVI My Lab (Peer) SDI Peer Group Mean 62 215 496 SD 7 19 40 CV 11.3 8.8 8.1 N 12 12 12 3. Calculate the TE for each control. TE in units of cells x 109 /L 4. If the criterion for acceptable performance for platelet count is 25%, is the TE acceptable? TEA in units of cells x 109 /L TE < TEA HILS1746 Effective Date: 12/06/2016 Investigation of Summary QC Problems Worksheet* Analyte Under Investigation _________________ Instrument _________________ Date _________________ QC Flags: TE > TEa ________ SEc < 2.0 ___________ SDI > ± 2.0 __________ CVI > 1.0 ____________ Controls Affected: Level 1 ________ Level 2 ___________ Level 3 __________ Other ____________ Data & Calculations Relating to Four Key Numbers (in units) Level 1 Level 2 Level 3 Target TEa Mean Assigned to chart N Value N Value N Value Current actual Cumulative Peer Group Method All-group Method SD Assigned to chart N Value N Value N Value Current actual Cumulative Peer Group Method All-group Method TE on current data SEc on current data Information: Current data meet quality specifications: Level 1 ________ Level 2 ___________ Level 3 __________ The target values are valid: Level 1 ________ Level 2 ___________ Level 3 __________ The TEa limits are valid: Level 1 ________ Level 2 ___________ Level 3 __________ The current actual mean and SDI indicate a positive ________ or negative ________ bias compared to the target ________ cumulative ________ or peer _____mean, beginning this month ________ or gradually since ___ /___/___. The current actual SD and CVI indicate increased imprecision compared to the assigned to chart ________ cumulative ________ or peer ________ SD, , beginning this month ________ or gradually since ___ /___/___. Patient results may be erroneously High ________ Low ________ Imprecise ________ Action: Verify validity of all statistics used to generate within-laboratory and peer data ________ Calculate mean and SD on recent data population ________ Re-assign mean on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign SD on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign target values: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign TEa limits: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Select QC rules and strategy to maximize error detection ________ or minimize false rejection ________ ________ Initiate corrective action to reduce bias ________ Initiate corrective action to reduce imprecision ________ Arrange instrument service from manufacturer ________ Temporarily discontinue reporting patient results ________ Investigation performed by __________________________________ on ___/___/___ * Worksheet template modified from Brooks, Zoe (2001) Performance-Driven Quality Control, p181 HILS1746 Effective Date: 12/06/2016 Investigation of Summary QC Problems Worksheet* Analyte Under Investigation Alkaline Phosphatase Instrument Unicel DxC_ Date _05/01/2013___ QC Flags: TE > TEa ________ SEc < 2.0 ___________ SDI > ± 2.0____√______ CVI > 1.0 ____________ - Controls Affected: Level 1 ___√_____ Level 2 ______√_____ Level 3 __________ Other ____________ Data & Calculations Relating to Four Key Numbers (in units) Level 1 Level 2 Level 3 Target 107.3 TEa 12.9 Mean Assigned to chart 114 N Value N Value N Value Current actual 60 119.0 Cumulative 1195 106.7 Peer Group Method 4576 (91) 106.6 All-group Method 17443(429) 100.7 SD Assigned to chart 4.0 N Value N Value N Value Current actual 60 3.10 Cumulative 1195 5.78 Peer Group Method 4576 (91) 4.95 All-group Method 17443 (429) 9.68 TE on current data 16.8 SEc on current data -1.26 (Less than 0) Information: Current data meet quality specifications: Level 1 ________ Level 2 ___________ Level 3 __________ The target values are valid: Level 1 ________ Level 2 __________ Level 3 __________ The TEa limits are valid: Level 1 ________ Level 2 ___________ Level 3 __________ The current actual mean and SDI indicate a positive _______ or negative ______ bias compared to the target ______ cumulative ____ or peer ______ mean, beginning this month _____ or gradually since ____ / / . ____ ____ The current actual SD and CVI indicate increased imprecision compared to the assigned to chart __________ cumulative or peer ________ SD, , beginning this month ________ or gradually since ___ /___/___. __________ Patient results may be erroneously High ________ Low ________ Imprecise ________ Action: Verify validity of all statistics used to generate within-laboratory and peer data ________ Calculate mean and SD on recent data population ________ Re-assign mean on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign SD on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign target values: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign TEa limits: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Select QC rules and strategy to maximize error detection ________ or minimize false rejection ________ ________ Initiate corrective action to reduce bias ________ Initiate corrective action to reduce imprecision ________ Arrange instrument service from manufacturer ________ Temporarily discontinue reporting patient results ________ Investigation performed by __________________________________ on ___/___/___ * Worksheet template modified from Brooks, Zoe (2001) Performance-Driven Quality Control, p181 HILS1746 Effective Date: 12/06/2016 Investigation of Summary QC Problems Worksheet* Analyte Under Investigation Calcium (ISE Parameters) Instrument Unicel DxC_ Date _05/01/2013 QC Flags: TE > TEa ________ SEc < 2.0 ___________ SDI > ± 2.0 __________ CVI > 1.0 ____________ Controls Affected: Level 1 ________ Level 2 ___________ Level 3 __________ Other ____________ Data & Calculations Relating to Four Key Numbers (in units) Level 1 Level 2 Level 3 Target TEa Mean Assigned to chart N Value N Value N Value Current actual Cumulative Peer Group Method All-group Method SD Assigned to chart N Value N Value N Value Current actual Cumulative Peer Group Method All-group Method TE on current data SEc on current data Information: Current data meet quality specifications: Level 1 ________ Level 2 ___________ Level 3 __________ The target values are valid: Level 1 ________ Level 2 __________ Level 3 __________ The TEa limits are valid: Level 1 ________ Level 2 ___________ Level 3 __________ The current actual mean and SDI indicate a positive ________ or negative ________ bias compared to the target ________ cumulative __ or peer ______ mean, beginning this month ___ or gradually since ____ / / . ____ ____ The current actual SD and CVI indicate increased imprecision compared to the assigned to chart ________ cumulative ________ or peer ________ SD, , beginning this month ________ or gradually since ___ /___/___. Patient results may be erroneously High ________ Low ________ Imprecise ________ Action: Verify validity of all statistics used to generate within-laboratory and peer data ________ Calculate mean and SD on recent data population ________ Re-assign mean on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign SD on QC chart: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign target values: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Re-assign TEa limits: Level 1 ________ Level 2 ___________ Level 3 __________ ________ Select QC rules and strategy to maximize error detection ________ or minimize false rejection ________ ________ Initiate corrective action to reduce bias ________ Initiate corrective action to reduce imprecision ________ Arrange instrument service from manufacturer ________ Temporarily discontinue reporting patient results ________ Investigation performed by __________________________________ on ___/___/___ * Worksheet template modified from Brooks, Zoe (2001) Performance-Driven Quality Control, p181 HILS1746 Effective Date: 12/06/2016 1. What is the root cause of the problem? When did it begin? 2. What is your immediate action to be taken? 3. Why did the daily internal QC fail to notify when the method no longer met quality performances (when TE became > TEA) 4. What is your long-term strategy to reduce the chances or eliminate the problem from reoccurring? HILS1746 Effective Date: 12/06/2016 Report Information Significance • Valid estimate of your laboratory’s ability to obtain the same average (mean) value for a specific sample as the Comparison of your mean to peer group the peer group’s performance • Be aware that the SDI measured by the peer group is divided by a smaller SD value than that of the all-lab’s SD, thus producing a smaller SDI value. • Helpful when investigating discrepancies between your laboratory’s performance to your peer’s values • Helpful to compare when the number of laboratories Comparison of your mean to reporting for your specific method is small (N < 5) the all-lab group’s mean • Can investigate PT problems –you can determine if the method you are using has an overall significant bias that may be reflected in the PT samples and not by a problem specific to your laboratory Your laboratory’s mean • Are you reporting the right method and are included compares well to all-lab with the right peer group? group’s mean but not well to • Is the peer data valid (e.g. small number of labs which the peer’s mean includes 1 or 2 aberrant results)? • Excellent source for determining the usual SD for your method • By comparing each month’s SD to the usual SD, you can monitor unexpected increases in imprecision • Monitor shifts in SDI or CVI that occur over a period of time Historical Data Review • If your mean value is drifting and the mean values from peer and all-lab groups are drifting in the same direction, then the problem is probably due the control material itself. • If however, your mean is drifting while the mean values reported for the peer and all-lab groups remain constant, then the problem is within your laboratory • Quick comparison of method performance in the current Cumulative Data Review month relative to the mean and SD since started using the control CVI Flag Present • An alert to a statistical difference between your laboratory’s CV and the CV obtained by the peer group • An alert to a statistical difference between your SDI Flag Present laboratory’s mean and the mean value obtained by the peer group HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 13 Improvement Project Assignment PURPOSE: The knowledge gained from a workshop becomes effective when the improvement is applied at the site for better patient care. In this activity, participants will map the process needed to perform an improvement project (IP) assignment. This activity supports the following laboratory management tasks and SLIPTA checklist items Management Tasks 1.4 Assess personnel competency against standards and determine corrective action and training needs 1.5 Conduct weekly staff meetings to coordinate activities, review lab operations, reward success, celebrate accomplishments, and resolve issues 1.6 Meet with staff individually to communicate expectations, provide feedback, coaching, or on-the-job training to ensure competency and productivity 1.9 Create a work plan and budget based on personnel, test, facility, and equipment needs 1.12 Develop and implement lab improvement plans based on best practices and feedback from staff, patients, customers, quality indicators, and external assessment 1.13 Communicate to upper management regarding personnel, facility, and operational needs 5.6 Follow up on all corrective action – see if equipment is properly functioning, observe for trends or determine training needs 5.7 Communicate to upper management equipment specifications and maintenance needs 6.1 Ensure that the Quality Manual with quality assurance policies and procedures is accessible to and reviewed by all staff 6.2 Ensure that QC material is tested according to SOP 6.3 Establish acceptable ranges for control material 6.4 Validate new equipment, reagents, and supplies 6.5 Track test performance (e.g., Levy-Jennings chart) for trends 6.7 Review records of environmental checks & QC trends to assess impact on testing and take corrective action 6.8 Review occurrence log for patterns/trends and take corrective action 6.9 Monitor reagent performance 6.10 Customize site-specific SOPs as needed 6.11 Ensure that SOP are read and understood by staff 6.12 Enroll in EQA program, monitor results, and take corrective actions 6.13 Periodically observe/assess accuracy of staff performance and take corrective action Checklist Items 1.5 Laboratory Policies and Standard Operating Procedures Are policies and/or standard operating procedures (SOPs) for laboratory functions, technical Laboratory Strengthening Checklist and managerial procedures current, available and approved by authorized personnel? 1.6 Policy and SOPs Accessibility Are policies and SOPs easily accessible/available to all staff and written in a language commonly understood by respective staff? 2.1 Routine Review of Quality and Technical Records Does the laboratory routinely perform a documented review of all quality and technical records? 2.2 Management Review Does the laboratory management perform a review of the quality system at a management review meeting at least annually? 2.3 Are findings and actions from MR communicated to the relevant staff? HILS1746 Effective Date: 12/06/2016 3.3 Laboratory Director Is the laboratory directed by a person(s) with the competency, delegated responsibility to perform? 3.4 Quality Management System Oversight Is there a quality officer/manager with delegated responsibility to oversee compliance with the quality management system? 3.8 Staff Competency Assessment and Retraining Is there a system for competency assessment? 3.9 Staff meetings Are staff meetings held regularly? 4.1 Advice and Training by Qualified Staff Do staff members with appropriate professional qualifications provide clients with advice and/or training regarding required types of samples, choice of examinations, repeat frequency, and interpretation of results? 4.4 Communication Policy on Delays in Service Is timely, documented notification provided to customers when the laboratory experiences delays or interruptions in testing (due to equipment failure, stock outs, staff levels, etc.) or finds it necessary to change examination procedures and when testing resumes? 5.2 Are equipment operated by trained, competent and authorized personnel? 5.9 Equipment Calibration and Metrological traceability Protocol 5.10 Equipment Preventive Maintenance Is routine user preventive maintenance performed on all equipment and recorded according to manufacturer’s minimum requirements? 5.12 Equipment Malfunction - Response and Documentation Is equipment malfunction resolved by the effectiveness of the corrective action program and the associated root cause analysis? 5.13 Equipment Repair Monitoring and Documentation 7.1 Inventory and Budgeting System Is there a system for accurately forecasting needs for supplies and reagents? 7.5 Budgetary Projections Are budgetary projections based on personnel, test, facility and equipment needs, and quality assurance procedures and materials? 8.7 Documentation of Examination Procedures Are examination procedures documented in a language commonly understood by all staff and available in appropriate locations? 8.8 Reagents Acceptance Testing Is each new reagent preparation, new lot number, new shipment of reagents or consumables verified before use and documented? 8.9 Quality Control Is internal quality control performed, documented, and verified for all tests/procedures before releasing patient results? 8.10 Quality Control Data Are QC results monitored and reviewed (including biases and Levy-Jennings charts for quantitative tests)? 8.12 Are environmental conditions checked and reviewed accurately? Are the following environmental conditions checked and recorded daily? 8.14. Does the laboratory participate in inter-laboratory comparison program or alternative assessment systems for all tests? 10.1 Are all identified nonconforming activities/ work identified and documented adequately? 10.2 Root Cause Analysis Is documented root cause analysis performed for non-conforming work before corrective actions are implemented? 10.3 Is corrective action performed and documented for non-conforming work? 10.4 Are implemented corrective actions monitored and reviewed for their effectiveness before closure/clearance? 11.2 Quality Management System Improvement Measures Does the laboratory identify and undertake continual quality improvement projects? 11.3 Communication System on Laboratory Operations Does the laboratory communicate with upper management regularly regarding needs for continual improvement? 11.4 Are quality indicators (TAT, rejected specimens, stock-outs, etc.) selected and tracked? 11.5 Is the outcome of the review of quality indicators used to improve lab performance? HILS1746 Effective Date: 12/06/2016  KEY MESSAGES Can they:  The planning phase is essential in any improvement  Recognize the need for hospital project. management support and  The support of hospital management and analytical analytical staff adherence when staff is essential for implementing an effective QC planning a QC program?  program. Identify the steps needed during the planning phase to  The QC IP plan involves many sections of the accreditation checklist. successfully implement a QC program?  Further refine the process plan to identify what happens, who’s responsible, and by when?  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 Process MMp M ComplePe QC ProgrMm for Two JorksPMPions 7.5 Are budgePMry projecPions bMsed 1.5 Are policies MndCor sPMndMrd on personnel, PesP, fMciliPy Mnd operMPing procedures (SOPs) for 8.10 Are QC resulPs moniPored Mnd equipmenP needs, Mnd quMliPy lMborMPory funcPions, PechnicMl Mnd reQiewed (including biMses Mnd MssurMnce procedures Mnd mMnMgeriMl procedures currenP, IeQy-Jennings chMrPs for mMPeriMls? MQMilMble Mnd MpproQed by quMnPiPMPiQe PesPs)? MuPhorized personnel? INPUT PROCESS OUTPUT Knowledge 1. Apply appropriate QC rules to gained from alert when a medically significant QC change has occurred in the Workshop measurement procedure. 2. Use quality indicators (QIs) to Handouts monitor the analytical performance Worksheets (accuracy and precision) of the Job Aids workstations on a monthly basis. 3. 5ocument the corrective action taken when alerted by QI or QC rule. Utilize monthly SEc as a quality hospital management phase of testing indicator (QI) for the analytical Communicate IP with staff and SLMTA Training 1. 10. 11. 12. 13. 5. 2. 3. 6. 7. 9. 8. 4. 3.8 Hs Phere M sysPem for compePency MssessmenP PhMP coQers: M) compePency MssessmenPs performed Mccording Po defined criPeriM, b) new hires, c) exisPing sPMff, Mnd d) rePrMining Mnd re-MssessmenP where needed? 1 HILS1746 Effective Date: 12/06/2016 You have six months to implement a complete QC program for two quantitative analytical workstations at your site INPUTS PROCESS OUTPUTS Knowledge 1) Communicate IP assignment from QC workshop with staff and  gained from hospital management Apply appropriate QC  QC workshop 2) rules to alert when a medically significant Handouts,  job aids, and 3) change has occurred in the measurement worksheets procedure. from QC 4) Use quality indicators  workshop (QIs) to monitor the 5) analytical SLMTA performance  Training 6) (accuracy and precision) of the workstations on a 7) monthly basis. Document the  8) corrective action taken when alerted by 9) QI or QC rule. 10) 11) 12) 13) Utilize monthly SEc as a quality indicator (QI) for the analytical phase of testing HILS1746 Effective Date: 12/06/2016 Monthly SEc Log[i] Laboratory Department Date Signature Control Target TEa (in TE (in Instrument Analyte Sample Units Value # TEa% units) * Mean SD units) N SEc QC Rule [i] spreadsheet modified from Brooks, Zoe (2001). Performance-Driven Quality Control (Appendices B1 & B2) # Target based on * TEa limits based on 1.inter-laboratory peer comparison 1. biological variation - desirable 2. package insert 2. biological variation - minimum 3.historical/cumulative date 3. biological variation - optimal 4. other ______________________ 4. PT criteria from _________________ 5. other ______________________ HILS1746 Effective Date: 12/06/2016 Action Plan Workstation: ___________________________________________________________ Step Action Person Resources Needed Date Timeline # Responsible Completed Immediate Month Month 3- Long term 1-3 6 1 Schedule a staff meeting HILS1746 Effective Date: 12/06/2016 Date Due Step # 1 2 3 4 5 6 7 8 9 Activity Meet with Select Select QC Determine Select QC Create Chart Investigate Develop Lab Staff %TEA Materials SEc rules QC Review Poorly QC & Charts Performing Protocols Adminis- Methods trators Completion Date/s Person Responsible Signature Comments DATE/S Site Visit # 1 2 Mentor/s Site Visit Form Complete HILS1746 Effective Date: 12/06/2016 QC FOLLOW UP VISIT REPORT Facility Name: SLMTA Trained Person(s) Title Start Date: ___________ End Date: ______________ 1. Improvement Project: QC Program for 2 Workstations QC Workshop Visit Date of Visit Facilitator Name #1 QC Checklist Item Yes No Comment Checklist Points Is there documentation Please rate progress 0-5 (0= No meeting, 3 =Meeting with no that laboratory staff has documentation & 5 =Meeting with documentation) Score = been informed of the project? Is there documentation Please rate progress 0-5 (0= No meeting, 3 =Meeting with no that upper management documentation & 5 =Meeting with documentation) Score = has been informed of the project? Is there documentation Please rate progress 0-5 (0= No updates, 3 =Updates with no that project updates have documentation & 5 =Updates performed on a weekly basis) Score = been forwarded to the mentor? Does the site have a Please rate progress 0-5 (0=Cannot locate plan from workshop & written plan that indicates 5=Plan has been updated since returning to site) Score = they understand their project? Is the plan detailed and Please rate progress 0-5 (0=Cannot locate plan from workshop & explicit so that 5=Plan is implantable in its current state) Score = understanding is clear? Is the whole laboratory Please rate progress 0-5 (0=No involvement & 5= Tasks have been team involved in the assigned with some evidence of completion) Score = improvement project? Has the SEc baseline data Please rate progress 0-5 (0=No baseline data & 5 excellent data been collected & analysis present with analysis done well) Score = completed? Have positive Please rate improvement from 0-5 (0=Not implemented & 5 improvements been excellent improvements) Score = implemented, and if so, what are they? HILS1746 Effective Date: 12/06/2016 QC Checklist Item Yes No Comment Checklist Points Were improvements Please rate effectiveness from 0-5 (0=Not effective & 5 very effective? Describe. effective) Score = Total Score________ out of 45 possible points QC Workshop Visit Date of Visit Facilitator Name #2 QC Checklist Item Yes No Comment Checklist Points Is there documentation Please rate progress 0-5 (0= No meeting, 3 =Meeting with no that laboratory staff has documentation & 5 =Meeting with documentation) Score = received updates on the project? Is there documentation Please rate progress 0-5 (0= No meeting, 3 =Meeting with no that upper management documentation & 5 =Meeting with documentation) Score = has received updates on the project? Is there documentation Please rate progress 0-5 (0= No updates, 3 =Updates with no that project updates have documentation & 5 =Updates performed on a weekly basis for the first two months and periodically thereafter) Score = been forwarded to the mentor? Does the site have a Please rate progress 0-5 (0=Cannot locate plan from workshop & written plan that indicates 5=Plan has been updated since first site visit and recommendations have been incorporated into the plan) Score = they understand their project? Is the plan detailed and Please rate progress 0-5 (0=Cannot locate plan from workshop & explicit so that 5=Plan is implantable in its current state) Score = understanding is clear? HILS1746 Effective Date: 12/06/2016 QC Checklist Item Yes No Comment Checklist Points Is the whole laboratory Please rate progress 0-5 (0=No involvement & 5= Tasks have been team involved in the assigned with some evidence of completion since the last visit) Score = improvement project? Has the SEc baseline data Please rate progress 0-5 (0=No baseline data & 5 excellent data been collected & analysis present with analysis done well) Score = completed? Has laboratory Please rate improvement from 0-5 (0=Not implemented & 5 management designed and excellent improvements) Score = monitored staff adherence of the QC protocol for the workstations? Have positive Please rate improvement from 0-5 (0=Not implemented & 5 improvements been excellent improvements) Score = implemented, and if so, what are they? Were improvements Please rate effectiveness from 0-5 (0=Not effective & 5 very effective? Describe. effective) Score = Total Score________ out of 50 possible points Quality Indicators Are the following QI in place and being monitored Yes NO Comment SEc or Sigma EQA Sample Rejection rate Stock Outs Customer Complaints TAT HILS1746 Effective Date: 12/06/2016 What coaching was provided during this visit First Visit: Second Visit: Site visit Summary report First Visit: Second Visit: Follow up actions First Visit: Second Visit: HILS1746 Effective Date: 12/06/2016 Quality Improvement Project PlanXC30 PLAN Use all the resources available to you to try and understand the problem, propose solutions and develop an action plan. SECTION A- Identifying the problem I. State the apparent problem: _________________________________________________________________________________ _________________________________________________________________________________ II. Collect Baseline Data: What data will be collected? _______________________________________________________ Method - How will the data be collected? ____________________________________________ Who is responsible for collecting data? ______________________________________________ What are the tools/forms/checklists to be used? _______________________________________ Over what period of time will the data be collected? ____________________________________ When will the data be reviewed? ___________________________________________________ III. Analyze the baseline data: What is wrong? _________________________________________________________________ Where is it happening? __________________________________________________________ When is it happening? ___________________________________________________________ Who is involved? ________________________________________________________________ IV. Identify possible causes: ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ V. Propose possible solutions: ______________________________________________________________________________ ______________________________________________________________________________ HILS1746 Effective Date: 12/06/2016 SECTION B: Action Plan I. Identified problem: _______________________________________________________________ II. AIM Statement (overall goal of this project) __________________________________ ______________________________________________________________________________ III. Actions to be implemented (following brainstorming of possible solutions). Action item Responsible Timeline Signature Person IV. Select and Define ELEMENT TO BE MEASURED (to monitor effectiveness of implemented actions) V. Results of element measured at baseline ___________________________________ VI. Acceptable results (target for this measure) _________________________________ VII. Data Collection How will the data be collected? ____________________________________________________ Who is responsible for collecting data? ______________________________________________ What are the tools/forms/checklists to be used? _______________________________________ How often will the data be collected? ________________________________________________ How often will the data be reviewed? ________________________________________________ How often will the data be analyzed to monitor effectiveness of implemented actions? ______________________________________________________________________________ HILS1746 Effective Date: 12/06/2016 DO IMPLEMENT Action Plan Collect data on element to be measured (to be done throughout the implementation period; document problems and unexpected observations) Summary of data collected on element to be measured Date of Review Results Depending on the element measured, results may be presented in a different format than table above e.g. before and after pictures. Monitor how the plan is being executed. Action item Responsible Person Timeline Signature Action Plan review R 1 R 2 R 3 CHECK Was change effective? _____________________________________________________________ If yes, how easy or difficult was it to achieve results? _______________________________________ _________________________________________________________________________________ _________________________________________________________________________________ Unexpected Observations: _________________________________________________________________________________ _________________________________________________________________________________ ACT If successful develop and implement plans to standardize the process, communicate changes and train as necessary. If unsuccessful, use information collected during DO and CHECK for problem analysis (Repeat PDCA) PLAN-DO-CHECK-ACT (Next Cycle ) Plan & Implement Cycle II of Improvement Project: Proposed date to begin Cycle II of improvement project _____________________________________ Signature of Reviewer ______________________________ Date _________________________ Laboratory Director ________________________________ Date _________________________ HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 14 Introduction to Method Evaluation PURPOSE: Before a method can be placed into routine service, it must be evaluated to ensure that the measurement procedure meets defined criteria, such as sensitivity, specificity, precision, accuracy, and linearity. In this activity, participants will perform method verification experiments and assess data to determine if the method is acceptable or not. This activity supports the following laboratory management tasks and SLIPTA checklist items Management Tasks 6.4 Validate new equipment, reagents, and supplies 6.5 Track test performance (e.g., Levy-Jennings chart) for trends Checklist Items 1.5 Laboratory Policies and Standard Operating Procedures Are policies and/or standard operating procedures (SOPs) for laboratory functions, technical Laboratory Strengthening Checklist and managerial procedures current, available and approved by authorized personnel? 5.3 Equipment and Method Validation/Verification and Documentation Are all equipment and methods validated/verified on-site upon installation and before use and is documented evidence available?  KEY MESSAGES Can they:  Even though manufacturers test their methods  List the steps required to extensively, there are many factors in an individual introduce a new method? laboratory that can affect the actual precision and  Collect and analyze the accuracy of the method. experimental data needed to  Method evaluation is performed to reveal the amount of quantify the error present? error present in the new method.  Judge if the method meets pre-  Once the performance of the method has been judged defined quality requirements? to be acceptable from method evaluation studies, then  Verify a reference interval? statistical QC procedures need to be selected that can detect medically important errors.  ACTIVITY OBJECTIVES MET? HILS1746 Effective Date: 12/06/2016 Quantitative Validation Overview Validation of a quantitative system (for example Chemistry analyzer or Hematology analyzer) consists of an established set of required experiments. Each laboratory should first design a validation plan describing how they will satisfy each of these requirements. The validation plan must also detail the acceptability criteria for each element. After completing all of the validation experiments, results should be compiled and filed in an organized manner. All validation records should be retained for the life of the instrument. A validation summary should be prepared that contains a place for the Laboratory Director to sign, indicating the validation has been reviewed and approved. The following are the required components of validation: 1. Precision is reproducibility - the agreement of the measurements of replicate runs of the same sample. Replication experiments are performed to estimate the imprecision or random error of the analytical method. (See Precision Guidelines.) 2. Accuracy is the true value of a substance being measured. Verification of accuracy is the process of determining that the test system is producing true, valid results. (See Accuracy Guidelines.) 3. Linearity - A quantitative analytical method is said to be linear when measured results from a series of sample solutions are directly proportional to the concentration or activity in the test specimens. This means that a straight line can be used to characterize the relationship between measured results and the concentrations or activity levels of an analyte for a determined range of analyte values. (See Linearity Guidelines.) a. The Analytical Measurement Range (AMR) is the range of analyte values that a method can directly measure on the specimen without any dilution, concentration, or other pretreatment not part of the usual assay process. AMR validation is the process of confirming that the assay system will correctly recover the concentration or activity of the analyte over the AMR. The manufacturer defines the AMR – but it is the laboratory’s responsibility to verify it. b. The Clinical Reportable Range (CRR) is the range of analyte values that a method can report as a quantitative result, allowing for specimen dilution, concentration or other pretreatment used to extend the AMR. The laboratory must specify the maximum concentration or dilution that may be performed to obtain a reportable numeric result. 4. Sensitivity is the lowest concentration of an analyte that can be measured (Lower Limit of Detection). For an FDA approved, unmodified method, the manufacturer’s stated sensitivity will be used. The LoD will be verified for immunoassays, therapeutic drugs, drugs of abuse, cardiac markers, and tumor markers. 5. Specificity is the determination of the affect of interfering substances. For an FDA approved, unmodified method, the manufacturer’s stated specificity will be used. 6. Reference Interval is the range of test values expected for a designated population where 95% of the individuals are presumed to be healthy (or normal). (See Reference Range Guidelines.) 7. Summary and Approval. See Validation Summary Report template. HILS1746 Effective Date: 12/06/2016 1. Pre-purchase Assessment 2. Installation and Calibration 3. Familiarization Period 4. Method Validation Plan* 4.1. Define quality requirement in the form of an allowable total error 4.2. Select the appropriate types of experiments to reveal the expected types of errors 4.2.1. Validation vs. Verification 4.2.2. Types of Error 4.2.2.1. Random Error (RE) 4.2.2.2. Systematic Error (SE) 4.2.2.2.1. Constant 4.2.2.2.2. Proportional 4.2.3. Types of Experiments for Verification of Manufacturer’s Claims 4.2.3.1. Linearity Experiment 4.2.3.2. Precision Experiment for RE 4.2.3.3. Comparative of Method Experiment for SE 4.3. Collect necessary experimental data 4.4. Use statistical tools to estimate the size of the analytical error 4.5. Compare the observed errors with the defined allowable error 4.6. Judge the acceptability of the observed method performance 4.7. If acceptable, then perform reference range experiment 5. Introduction into Routine Service HILS1746 Effective Date: 12/06/2016 Chemistry Validation Plan Validation Plan for Roche Cobas c501 Chemistry Analyzer I. Overview 1. Precision 2. Accuracy 3. Linearity 4. Sensitivity 5. Specificity 6. Reference Range 7. Method Approval II. Plan: The validation will be conducted on the Roche Cobas c501 Chemistry Analyzer (serial number 12345) for the following analytes and methods: AST 1. Precision a. Precision is reproducibility - the agreement of the measurements of replicate runs of the same sample. It is the process of determining the range of random error. The precision is measured in terms of coefficient of variation (CV). b. Random Error will be evaluated by running between day (intermediate precision) and within day (repeatability) precision using normal (QC Multi 1) and abnormal (QC Multi 2) control samples. Between-day precision will be tested by running each sample once per day for 20 days. Within day precision will be tested by running each sample 20 times in one day. The mean, standard deviation (SD), and CV of the replicates will be calculated. c. Acceptability criteria: The % CV for each assay is expected to be equal to or less than the manufacturer’s performance specifications for precision. In the event that an assay does not perform as expected, the %CV will be compared to the allowable random error (33% of CLIA Total Allowable Error Limits for between day and 25% of CLIA Total Allowable Error Limits for within day). . Analyte Mfg Precision 33% of TEA 25% of TEA AST 1.3% 6.67% 5% 2. Accuracy/Correlation a. Accuracy is the true value of a substance being measured. Verification of accuracy is the process of determining that the test system is producing true, valid results. b. Accuracy will be determined by a minimum of 40 samples, tested in duplicate. These will primarily be patient samples, but may include commercial proficiency testing or control samples in order to provide material that covers the reportable range. The samples will be tested on a XYZ Chemistry Analyzer located at Cape Clinic Laboratory chemistry section. The samples will be tested in duplicate on both the test and comparative instruments and duplicates will be averaged. Ideally, testing will occur on both instruments within 2 hours. c. Acceptability criteria: Linear regression analysis will be used to determine if the methods are accurate within the specified TEa when the Correlation Coefficient (r) is >0.975. If the Correlation Coefficient is < 0.975, then more patient data must be collected. If the Correlation Coefficient remains < 0.975, then paired data calculations or another regression analysis technique will be used. 3. Linearity a. A quantitative analytical method is said to be LINEAR when measured results from a series of sample solutions are directly proportional to the concentration or activity of an analyte in the test specimens. This means that a straight line can be used to characterize the relationship between HILS1746 Effective Date: 12/06/2016 measured results and the concentrations or activity levels of an analyte for some stated range of analyte values. b. Linearity verification will be determined using a patient samples previously analyzed on the XYZ analyzer. A known high near or slightly above the upper measurement range and a known low near the limit of detection will be used. i. Samples will be run in triplicate. ii. The mean value for each point will be calculated. iii. The recovered mean values will be plotted versus the corresponding known theoretical X values. A best-fit straight line will be drawn to connect the points on the graph with greater emphasis on the first three points when drawing the best-fit line. iv. The plot will be visually inspected for a linear relationship. v. The predicted Y value will be subtracted from the associated recovered mean value. The absolute difference and % difference will be calculated. This difference is the systematic error due to non-linearity. vi. Systematic error will be compared to 50% of the total error. b. Acceptability criteria: i. Visual assessment of the best-fit line on the linearity plot must demonstrate a linear relationship. ii. The method is linear if the % difference (or absolute difference) between the predicted Y’ and the recovered mean is less than the allowable error for each specimen point. iii. The systematic error must be less than 50% of the total error. 4. Sensitivity is the lowest concentration of an analyte that can be measured (Lower Limit of Detection). For an FDA-approved, unmodified method, the manufacturer’s stated sensitivity will be used. 5. Specificity is the determination of the effect of interfering substances. For an FDA-approved, unmodified method the manufacturer’s stated specificity will be used. 6. Reference Ranges a. Reference ranges are measured sets of values determined to occur in a healthy, non- diseased population. The laboratory must verify that their choice of reference ranges is valid for their study population. To verify or transfer a published range, the lab must analyze specimens from 20 healthy, non-diseased individuals for each subgroup. If 2 or fewer results fall outside the published range, it is considered verified. If, however, more than 2 results fall outside the published range, a more extensive study must be conducted. b. The reference range studies have been verified for the following populations: i. Adult reference ranges– The population sample used will be from the Cape Clinic Employee’s Health Clinic. Forty individuals (twenty adult males and twenty adult females) screened using a patient questionnaire and identified to be in good health will be used for the reference interval study. Any individual answering in the affirmative to unexplained weight loss, fever, jaundice, recent operation, pregnancy, or medications will be excluded from the experiment. A red top tube will be collected and transported to the laboratory in accordance with Cape Clinic’s collection and handling procedures for immediate analysis. c. Acceptability criteria: i. Establishment: ranges will be determined using a non-parametric statistical method to determine the 95% reference limits. For most analytes the lower and upper reference limits are defined as the 2.5 and 97.5 percentiles, respectively. ii. Verification: ranges will be considered verified if 90% of values fall within the proposed range iii. Verification of pediatric ranges will be dependent on the ability to collect sufficient pediatric samples in each age category. Additional time may be required, or fewer samples may be acceptable. The Medical Director will give final approval of the acceptability of pediatric reference range verification. HILS1746 Effective Date: 12/06/2016 7. Method Approval The final decision regarding methodology validation and acceptance is made after a careful review of all the studies performed as part of the complete method validation process. The Laboratory Director shall make the ultimate decision on method validation. Method acceptance is based on the results from the above studies, plus an evaluation of the new method’s cost effectiveness, turn- around-time, laboratory staff training needs, and any other relevant operational considerations. 8. References a. NCCLS EP9-A2 Vol.22 No.19 (Method Comparison and Bias Estimation Using Patient Samples), Sep 2002. b. NCCLS C28-A Vol. 15 No. 4 (How to Define and Determine Reference Intervals in the Clinical Laboratory), Jun 2000. c. NCCLS C10-A2 Vol. 22 No. 29 (Preliminary Evaluation of Quantitative Clinical Laboratory Methods), Dec 2002. d. NCCLS C6-A Vol. 23 No. 16 (Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach), April 2003. e. NCCLS C5-A2 Vol. 24 No. 25 (Evaluation of Precision Performance of Quantitative Measurement Methods), Aug 2004. f. NCCLS EP21-A Vol. 23 No. 20 (Estimation of Total Analytical Error for Clinical Laboratory Methods), Apr 2003. g. CAP Commission on Laboratory Accreditation, Laboratory General Checklist, 2007. h. Fawzi, W.W, et al. Vitamins and Perinatal Outcomes among HIV-Negative Women in Tanzania. N Engl J Med 2007; 356: 1423-31. i. Roche Cobas package inserts for ALT, AST, ALB, 2006-08 and 2006-05. HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 14A Linearity Experiment PURPOSE: The linearity experiment determines the reportable range of the method. Linearity is important because it validates that a test continues to work properly throughout the entire reportable range. Test specimens with analyte concentrations spanning the entire manufacturer’s reportable range are used to verify the manufacturer’s linearity claims. In this activity, participants will perform and analyze a linearity experiment. HILS1746 Effective Date: 12/06/2016 Chemistry Linearity Guidelines Linearity studies are performed to determine the linear reportable range for an analyte. The linearity for each analyte is assessed by checking the performance of recovery throughout the manufacturer’s stated range of the testing system. This is done using a set of standards containing varying levels of an analyte in high enough and low enough concentrations so as to span the entire range of the test system. Therefore, the demonstration of the linear range requires a series of known concentrations or know relationships established by dilution. Linearities are performed whenever a new analyzer, analyte, or method is introduced into the laboratory, or when an analyzer is replaced. Linearities may also be performed for troubleshooting purposes when quality control is unacceptable and deviations from acceptable data cannot be explained, when major analyzer repair or replacement of components has taken place, or at intervals prescribed by the manufacturer in the instrument’s user manual. I. Linearity A. Sample Criteria 1. A minimum of 5 samples that cover the reportable range of the method. 2. When plotted, the values should ideally be equidistant from each other. 3. Quality control, commercial linearity standards, and calibrators (if a different lot number is used to calibrate the instrument) may be used. 4. Patient specimens may be used if a high value near the expected upper range can be found. 5. Sufficient volume of each sample must be available to analyze in triplicate and for possible troubleshooting. B. Preparation 1. If using purchased materials, refer to manufacturer’s instructions. 2. If using patient specimens, then perform the dilutions using the manufacturer’s recommendation of the diluent to use with out-of-range specimens. 3. Select a patient specimen near the detection limit and another patient specimen near or slightly above the expected upper limit of the working range. Ensure that both specimens meet storage and stability requirements as stated by the manufacturer. 4. Prepare 5 pools for testing as follows: a. Label the low specimen Pool 1 and the high specimen Pool 5. b. Prepare Pool 2 (75/25) with 3 parts Pool 1 + 1 part Pool 5. c. Prepare Pool 3 (50/50) with 2 parts Pool 1 + 2 parts Pool 5. d. Prepare Pool 4 (25/75) with 1 part Pool 1 + 3 parts Pool 5. 5. Care must be taken to mix each pool thoroughly, and to protect the pools from evaporation or other deterioration. C. Testing 1. A set of linearity standards will be tested in the same manner as patient samples. 2. Testing should be performed in triplicate and performed within a single run. If one value deviates greatly from the others due to random error, it may be removed from the data analysis and repeated. HILS1746 Effective Date: 12/06/2016 3. Data should be plotted immediately to identify and correct any outliers. 4. Save the instrument print-outs to be filed with the summary statistics D. Evaluation of data: 1. Record the raw values on the Linearity Experiment Worksheet 2. Calculate Mean (Y) for each data point 3. Determine the Assigned Value (X) for each data point: a. If standards have known values, then insert them into the Assigned Value (X) column. b. If using the known relationship between dilutions, then follow the manufacturer’s instructions. c. If using patient dilutions, i. Pool 3 will be used as a true value; therefore, the mean value (Y) obtained will be the assigned value (X). ii. The remaining pools will be calculated using the known relationship between dilutions as follows: i. Pool 1 = mean of Pool 3 x 0 = 0 (Pool 1 must be zero or near zero, or else the actual value must be taken into account) ii. Pool 2 = mean of Pool 3 x 0.5 iii. Pool 4 = mean of Pool 3 x 1.5 iv. Pool 5 = mean of Pool 3 x 2.0 4. Visual Assessment a. Plot the assigned values on the X-axis on graph paper b. Plot the mean of the measured values on the Y-axis. c. Manually draw a straight line through as many points as possible, making sure that the line adheres to the lower points. d. An alternative to creating a graph is to use the Linear-data Plotter located on the www.westgard.com website. e. Visually inspect the plot for a linear relationship. f. Note on the graph the observed linearity range. g. Date and sign your initials. h. File the graph in the linearity portion of your method validation binder. 5. Quantifying Errors a. Plot the X and Y data in a regression analysis program. Calculate the slope and the y-intercept using linear regression from the website http://tools.westgard.com/cgi- bin/westgard/comp_calc.cgi?header=http:/www.westgard.com/images/headtoo lkit.gif b. Record the slope (m) and intercept (b) on the Linearity Experiment Worksheet. c. Using slope and intercept, calculate a predicted Y (Y’) value for each X value using the equation Y’ = mX + b. HILS1746 Effective Date: 12/06/2016 d. Subtract each measured Y value from the associated predicted Y value (SE= Y – Y’). This difference represents the systematic error due to non-linearity. Record these values on the Linearity Experiment Worksheet under the ± Diff e. Calculate % Diff (%Diff = : [± Diff/Y’]* 100% E. Determining Acceptability 1. Calculate % Limit by dividing the CLIA %TEA by a factor of 2 2. Calculate ± Limit by either inserting 50% of the CLIA absolute value or by multiplying the %Limit by Y’, whichever is greater. 3. Compare that systematic error to 50% of the total allowable error. The systematic error must be less than 50% of the total allowable error. 4. Complete the Acceptability column on the Linear Experiment Worksheet 5. Determine the linear range from the experiment 6. Determine the analytical measuring range (AMR). For each laboratory, it may not exceed the manufacturer’s stated AMR. 7. Place all documentation in the appropriate method validation binder. II. References A. GCLP Workshop and Workbook18-20 May 2008, Verification of Performance Specifications, pages 1-33. B. Clinical and Laboratory Standards Institute (CLSI). Evaluation of the Linearity of Quantitative Measurement Approved Guideline-Second Edition, CLSI document EP6- A (ISBN 1-56238-498-8) Clinical and Laboratory Standards Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania 19098-1898 USA, 2005. C. Westgard, James O. (2003). Basic Method Validation (2nd ed.). Madison: Westgard QC, Inc. D. Westgard, James O. (2008). Basic Method Validation (3rd ed.). Madison: Westgard QC, Inc. HILS1746 Effective Date: 12/06/2016 Linearity Experiment Worksheet Method Being Evaluated: Cobas c501 Date: November 4, 2013 Analyte: AST Analyst: Lisa Johnson Units: U/L Total Allowable Error (TEa): Concentration: Percent: 20% Manufacturer's stated analytical measurement range: Low: 5 High: 700 Sample Assigned Result Result Result Mean Predicted Y ± % # ID Value (X) 1 2 3 (Y) (Y') ± Diff % Diff Limit Limit Acceptability 1 Pool 1 0 6 6 6 6 5.60 0.40 7.1 0.56 10% acceptable 2 Pool 2 177 180 180 180 180 180.83 -0.83 -0.46 18.08 10% acceptable 3 Pool 3 354 355 354 353 354 356.06 -2.06 -0.58 35.61 10% acceptable 4 Pool 4 529 528 527 5 Pool 5 704 706 702 Intercept 5.60 Linearity range is Slope 0.99 AMR (analytical measurement range) is HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 14B Precision Experiment PURPOSE: The precision experiment, also known as a replication experiment, is performed to estimate the imprecision or random error of a measurement procedure. Data collected from the within-run and between-run precision experiments is used to quantify the amount of random error present in the method. In this activity, participants will perform and analyze the within- run and between-run precision experiments. HILS1746 Effective Date: 12/06/2016 Chemistry Precision Guidelines PRECISION is reproducibility -- the agreement of the measurements of replicate runs of the same sample. Replication experiments are performed to estimate the imprecision or random error of a given analytical method. III. Short -Term (Within-run/Day) A. Sample: 1. Two levels (Low / High or Normal / Abnormal) 2. Patient or quality control 3. Select values near the medical decision point(s) of interest for the analyte B. Testing: 1. Ensure there is a sufficient reagent to perform all 20 tests. 2. Run each sample 20 times on the same run, if possible, or least within the same day. C. Acceptability criteria: 1. Calculate the coefficient of variation (CV) for each level using 20 data points. 2. Compare the calculated CV to the manufacturer’s stated precision claims found in the package insert. 3. If manufacturer’s precision cannot be met, it is acceptable to attain precision that is <25% of the CLIA Allowable Error. 4. If Short -Term precision is unacceptable, consult the instrument’s manufacturer for technical assistance. 5. If unable to resolve issues with short-term precision, the method validation process should be discontinued and a new method selected for potential implementation. IV. Long-Term (Between-run/Between Day) A. Sample: 1. Two levels (Low/High or Normal/Abnormal) 2. Patient or control serum. A lab may already have this data available from their daily QC runs. 3. Select values near the medical decision point(s) of interest for the analyte B. Testing: Run each sample 1 time per day for 20 days for a minimum of 20 total data points for each level of material used. C. Acceptability criteria: 1. Calculate the CV for each level using 20 data points 2. Compare to manufacturer’s stated precision claims found in the package insert. 3. If manufacturer’s precision cannot be met, it is acceptable to attain precision that is <33% of the CLIA Allowable Error 4. If Long-Term precision is unacceptable, consult the instrument’s manufacturer for technical assistance. HILS1746 Effective Date: 12/06/2016 V. References A. GCLP Workshop and Workbook18-20 May 2008, page 16 B. Clinical and Laboratory Standards Institute (CLSI). User Verification of Performance for Precision and Trueness: Approved Guideline-Second Edition. CLSI document EP15-A2 (ISBN 1-56238-574-7). Clinical and Laboratory Standards Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania 19098-1898 USA, 2005. C. NCCLS. (Currently CLSI) Evaluation of Precision Performance of Quantitative Measurement Methods; Approved Guideline—Second Edition. NCCLS document EP5-A2 (ISBN 1-56238-542-9). NCCLS, 940 West Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 USA, 2004. D. Westgard, James O. (2003). Basic Method Validation (2nd ed.). Madison: Westgard QC, Inc. E. Westgard, James O. (2008). Basic Method Validation (3rd ed.). Madison: Westgard QC, Inc. HILS1746 Effective Date: 12/06/2016 Precision Experiment Worksheet Short Term (Within-run) Method Cobas c501(Serial #12345) Manufacturer Roche Analyte AST Units of Measure U/L Sample Name/Description: Material 1 QC Multi 1; Lot# QC123; exp 30-Aug 20XX Material 2 QC Multi 2; ; Lot# QC223; exp 30-Aug 20XX Material 3 Analyst Lisa Johnson Date November 5, 2013 Experiment Results Preliminary Estimate of Precision, Short Term Run Material Material Material Material Material Material # 1 2 3 1 2 3 1 35 120 User 2 34 Mean 35.0 U/L 120.0 119 U/L 3 35 120 SD 0.8 U/L 1.5 U/L 4 35 121 %CV 5 35 120 6 36 Manufacturer’ Claims 121 s 7 35 120 Mean 36.6 U/L 128 U/L 8 34 118 SD 0.3 U/L 1.0 U/L 9 36 120 %CV 0.8% 0.4% 10 35 122 11 35 119 25% of CLIA Allowable error is __5%_ 12 35 Short term precision is ______________ . 120 acceptable / unacceptable 13 35 121 14 35 117 15 34 119 16 35 118 17 35 120 18 37 122 19 36 123 20 34 120 HILS1746 Effective Date: 12/06/2016 Precision Experiment Worksheet Long Term (Between-run) Method Cobas c501(Serial #12345) Manufacturer Roche Analyte AST Units of Measure U/L Sample Name/Description: Material 1 QC Multi 1; Lot# QC123; exp 30-Aug 20XX Material 2 QC Multi 2; ; Lot# QC223; exp 30- Aug 20XX Material 3 Experiment Results Run Date Time Analyst’s Material 1 Material 2 Material 3 # Initials 1 6-Nov-13 0700 LS 35 122 2 7-Nov-13 0700 LS 34 119 3 8-Nov-13 0800 AM 34 120 4 9-Nov-13 1100 LS 37 121 5 10-Nov-13 1030 TZ 35 116 6 11-Nov-13 1000 TZ 36 121 7 12-Nov-13 1700 TZ 33 120 8 13-Nov-13 1400 AM 33 118 9 14-Nov-13 0800 AM 36 118 10 15-Nov-13 1500 TZ 35 122 11 16-Nov-13 1300 LS 35 119 12 17-Nov-13 0700 LS 34 120 13 18-Nov-13 0730 TZ 37 121 14 19-Nov-13 1000 TZ 35 117 15 20-Nov-13 0900 LS 35 119 16 21-Nov-13 1600 AM 35 118 17 22-Nov-13 1600 AM 35 124 18 23-Nov-13 0900 AM 36 122 19 24-Nov-13 0730 LS 36 123 20 25-Nov-13 1000 TZ 34 120 Preliminary Estimate of Precision, Long Term User Mean 35.0 U/L 120 U/L SD 1.1 U/L 2.1 U/L %CV Manufacturer’s Claims Mean 36.7 U/L 130 U/L SD 0.5 U/L 1.0 U/L %CV 1.3 0.8% 33% of CLIA Allowable error is ___6.6%. Long term precision is ____________ . acceptable / unacceptable HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 14C Comparison of Methods Experiment PURPOSE: The comparison of methods experiment is performed to estimate the systematic error present in a new method. In this experiment, verification of accuracy can be accomplished by comparing split-sample results obtained from a method with clinically valid results. In this activity, participants will determine the systematic error present and calculate the total error for a test method. HILS1746 Effective Date: 12/06/2016 Chemistry Accuracy (Trueness) Guidelines Trueness is the closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value. The results of a trueness evaluation are expressed numerically as bias. Verification of trueness is the process of determining that the test system is producing accurate and valid results. Ideally, a reference measurement procedure should be used if available for the analyte being verified. If a reference measurement procedure or traceability through standard reference materials is not available, then the comparative method must be carefully selected. In the instances where the comparison method is not a reference method, then the trueness of the new method cannot be determined. The laboratory would then be measuring the difference between the methods and not the bias of the new method. Any difference between the test method and the comparative method must be carefully interpreted. VI. Determine your comparison or reference method. A. The comparison method must be previously validated. B. The comparison method must be currently performing successfully in EQA. C. The ideal comparison method is a similar instrument/method. D. Comparison to an in-house method is preferred if the in-house instrument meets the above criteria. E. Samples with known values, such as proficiency testing samples or commercial standards, may be used as the reference method. VII. Sample Criteria A. A minimum of 40 samples that cover the reportable range of the method and include points near the Medical Decision Points. B. Patient, quality control, and proficiency testing materials may be used. C. 50% of the selected samples must lie outside of the current reference range. VIII. Testing A. Run each sample in duplicate on each instrument a. Ideally, samples should be run within 2 hours of each other unless the analyte has a shorter stability. b. Analyze the replicates (duplicates) in different runs and in a different order. B. Retain the instrument print-outs. C. Duplicates should be averaged. D. Data should be plotted immediately to identify and correct any outliers by reviewing the Comparison Plot or Difference Plot located at the Westgard website under Paired Data Calculator. a. Re-analyze any discrepant results between the test and comparative methods to confirm that the differences are real and not mistakes in recording the values or mix-ups of specimens. b. If an outlier is identified, then investigate the reason and take corrective action. c. Document the findings. d. Remove the outlier from the data set. E. Record values on the Comparison of Methods Experiment Worksheet. HILS1746 Effective Date: 12/06/2016 IX. Time Period of Testing A. A minimum of 5 separate days must be used for testing. B. This experiment can be performed simultaneously with the long-term precision study. X. Evaluation of Data A. Enter the data into the Paired Data Calculator located at the Westgard.com website. B. Calculate the slope, Y-intercept, Sy/x, and r. C. Evaluate the data using one of the options below: If Then r ≤ 0.975 • Data does not extend over acceptable range. • More data must be evaluated over larger range. r >0.975 • Proceed with Linear Regression Analysis to evaluate acceptability. XI. Acceptability criteria for Linear Regression Analysis A. Visually inspect the comparison plot for linearity and outliers a. If an outlier is removed, then recalculate the regression statistics B. Visually inspect the difference plot for constant scatter throughout the AMR a. Visually scan for significant and dramatic differences at the upper and lower ends of the range XII. Determine Bias or Difference between the Methods A. Enter Medical Decision Points (Xc) a. A Medical Decision Point (MDP) is the concentration of the analyte at which a medical decision is triggered and/or laboratory established critical values. 2. Using the linear regression equation, calculate the predicted Y value (Y’) that corresponds to the concentration of Xc. 3. Determine the bias (difference) by subtracting Y’ from Xc 4. Calculate the % bias (% difference) as bias/Xc * 100. XIII. References A. GCLP Workshop and Workbook18-20 May 2008, Verification of Performance Specifications, pages 1-33. B. Clinical and Laboratory Standards Institute (CLSI). User Verification of Performance for Precision and Trueness: Approved Guideline-Second Edition. CLSI document EP15-A2 (ISBN 1-56238-574-7). Clinical and Laboratory Standards Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania 19098-1898 USA, 2005. C. Clinical and Laboratory Standards Institute (CLSI). Method Comparison and Bias Estimation Using Patient Samples: approved Guidelines- Second Edition. CLSI document EP9-A2 (ISBN 1-56238-472-4). Clinical and Laboratory Standards Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania 19098-1898 USA, 2005. HILS1746 Effective Date: 12/06/2016 D. Clinical and Laboratory Standards Institute (CLSI).Preliminary Evaluation of Quantitative Clinical Laboratory Measurement Procedure: Approved Guideline – Third Edition. CLSI document EP10-A3 (ISBN 1-56238-622-0). ). Clinical and Laboratory Standards Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania 19098- 1898 USA, 2005. E. Westgard, James O. (2003). Basic Method Validation (2nd ed.). Madison: Westgard QC, Inc. F. Westgard, James O. (2008). Basic Method Validation (3rd ed.). Madison: Westgard QC, Inc. HILS1746 Effective Date: 12/06/2016 Comparison of Methods Worksheet Analyte: AST Units of Measure U/L Test Method: Cobas c501 Chemistry Analyzer Comparative Method: XYZ Chemistry Analyzer Manufacturer Roche Manufacturer: XYZ Company Run Date Time Initials Specimen 1st 2nd Mean Time Initials 1st 2nd Mean # ID Replicat Replicate (Y) Replicate Replicate (X) e 1 6-Nov-13 0700 LS 06111323 19 18 18.5 0645 LS 16 16 16 2 6-Nov-13 0700 LS 06111343 22 22 22 0645 LS 20 20 20 3 6-Nov-13 0700 LS 06111325 37 33 35 0645 LS 31 33 32 4 6-Nov-13 0700 LS 06111382 10 10 10 0645 LS 9 8 8.5 5 7-Nov-13 0700 LS 07111353 7 7 7 0645 LS 7 7 7 6 7-Nov-13 0700 LS 07111356 85 87 86 0645 LS 82 84 83 7 7-Nov-13 0700 LS 07111317 150 146 148 0645 LS 148 140 144 8 7-Nov-13 0700 LS 07111323 184 184 184 0645 LS 177 183 180 9 8-Nov-13 1030 AM 08111323 386 374 380 1000 AM 387 393 390 10 8-Nov-13 1030 AM 08111323 40 40 40 1000 AM 38 37 37.5 11 8-Nov-13 1030 AM 08111323 10 10 10 1000 AM 10 10 10 12 8-Nov-13 1030 AM 08111323 312 308 310 1000 AM 300 308 304 13 9-Nov-13 0800 LS 09111323 16 16 16 0715 LS 15 14 14.5 14 9-Nov-13 0800 LS 09111323 33 35 34 0715 LS 31 33 32 15 9-Nov-13 0800 LS 09111323 154 150 152 0715 LS 146 150 148 16 9-Nov-13 0800 LS 09111323 568 572 570 0715 LS 570 590 580 17 12-Nov-13 1500 TZ 12111323 39 39 39 1420 TZ 38 38 38 18 12-Nov-13 1500 TZ 12111323 9 8 8.5 1420 TZ 8 8 8 19 12-Nov-13 1500 TZ 12111323 235 233 234 1420 TZ 229 231 230 20 12-Nov-13 1500 TZ 12111323 25 25 25 1420 TZ 23 23 23 21 13-Nov-13 1030 AM 13111323 43 45 44 1015 AM 41 41 41 22 13-Nov-13 1030 AM 13111323 608 612 610 1015 AM 610 630 620 23 13-Nov-13 1030 AM 13111323 376 364 370 1015 AM 377 383 380 24 13-Nov-13 1030 AM 13111323 16 17 16.5 1015 AM 15 14 14.5 25 14-Nov-13 1120 AM 14111323 51 53 52 1100 AM 48 52 50 26 14-Nov-13 1120 AM 14111323 664 676 670 1100 AM 670 690 680 27 14-Nov-13 1120 AM 14111323 11 11 11 1100 AM 10 10 10 28 14-Nov-13 1120 AM 14111323 29 29 29 1100 AM 28 28 28 29 15-Nov-13 1530 TZ 15111323 45 45 45 1505 TZ 43 43 43 30 15-Nov-13 1530 TZ 15111323 260 260 260 1505 TZ 248 262 255 HILS1746 Effective Date: 12/06/2016 Run Date Time Initials Specimen 1st 2nd Mean Time Initials 1st 2nd Mean # ID Replicat Replicate (Y) Replicate Replicate (X) e 31 15-Nov-13 1530 TZ 15111323 32 32 32 1505 TZ 30 29 29.5 32 15-Nov-13 1530 TZ 15111323 27 27 27 1505 TZ 25 24 24.5 33 18-Nov-13 1215 TZ 18111323 39 41 40 1145 TZ 40 40 40 34 18-Nov-13 1215 TZ 18111323 31 31 31 1145 TZ 31 31 31 35 18-Nov-13 1215 TZ 18111323 9 9 9 1145 TZ 9 8 8.5 36 18-Nov-13 1215 TZ 18111323 270 260 265 1145 TZ 250 270 260 37 19-Nov-13 1220 TZ 19111323 27 27 27 1200 TZ 26 25 25.5 38 19-Nov-13 1220 TZ 19111323 388 392 390 1200 TZ 390 410 400 39 19-Nov-13 1220 TZ 19111323 67 66 67.5 1200 TZ 65 65 65 40 19-Nov-13 1220 TZ 19111323 564 572 568 1200 TZ 570 580 575 Slope (m) Correlation Coefficient (r) Y-intercept (b) Ordinary linear regression can be used (r > 0.975) S (Sy/x) Linear Regression Equation (Y = mX +b) Xc Regression Equation Y’ SE or Bias at Xc % Bias at Xc (Y’ – Xc) (SE/Xc)*100 35 U/L 120 U/L HILS1746 Effective Date: 12/06/2016 Judging Acceptability Worksheet Directions: 1. Determine your Medical Decision Point (Xc). 2. Calculate the allowable total error at Xc. 3. Provide the estimate of random error at Xc from your long-term precision experiment. 4. Provide the estimate of systematic error (difference) at Xc from your comparison of methods experiment. 5. Calculate the total error and the sigma-metric. 6. Determine the Sigma performance for each Xc using the Sigma table located on the following page. 7. Evaluate acceptability at each Medical Decision Point using the pre-defined quality requirement for the new method. Quality Requirement The Cobas c501 Chemistry Analyzer may be judged acceptable for the AST analyte if one of the following conditions is met: 1. Manufacturer’s claims for linearity, precision and accuracy have been verified. 2. The total error calculation (bias + 3 sd or %bias + 3CV) for the test method is less than the CLIA total allowable error for each Medical Decision Point (Xc). 3. The Sigma-metric is > 3.0 for each Medical Decision Point (Xc). In Units of U/L Concentration AST TEA at Long- Bias or Total Error Sigma Metric Sigma Acceptability of Xc concentration term Difference (bias + 3sd) ([TEA – bias]/sd) Performance of Xc Precision with (See Sigma table (%TEA * Xc)/ (in sd) Comparative on following 100% Method page) 35 U/L 120 U/L HILS1746 Effective Date: 12/06/2016 In Percent Concentration AST %TEA at Long- %Bias or Total Error Sigma Metric Sigma Acceptability of Xc concentration term %Difference (%bias + 3 %CV) ([%TEA-%bias]/%CV) Performance of Xc Precision with (See Sigma table (in %CV) Comparative below) Method 35 U/L 120 U/L Sigma Performance Table If Then The Sigma metric less than • The method has unacceptable performance and does not meet your requirement for quality, 2.0 even when the method is working properly. • It is not acceptable for routine operation. The Sigma metric is • The method has marginal performance and provides the necessary quality when everything is between 2.0-3.0 working correctly. • This method will require: o 4-8 controls per run o well-trained operators o reduced rotation of personnel o more aggressive preventive maintenance o careful monitoring of patient test results o continual efforts to improve method performance If the Sigma metric is • The method has fair performance and meets your requirement for quality and can be managed between 3.0-4.0 in routine operation. • This method will require a multirule procedure with 4-6 control measurements per run. If the Sigma metric is • The method has good performance and is clearly acceptable and can be well-managed in between 4.0-6.0 routine operation with 2-4 control measurements per run, using standard Westgard QC rules. If the Sigma metric is >6.0 • The method has Six Sigma performance and can be managed using a single control rule with wide limits (i.e. 1:3s, 1:3.5s). HILS1746 Effective Date: 12/06/2016 ACTIVITY SUMMARY SHEET ACTIVITY 14D Reference Interval Experiment PURPOSE: Medical providers can reach clinically misleading interpretations if a reference interval is inappropriate for a test. In this experiment, participants learn how to transfer a reference interval to a new method. HILS1746 Effective Date: 12/06/2016 Chemistry Reference Interval Guidelines I. The REFERENCE INTERVAL (OR REFERENCE RANGE) is the range of test values expected for a designated population in which 95% of the individuals are presumed to be healthy (or normal). When transferring or verifying a reference interval, ensure the comparability of the test subject population. If there are substantial differences in the geographic locations or demographic variables of the two populations that are known to cause differences in the reference values, then a reference interval must be established. II. Transference of Reference Ranges with Verification A. Select reference range to be verified. 1. Current laboratory ranges 2. Manufacturer’s ranges 3. Published reference ranges 4. Locally established reference ranges B. Determine population to be used to verify reference range. 1. Qualify healthy volunteers --this can be done through a questionnaire or health assessment. (See Appendix 1 Sample Health Questionnaire.) 2. Obtain samples from 20 healthy participants for each range to be verified. 3. Test each sample immediately and evaluate. C. If Then ≥ 90% of samples • The reference range is are within the verified. reference range < 90% of samples • Re-evaluate the range are within the reference range being verified. • Re-evaluate the healthy volunteer qualifications. • Collect and evaluate 20 additional samples. ≥ 90% of the • The reference range is additional samples verified. are within the reference range < 90% of the • Proceed with step III additional samples below (Establishment of are within the reference range Reference Ranges) HILS1746 Effective Date: 12/06/2016 III. Establishment of Reference Ranges A. Determine population to be used to establish reference range. 1. Qualify healthy volunteers. This can be done through a questionnaire or health assessment. (See Appendix 1 Sample Health Questionnaire.) 2. Obtain samples from 120 healthy participants for each range to be verified. The 40 samples previously collected in step I above can be used as part of the 120 samples. 3. Test each sample immediately after collection and evaluate. It is not advisable to collect and test all samples on the same day. B. Evaluation of data 1. Plot the data in a histogram and visually evaluate the frequency distribution and outliers. 2. Eliminate outliers based on visual examination and clinical experience. 3. Use a non-parametric method to determine the reference range. a. Rank (order by size) the values from lowest to highest. Example: Female Calcium Results (mg/dL) (Data from samples 6 - 115 omitted for example purposes) Sample 1 8.8 Sample 116 10.1 Sample 2 8.9 Sample 117 10.1 Sample 3 8.9 Sample 118 10.2 Sample 4 8.10 Sample 119 10.3 Sample 5 8.11 Sample 120 10.4 b. Multiply the total number of samples +1 by 0.025 to determine the sample number that represents the low end of the range. Example: Total number of samples= 120. Low end = (120 + 1) x 0.025 = 3.025 = 3. Sample 3 is the low end: 8.9 mg/dL. c. Multiply the total number of samples +1 by 0.975 to determine the sample number that represents the high end of the range. Example: Total number of samples= 120. High end = (120 + 1) x 0.975 = 117.975 = 118. Sample 118 is the high end: 10.2 mg/dL d. Use these rank values to estimate the upper and lower reference limits. Example: Reference range is “Sample 3 to Sample 118” or 8.9 - 10.2 mg/dL HILS1746 Effective Date: 12/06/2016 IV. Transference of Reference Ranges without Verification The CLSI C28-A2 describes different ways for a laboratory to validate the “transference” of established reference intervals. Pediatric reference intervals often require this approach because of the difficulty in obtaining sufficient specimens to establish or verify reference intervals. If a laboratory wishes to transfer a reference interval established by another laboratory or publication, the acceptability should be assessed based on several factors: A. Similarity of geographics and demographics. B. Similarity of test methodology. C. Sound clinical judgment and consultation with local medical professionals. D. Approval by the laboratory medical director is required and must be documented. V. References A. GCLP Workshop and Workbook18-20 May 2008, Verification of Performance Specifications, pages 1-33. B. Clinical and Laboratory Standards Institute (CLSI).Defining, Establishing, and Verifying Reference Intervals in the Clinical Laboratory Proposed Guideline-Third Edition, March 2008. CLSI document C28-A3c (ISBN 1-56238-682-4). Clinical and Laboratory Standards Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania 19098-1898 USA, 2005. C. Westgard, James O. (2003). Basic Method Validation (2nd ed.). Madison: Westgard QC, Inc. D. Westgard, James O. (2008). Basic Method Validation (3rd ed.). Madison: Westgard QC, Inc. HILS1746 Effective Date: 12/06/2016 Annex 1: Sample Health Questionnaire General Health Questionnaire for Donors for Verification of Reference Range Study Date: Place of Session: ____________________________________ Sample Number: _____(also record below)_ Surname: First Name: Middle Name: Sex: Date of Birth: Contact Address: Telephone: I understand that my blood will be tested for INSERT NAME(S) OF TEST(S) Phlebotomist’s Name: _____________________________________ Signature of Donor: ____________________________________________ --------------------------------------------------------------------------------------------------------------------- Y N Have you had any unexplained fever in the past 3 months? Have you lost more than 10% body weight in the past 6 months? Have had jaundice in the past 12 months? Have you ever had a blood transfusion? Have you ever had an operation? Are you on any medication? Ladies, are you pregnant, lactating or menstruating? Is this potential donor in good health? Y / N Phlebotomist Name: ________________ Phlebotomist Signature: ____________________________ Sample Number: (from above) ______ HILS1746 Effective Date: 12/06/2016 Reference Interval Worksheet Method being evaluated: Cobas c501 Analyte: AST Transfer Measurement Procedure: AST performed on XYZ analyzer Units: U/L Proposed Reference Interval: Males ≤ 38 U/L Females ≤32 U/L Partition 1: Male Partition 2: Female Run Date Collect Analysis Initials Result Date Collect Analysis Initials Result # Time Time Time Time 1 27-Nov 13 0800 0900 AM 28 27-Nov 0930 1000 AM 12 13 2 27-Nov 13 0910 1000 AM 8 27-Nov 1030 1100 AM 22 13 3 27-Nov 13 0930 1100 AM 37 28-Nov 0900 1000 LS 6 13 4 27-Nov 13 1000 1100 AM 32 28-Nov 0930 1000 LS 18 13 5 27-Nov 13 1015 1100 AM 27 28-Nov 1100 1200 LS 25 13 6 27-Nov 13 1030 1100 AM 39 28-Nov 1115 1200 LS 15 13 7 27-Nov 13 1045 1100 AM 19 29-Nov 0930 1000 TZ 21 13 8 28-Nov 13 0900 1000 LS 11 29-Nov 1000 1100 TZ 19 13 9 28-Nov 13 0915 1000 LS 37 29-Nov 1015 1100 TZ 28 13 10 28-Nov 13 0930 1000 LS 27 29-Nov 1030 1100 TZ 12 13 11 28-Nov 13 1015 1200 LS 32 29-Nov 1045 1100 TZ 22 13 12 28-Nov 13 1030 1200 LS 7 29-Nov 1200 1400 TZ 20 13 13 28-Nov 13 1100 1200 LS 16 29-Nov 1230 1400 TZ 25 13 14 29-Nov 13 1300 1400 TZ 12 30-Nov 1300 1500 LS 16 13 15 29-Nov 13 1315 1400 TZ 37 30-Nov 1315 1500 LS 21 13 16 29-Nov 13 1330 1400 TZ 15 30-Nov 1330 1500 LS 23 13 17 30-Nov 13 1000 1200 LS 25 30-Nov 1400 1500 LS 28 13 18 30-Nov 13 1015 1200 LS 20 01-Dec 13 0800 0900 AM 23 19 30-Nov 13 1030 1200 LS 36 01-Dec 13 0910 1000 AM 7 20 30-Nov 13 1045 1200 LS 22 01-Dec 13 0930 1100 AM 23 Reference interval is verified / needs to be re-evaluated. Reference interval is verified / needs to be re-evaluated. (circle one) (circle one) Analysis of transference performed by ___________________________ Date: ______________ Analyte Adult Reference Ranges Reference Range Cited % Verified (Expected ≥90%) Males ≤ 38 U/L XYZ Manufacturer Range AST Females ≤32 U/L HILS1746 Effective Date: 12/06/2016 Method Validation/ Verification, TE < TEA To reveal the Linearity Studies TE present Determines Random Error Systematic Error reportable range Precision Studies Comparison of Method Studies SD, %CV Bias, % Bias or the Difference between Methods HILS1746 Effective Date: 12/06/2016 On the Road to Laboratory Improvement Taking a turn towards QC QC Workshop Participant’s Manual HILS1746 Effective Date: 12/06/2016