Guidance on the establishment, implementation and performance for proficiency testing programs for thrombosis and hemostasis

Introduction

This guidance document is designed for regulatory agencies, assessment organizations and program planners required or aiming to start a national, regional or local External Quality Assessment Scheme for testing of hemostatic parameters. These guidance documents describe the major principles and the general practical aspects of an EQAS program for tests of hemostasis. External Quality Assessment (EQA) is now required for Good Laboratory Practice within all laboratories as part of the quality assurance and quality improvement programs for diagnostic testing in general. In many countries, EQA is required for laboratory accreditation. EQA is one of the components of a total quality assurance program of all laboratories. Note: Supplemental file at the end of the document contains the content list, scope, objectives and terminologies of this document.

When establishing an EQAS program for coagulation testing, the regulatory agency should consider linking or collaborating with an existing all-encompassing EQAS program or at the minimum, a program with hematology testing to consider if combining will facilitate logistical efforts and decrease cost. Unfortunately, in many developing countries such programs do not exist; therefore, it is important to motivate laboratory directors and staff to participate by discussing in detail the EQAS program’s objectives and benefits to gain support for the program.

The availability of excellent reagents and instruments using time established test methods does not automatically guarantee accurate hemostasis laboratory results. Numerous steps are involved in the process from when a sample enters the laboratory to the reporting of a result to the clinical provider. At each step along the process, errors can and do occur. The laboratory must always strive for accuracy and error minimization. Regulatory agencies should ensure that sufficient protocols are available for an EQAS Organization to provide and support a program to monitor the testing process. As with all parts of the clinical laboratory processes, one aim would be to improve the quality of the process and testing for all laboratories. A well-structured and functioning EQAS program is a major and important step to achieving the highest quality testing performance possible for all laboratories within the regulatory authority.

The purposes and benefits of having a national or regional EQAS program are numerous with mutual benefits to both the regulatory agency and the laboratory participants.

• The program will generate valuable information on accuracy and comparability of different method types and brands as well as reagents and instruments being used.
• Provide regulatory officials with a better overview of laboratories testing parameters and performance of different test methodologies.
• EQAS will be able to monitor each laboratory’s performance over time.
• EQAS will be able to identify those laboratories that require additional training to improve performance.
• Participating laboratories will learn from their performance compared to other laboratories.
• Laboratories will be able to identify opportunities to improve their testing process.
• Providers and patients will be reassured that they can rely on the laboratory’s results as being accurate and reliable.
• The EQAS program can provide laboratories, providers and patients with information on updated methodologies, reagents and instrument status.
• EQA helps reinforce the importance and relevance of quality assurance within the laboratory.

Without regulatory control, it may be difficult to motivate and convince all laboratories to participate in an EQAS program. However, every effort should be made to convince each lab to participate. The EQAS Organization should:

• Advertise the scheme widely;
• Enlist support of professional societies;
• Explain the purpose of the EQA scheme;
• Stress educational benefits;
• Emphasize the purpose of the scheme as a tool to help the laboratory improve their accuracy;
• Offer advice and help for persistent problems;
• Consider offering training courses in quality assurance for laboratory staff, if resources permit.

If an organization is designated or decides to start and/or administer a hemostasis EQAS program, then that organization should aim to follow a set of standards and guidelines that have been developed and used by thrombosis and hemostasis EQAS programs as well as general clinical EQAS programs (1-4). There are also several important guidelines for the clinical laboratory EQAS program. These include:

• ISO 9001—standards focusing on Quality management and Quality Assurance certification; specifies requirements for a quality management system (QMS).
• ISO/IEC 17025:2005—standards for general requirements for calibration and testing laboratories including the laboratory process for quality accreditation.
• ISO 15189 Medical Laboratories—Requirements for quality and competence—standards specifically for accreditation of medical laboratories; many hospital laboratories seek accreditation to this standard; one component is requirement to perform EQA with an ISO17043 accredited program.
• ISO/IEC 17043:2010—Conformity assessment—General requirements for proficiency testing. This standard details the requirements of proficiency testing providers to ensure the quality and functioning of the EQA program.
• ISO/IEC 13528:2015 (correction 2016)—statistical methods for use in proficiency testing by inter-laboratory comparison—includes recommendations on how to determine “target values” for EQA samples and methods for assessment of performance.

Aim of the hemostasis testing EQAS program

The main aims of a coagulation testing EQAS program are:

• To evaluate and confirm the quality of test results in a laboratory;
• To allow comparison of testing among laboratories and methods;
• To increase the awareness of quality in the participating laboratories;
• To allow participants to identify and eliminate problems with their testing process;
• To educate and identify improvement opportunities;
• To maintain control of laboratories with poor testing quality (if applicable).

It is also helpful if the EQA organization is aware of any goals and requirements of any laboratory accreditation or regulatory agency with respect to proficiency testing, to enable these requirements to be met.

Quality aspects of EQAS program

It is extremely important that the EQAS program be of the highest quality. The EQAS program should aim to comply with standards in ISO17043, and careful attention to sample integrity, and robust, appropriate statistical analysis is required. It is important for both the EQA program and the participating laboratories to have confidence that any proficiency testing failure is due to a laboratory-testing problem rather than problems with the EQA samples or analysis.

The following are the aspect of quality that must be present in an EQAS program and the administering organization:

• The EQAS program must be well planned;
• The entire process must be well documented;
• The Quality Management program of the EQAS must strive to continually improve;
• Each step of the process of the EQAS program must be traceable;
• The EQAS program must provide very good quality samples in stable condition.

Authority of EQAS program

EQAS programs usually encompass one of three types of testing.

• Educational EQAS;
• Non-sanctioned EQAS;
• Sanctioned EQAS or proficiency testing.

Educational EQAS is a scheme that emphasizes quality improvement. The EQA provider will co-ordinate preparation, distribution, data collection and analysis, but will not apply performance assessment criteria to the data. It is up to the participating laboratory to review and compare their data with that of the group as a whole. If the laboratory consistently fails or does not meet expectations, it is the responsibility of the laboratory to identify the cause and rectify the problem. This type of scheme is usually associated with long-term evaluation, continual follow-up and ultimately increased quality improvement.

Non-sanctioned EQAS programs can only provide guidance on correcting failures but usually have no consequences for failure. The scheme is usually scored but there are no regulatory consequences for the laboratory if the data is outside the acceptable range. Most EQAS programs fall into this category.

Sanctioned EQAS is a true “proficiency testing” scheme that is scored with consequences for failure. The EQAS program is accredited either directly or indirectly with a governing body. Depending on the set up in the individual country, the EQAS program can be a direct part of the government or national/regional health organization, or it can be an independent organization that is contracted or sanctioned by the government to provide the proficiency testing. The standards for “failure” are established by the ultimate regulatory agency. Failure can mean immediate dis-accreditation or some set of failures that constitute repeated evidence of inability to perform the testing. Usually these EQAS programs are mandatory and must be subscribed to for laboratories to be able to report results. Immediate or longitudinal failure can lead to loss of accreditation.

The EQAS program type is dependent on the government responsible for laboratory testing. Usually government laws determine the type of EQAS program; however, they may also be based in a private or professional organization with consequences of failure being the loss of accredited status with the EQA organization.

EQAS process

The EQAS Organization should have the necessary expertise in hemostasis testing and be capable of providing training and information on new developments in the area of coagulation and hemostasis testing. The organization must be knowledgeable with the complete range of diagnostic methods and assays used in clinical hemostasis testing.

Program plan

The major steps in the overall EQAS process are summarized below. Figure 1 provides a graphic representation of the development of a program. ISO17043 details the steps involved in the EQA program process, and requires that the process be planned in advance of commencing survey distributions.

• Setting up the EQAS program—this portion is for the initial establishment as well as the start of the cyclic (yearly) process.
• Registration of participating centers—contact details, tests performed and method details.
• Preparation of samples—selection of plasma, testing, aliquoting and packaging.
• Specimen distribution—transport and laboratory preparation of specimens.
• Data collection—data from each laboratory must be collected.
• Initial analysis of data—evaluation of data to ensure that the sample integrity was of good quality.
• Data analysis—the results are analyzed and decisions made as to acceptability or failure of results.
• Final report—report is generated and provided to the participant with accompanying sanctions (as necessary).
• Follow up—support, education, sanctions if applicable.

Figure 1 Flowchart outlining the steps required with important considerations.

There are numerous processes for each of these major headings. The EQAS program organization must be able to address and document each step to justify an excellent quality program. For every step of the process for the EQAS program, the organization staff must address and document the main questions (who, what, where, when, and how).

Internal structure of the EQAS organization

If the EQAS program is established by a regulatory agency or governing body, the agency must decide who the EQAS provider will be and what type of program will constitute the hemostasis proficiency testing program. The mission and purpose of the EQAS program must be succinctly detailed by the regulatory agency. It must be decided by the regulatory agency to determine if the EQAS program will be local, regional, national or international.

During these planning stages, the regulatory agency must determine the funding source. A quality EQAS program must have adequate funds for the staff, samples procurement and processing, work area, laboratory testing facilities, computer and software capabilities and either printing or web access for report submission. Without adequate monetary support, the quality of the EQAS program could be seriously compromised. Funding for the program may come from a central (government or agency) source, from participation fees, or from sponsorship. In short, there is a need for a business plan. If the program is sponsored, the relationship of the sponsor to the program should be clarified—for example, whether an unrestricted educational grant is provided.

Personnel

The EQAS provider must be able to supply the appropriate number of staff with the expert qualifications to ensure that the EQAS program will be able to provide the highest quality and meet the standards of the regulatory agency, the expectations of the international community and the expectations of the participating laboratories. The following are the description of the duties of the EQAS program provider. Some of these duties may be assumed by a single individual, a number of individuals on a part time basis or multiple duties performed by a single individual. The individuals must have appropriate education, training, technical knowledge and experience for their respective duties. Collaborators or contractors may provide some of the required work, for example preparation of the sample vials but these services must be vetted and have the same high quality and competence that the EQAS program employees have. Any testing of the EQA samples to determine values, homogeneity or stability should be performed by a laboratory with a quality management program that demonstrates the highest quality, good laboratory practice and a continued quality improvement program. Ideally, such a laboratory will be accredited to the appropriate standards.

Minimal staff duties

Program Director—serves as liaison with the regulatory agency, participating laboratories and individual setting the goals.

Chief Scientific Officer—individual responsible for acquiring, testing and making the samples for EQAS program.

Scientists/technologists—individuals performing the work to acquire test and prepare the samples. Competence and expertise in sample preparation and testing is required to ensure that the sample is of the highest quality.

Data coordinator—individual responsible for the integrity of the data submitted by the participating laboratories. This individual must be familiar with data entry, storage and manipulation. This individual should also be very knowledgeable in computer systems and the internet, and may perform or oversee manual input of data.

Clerical—individuals to produce and distribute the final report to participating laboratories. This individual may also be the point of contact for the participating laboratories, and be responsible for maintenance of the participant contact system.

Statistician support—the individual responsible for statistical manipulation of the data and performance analysis. Not all programs will employ a statistician, so statistical advice from an appropriate expert may be subcontracted. The statistician should be familiar with the principles of ISO13528.

Collaborators—small EQAS organizations may need collaborators for a number of aspects of the EQAS program. These collaborators must be of the highest quality possible. If they are not, then this becomes the weakest aspect of an otherwise high quality program. The EQAS organization must be able to document (in writing) collaborators ability to perform the duties with high quality. They must also be able to document all duties performed for the EQAS program. During the selection process of a collaborator, these processes must include criteria such as an established quality management program, demonstration of quality improvement program, staff training programs accreditation (when possible). It is also important to document in the agreement for the collaborators involvement the expectations, terms and quality improvement expected.

Staff involved in the EQAS program should be fully aware of their tasks and responsibilities—careful planning of the program and job descriptions will help in this respect.

Equipment and space

The EQAS organization must have the laboratory and office space to perform the necessary functions to carry out the EQAS program. This is in addition to the ability to produce the samples for the proficiency testing program. The EQAS Organization must be able to analyze the samples to be sent to the participating laboratories, though this function may be subcontracted. The space and facilities required for the functions of the EQA program should not be underestimated.

Computers and software should be available for managing the EQAS program. The computer capabilities must include ability to generate the necessary documents for the program, maintaining the characteristics and demographics of the laboratories, analyzing the results and generating final reports. Specifically the computer system will need to:

• Maintain the demographics of the participating laboratories;
• Recording characteristics of the proficiency testing samples;
• Recording distribution of proficiency samples and final reports;
• Analyzing data for the program;
• Generation of final reports;
• Mailing list and delivery service addresses.

Participant registration

Participants should be enrolled in the program having been given clear details of the aim, structure and format of the program. Sufficient contact information should be collected to ensure successful delivery of sample packages, and for distribution of information and reports.

Confidentiality is important, and data protection of participant details should be ensured. The identity of the participating laboratories should be known only to the EQAS organization. Each participating laboratory will be given a unique code when the participating laboratory starts the EQAS program. These codes should be the only way that group data can be viewed when studying general issues with sample results.

Individual performance of a participating laboratory should only be discussed with the laboratory concerned, as it is essential to maintain the trust of the participating laboratory.

Decision on assays and analytes to provide

The range of tests/analytes to be performed will be determined by:

• The test repertoire of the participating laboratories
• Availability of suitable material/plasma
• The purpose/aim/focus of the program, e.g., diagnosis of bleeding disorders, anticoagulation control.

Although analytes may vary from region to region, the most common assays and analytes performed in the routine coagulation laboratory are screening tests [prothrombin time (PT), activated partial thromboplastin time (APTT), Fibrinogen], anticoagulant monitoring [International Normalized Ratio (INR)], and Venous Thromboembolism (VTE) investigation (D-Dimer). Larger or more specialized laboratories are likely to perform additional screening tests, factor assays, thrombophilia screening tests and monitoring of non-Coumadin anticoagulants [Direct Oral Anticoagulants (DOAC), heparins].

The EQAS organization can decide how to package the assay and analyte samples and develop the reporting form. This may be guided or determined by the regulatory agency, who may decide which assays and analytes are to be sanctioned versus those that must be performed for “educational” EQA (not graded or graded but not sanctioned).

Since most of the routine assays and analytes are used for multiple purposes, an attempt needs to be made to test all uses of the assays or analytes. As an example, since the APTT is used to monitor unfractionated heparin, assess for Lupus Anticoagulant (LA) and detect factor deficiencies, the EQA samples should assess for these differing aspects of that assay throughout the evaluation cycle.

The number of challenges per distribution is dependent on the need to evaluate multiple functions. This may include 2–5 samples per distribution depending on the importance and variability of the assay. There should be a roughly equal distribution between the number of normal and abnormal samples covering the range of results that may be encountered in clinical practice during the whole yearly cycle. These do not have to be equally distributed with each set of samples. In most cases of hemostasis analysis, only abnormalities in the upper abnormal range are important while in other analytes the lower portions of the abnormal range are of analytical value. This depends on the assay or analyte being determined.

The number of normal and abnormal samples should be varied with no set pattern distribution. In addition, it can be useful to include if possible a duplicate sample in subsequent mailings to evaluate improvement of testing and precision of a laboratory.

Process to determine EQAS program and analytes

Once the regulatory agency has deemed the need for an EQAS program and an organization has been selected for the EQAS program, the EQAS Organization must plan and implement (with documentation), the goals and details of the EQAS program. These goals and missions must be communicated to all potential laboratories that may participate. This information should be summarized in a letter and/or other documents so that the participating laboratories can understand the expectation of the EQAS program, and hence the regulatory agency. The instructions and communications must be provided to the participants so that they can easily and accurately understand all aspects of the EQAS program. This “beginning of the cycle” notification process informs all participating laboratories that the program is ready, when to expect samples, and when and how results are reported.

The following information should be sent to the participating laboratories at the start of the program and at the beginning of each new cycle (usually at the beginning of the year).

• Information so that the participants understand the scheme that will be used. It should describe the aims, objectives and goals. This includes the expectations of the regulatory agency. It should also detail the process and materials to be sent.
• The time scale of when the materials will be sent out and the result return date. These should be set up at the beginning of the cycle so all participants know the deadlines.
• The methods that each participant must use to reconstitute the samples before for analysis must be included in detail. Poorly described instructions will increase the likelihood of erroneous results.
• Determine and communicate the statistical analysis that will be used to evaluate the results for each of the analytes.
• Determine the report format that the participants will receive.
• Communicate how performance will be evaluated, the transmittance of poor performance to the regulatory agency, and to what extent the data will be made public (as may be regulated by law). It is important that all laboratories be identified only by a code and not by name. It should be the regulatory agency’s responsibility to publicize this be required.

Initial (annual) process of the EQA scheme

At the start of the EQAS program cycle, the EQAS Organization must provide adequate instructions to the participants providing the necessary information in an easy to understand, clear and correct manner. This is to eliminate confusion of which assay and/or analytes to determine and the process to manage the analyte data. This should be accomplished with a letter (with additional information) clearly stating:

• What analytes the laboratory is testing;
• When the samples will be distributed;
• The type of samples to be sent (frozen, lyophilized);
• How to store the samples;
• What testing procedures should be followed;
• What equipment, reagents and methods are used;
• How long the participants have to complete and return the results;
• How the results should be returned (website, fax, mail).

Number of analytes per distribution

The number of analyte samples per distribution set can help a laboratory to distinguish between random errors and systematic error. There is an important difference between a one-sample and a two-sample per distribution approach. The one-sample approach assesses the accuracy of the single evaluation that is performed. However, using a two-sample per distribution with results at different levels, the difference plots or z-score plots of both analyte results may help differentiate between a random error and if the laboratory has a systematic error with the assay system. See Figure 2.

Figure 2 Example of plot of two-analyte evaluation for random versus systemic error.

Sample type, production and distribution

Sample type

In thrombosis and hemostasis EQA schemes, numerous types of specimens can be used as the array of assays types and methodologies require multiple sample types. The EQAS Organization must evaluate the testing methodology that is available within their program clientele and determine the best type of sample matrix that can be used. It may be impossible to accommodate all testing methodology and sample requirements for each test. It is the responsibility of the EQAS Organization to either provide multiple sample matrix materials or determine the most common sample matrix used for a particular assays and analyte and then provide that sample matrix. In most, but not all cases, this will be plasma, but could for example, be serum (e.g., for antibody related tests).

The EQAS is to provide both normal and abnormal samples types in a random mix throughout the testing period. In many cases, it is difficult or impossible to obtain sufficient material from a true abnormal patient. Therefore, alternative sample matrix types will need to be produced, for example by dilution of sample with buffer (example: low factor VIII due to dilution) or adding material to create a sample matrix able to be utilized (e.g., adding Heparin or Direct Oral Anticoagulants to plasma). It is important that the EQA program determine the effects of any sample manipulation, including consideration of how it may affect different methods measuring the same analyte. If the sample is commutable (i.e., the manipulated sample gives similar patterns of results to an unadulterated sample) then the data handling and analysis can proceed as if the sample was a clinical sample.

Samples for molecular analysis should be prepared from whole blood or isolation from other sources to acquire an adequate amount of DNA from a single individual to provide consistency of testing with an adequate amount of DNA to be able to accommodate all of the different methodologies and be able to repeat if an assay should initially fail.

Other types of specimens (such as whole blood) can be used but may pose significant issues because of stability and storage conditions.

If material is obtained from patients or normal donors, it is important that appropriate ethical considerations and regulations be met, including patient consent. If material is obtained from commercial sources, ethical procurement should be confirmed.

All samples from procurement from individual to receipt by the participating laboratory must have traceability. All samples used in an EQAS program must have written and documented traceability. This includes not only what was performed to each batch of specimens but each individual or individuals that performed each task must have documented these steps. This is also required if an outside organization or commercial enterprise performed some or all of the tasks such as mailing the samples, diluting the plasma, etc. A checklist(s) for recording all aspects the process from patient selection through each subsequent task, manipulation and handling must be documented. If problems do arise, then the EQAS Organization will be able to trace the issue, determine, and correct the cause of the problem.

Some of the test material should be preserved during the preparation stage. This allows for a comparative sample so that at the end of the process the sample to be distributed can be compared to the original sample to assure that the processing of the sample did not alter EQAS goals for that sample make-up. The EQAS Organization must ensure that the time for processing of the sample is kept at the absolute minimum so any sample deterioration is minimized.

Sample production

EQAS sample must be made to the highest standards so that any issue or problem with the result must be attributable to the participating laboratory testing and/or their process. This includes not only the sample type being used but also the processing and shipping issues that can arise in EQAS challenge samples. As an example, if the sample matrix is altered then the potential that some instrumentation will not detect the correct end point and thus generate an erroneous result. If samples are difficult to reconstitute, then analyte results may be incorrect and the participating laboratory is trying to correct a problem that does not exist in genuine clinical samples.

Each sample for the EQAS program must be labeled with an identification so the performing laboratory can easily determine what assays and analytes to assess and report. This can be labeled with the actual assays and analytes to be determined or with a code that accompanies the samples as to what assays and analytes to determine. The label should clearly identify the specimen type and which challenge of the current challenge and year for that sample vial. It is useful for the sample label or accompanying paperwork to have the following information:

• Test or vial identification;
• Challenge identification;
• Date or year and challenge number;
• Type of material;
• Warning of potential infectious hazard.

Samples to be used for EQAS distribution must be tested for the presence of infectious agents. The best protocol is to test the sample donor before the samples are obtained and then test again after collection. This is a double check to ensure the sample donor did not convert to being positive between initial testing and sample acquisition. Each sample or accompanying paperwork should indicate that the sample is from human origins, may contain infectious agents, and should be handled with caution and participants should be alerted if the sample has not been tested prior to dispatch. There is national and international transport safety regulation for shipping potentially infectious agents. The samples should be packaged to ensure that there is no potential for leakage or external contamination. The samples must be transported by the fastest method to ensure sample stability and maintain transportation regulation.

Sample distribution

Samples are sent from the EQAS Organization’s distribution point to the participating laboratory site. At this point, the samples leave the authority of the EQAS Organization, so the labeling and packaging requirements are very important. The most efficient way for regulating distribution is to use a computer based distribution and tracking system. The label and outside container for shipping must contain the following components:

• Laboratory’s name and laboratory code;
• Responsible individual (who the sample will be sent to);
• Laboratory address with appropriate locations for direct delivery;
• EQAS Organization (and address);
• Type of sample;
• What to do with the sample upon arrival (refrigerate, freeze).

Laboratories should be encouraged to log receipt and storage of their samples.

Before the samples are sent to the participating laboratories, the EQAS Organization should establish the stability of the samples during the transport process to the participating laboratories. If the samples are found to deteriorate during transport, then alternative methods of transportation must be found. This could be established with 1-3 participating laboratories to ensure the stability of the samples. These laboratories should be the laboratories with the longest delivery times and with the harshest environmental conditions. Alternatively, temperature monitoring devices and in-house stability studies may be employed.

The frequency of distribution is dependent on the regulatory requirements of the EQAS regulatory body and the laboratory’s EQA requirements. In most cases, testing on a quarterly basis is probably the optimum number of times for maintaining an excellent EQAS program. In some instances, performing proficiency testing twice per year meets the regulatory requirements of many programs.

An instruction sheet with all of the general information, the specific proficiency sample information and case studies (if appropriate), detailed instruction about opening the vial, reconstituting the contents of the vial and time table for returning the results. Every instruction sheet must have a prominent statement about the potential hazards of each of the samples.

The data report form should be specifically designed for easy and accurate data entry whether by the participating laboratory into a web-based data entry system or whether data submitted on paper to the EQAS Organization for data entry by the EQAS Organization directly into the computer program. The participating laboratory and/or the EQAS Organization must double check all data entry to ensure accuracy and eliminate clerical errors. The report form should contain three parts:

• Participating laboratory identification code and signature of entering individual;
• Section for instrumentation, reagents and lot numbers (if necessary);
• Data results section with the vial identification and tests pre-listed on the form.

Time frame for completion of testing

The amount of time allowed for completion of the EQAS sample testing is dependent on several factors. Each of these factors will add time from the “send-out” date to the receipt date. Factors affecting the time-to-completion period are:

• Delivery time for samples- from the distribution point to the laboratory;
• Complexity of the assay and analyte methodologies.

Some assays are performed frequently, others less so, or may be batched to minimize costs.

• Time to perform and analyze multiple assays and analytes;
• Time to interpret assay and analyte results (as necessary);
• Time for data entry or return data to EQAS Organization.
• The EQA provider should aim to establish time frames and review these on a regular basis.

Data collection and entry

The participating laboratories must be instructed to incorporate the EQAS sample into their routine test workflow and report the results in the same fashion as the routine samples are reported. This is important because the premise of the EQA scheme is to assess the complete testing process. The closer the EQAS sample can be analyzed as a routine sample, the better the assessment of the accuracy of the participating laboratory.

Data collection by the participating laboratories must be accurate and complete. Therefore, to assure the accuracy of reported data and interpretations, the method of data recording must be easy to follow, simple and complete. The return of data results must be straightforward for the participating laboratories. Either an online data collection system or a paper-based system may be established. The method of submission depends on the data processing capabilities of the EQAS Organization. There are numerous methods for data submission; however, the best methods are those that reduce the potential for clerical errors. The most prevalent errors occur when the participating laboratory must transcribe the data on to the reporting form and then the EQAS Organization must transcribe into the computer database. If the EQAS Organization can set up the system to directly report into their computer system, then the potential for clerical error is significantly reduced. Most clerical errors then become the responsibility of the participating laboratory.

If paper reporting is to be utilized, a specific form should be included with the assays and/or analytes being evaluated. The form should have entry lines to record all data results. The form must be easy to fill out by the performing personnel to ensure that the data results returned to the EQAS Organization are accurate and the correct assays or analyte can be assessed and compared. Information relating to the assay and enabling appropriate analysis of the assay should be collected, which may include the following:

• Full name of assay;
• Manufacturer of the test kit or reagents;
• Instrumentation;
• Lot number of each reagent (and expiration date);
• Data results;
• Assay or analyte interpretation;
• Final interpretation;
• In addition, it is useful to record the following:
  • Technical personnel performing;
  • The test.
• Supervisor or Director approving the submission of the result;
• Individual making the initial interpretation and final interpretation.

Proficiency testing of hemostasis tests and analytes are very difficult to assess as a number of the most common tests are reported in arbitrary units such as seconds. In addition, the reagents are not standardized and reference intervals are variable. These issues raise a number of problems for the coagulation EQAS program, especially if the program is small with few participating laboratories using a particular reagent. There may be issues with lot-to-lot variation as well, for some tests such as PT, APTT and thrombin time (TT). There may be insufficient data points to provide reliable statistics. Many other analytes such as factor assays are difficult to compare, as they do not necessarily have a common standard that is used by all manufacturers. In light of these issues, it becomes important not only to evaluate the actual value the participating laboratory provided but also if that value was considered normal or abnormal. Thus, the results submitted to the EQAS program should include interpretation as well as the numerical value.

The data processing and statistical analysis portion of the EQAS process may encounter difficulty when results are submitted with a different format than expected or with symbols that the data processing program and/or the statistical program does not recognize. These include non-numerical mathematical symbols (<, >, +, −) or the participating laboratory uses different units than is standard. Non-harmonization of units may make it close to impossible to compare the results of a single assay or analyte. The EQAS Organization must be aware of their potential problems and take measures to evaluate the submitted data appropriately. It may be possible to specify the units in which results should be reported, or to have data format specifications for online data entry. In addition, the EQAS Organization must have a set of responses for interpretation results when the results are reported differently. The EQA organization should have a plan and policy for how to deal with clerical and transcription errors.

Statistical analysis of data

One of the most difficult aspects of the EQAS program is analysis of the data. Statistical analysis of data in and of itself is complex requiring a knowledgeable statistician to develop the analytical methods for the size and complexity of the data that is being analyzed. A key issue in the data analysis is to minimize the effect of outlying results by either exclusion of outliers or the use of robust statistical evaluation tools. An EQA program should follow internationally accepted data analysis guidelines (ISO 13528). The EQAS program, with approval from the regulatory agency, must ensure that the performance statistics are consistent with the established goals and objectives of the EQAS program and the overall model of the EQAS. In addition, the statistical analysis of the data must be understandable and useful to the participating laboratories. A basic primer should accompany the results that explains in sufficient detail the results in general and the results from the participating laboratory. The participating laboratory data in relation to the peer-group and overall data should be understandable so a laboratory with outlying data can determine how their data is out and lead them to possible explanations to the reason that their data is not within the reference interval of the main group. In other words, understand the data and correct any issues.

The analysis of the data may be performed with sophisticated statistical software. The analysis of the data usually reports a target value, as the average or median for the group (whether the whole group or a peer group using the same instrument and/or methodologies) with the standard deviation.

In small groups of data either with small peer group size or even small EQAS programs, outliers can cause significant changes in the overall data. Therefore, robust type statistical analysis must be used to for these types of data analysis. Another alternative is to remove those obvious outliers (due to gross errors, transcription errors or miscalculations) but those removed must be documented by the Program Director and commented on in the report. If the statistical evaluation allows for removal of obvious outliers, a procedure with the criteria for outlier removal and the format of outlier removal criteria must be established before the data is evaluated.

Criteria for acceptance

There are no universal acceptance criteria rules. The best way to determine the acceptance criteria is to follow a good quality management plan with outlining the expectations for each assay or analyte. If the EQAS program has good quality associated with the program, then with time the participating laboratories will increase their quality. If the material is commutable material, then the performance evaluation can be reliably be done on the total group as well as different method groups, however if the material is non-commutable, then only evaluation on the level of the method group is valid.

The “acceptable range” of results is dependent on the regulatory agency and their required standards. Several options to “score” performance include Deviation Index or % deviation from the target value, quantiles and ranked grading analysis, and z-scores. Performance limits may be for example +/−15% deviation from the target value, or a z-score > +/−2.

When the sample size is low (often <10 laboratories (either by peer group or test method group or total laboratory size), the statistics are very unreliable and the individual laboratory cannot be accurately assessed by the standard statistical methods. The participating laboratory is usually required to evaluate their own performance against the performance of their peer group to determine if their result is acceptable or not. Comments to this effect should be included in the laboratory’s report.

Each individual hemostasis factor must be evaluated for acceptable criteria prior to analysis of the data or at the very least in a preliminary evaluation with all of the data to determine the acceptability of the standard acceptance criteria.

The results should be evaluated in a variety of different parameters to determine if major differences by methodology or reagents cause changes in the reported values. So, not only are all values to be determined as a single group, but subgroups such as reagent-instrument combinations or assay method types need to be analyzed as well. Along the same lines, sample types should be evaluated as a single group because some substances may interfere with one type of assay and not another.

Each sample tested should also have an interpretation, this is especially important when the assay is an arbitrary unit-based test. Different units will give different results, making comparison difficult. The one common aspect of this type of variation in method results is that the interpretation should be the same. However, it is more difficult to analyze the normal-abnormal analysis from a statistical perspective. With large enough sample size valid statistics can be determined.

Final report

The final report is the most important part of the EQAS program as it is the communication and feedback to the participating laboratories. A sample report is presented in Table 1. The final report provided to the participating laboratory must contain all of the information necessary to address all of the issues associated with the results. There is no set format for EQAS reports but the report must be clear, comprehensive and consistent. In addition, it is usually very helpful to include graphs that show the distribution of the data and where the participating laboratory result fell within the distribution of the data. The statistical data must also be included. Minimally, a report should contain the following information:

• Name and address of the Director of the Laboratory;
• Laboratory code;
• Date report issued;
• Name of the EQAS program;
• Challenge identification number;
• Sample description including how they were prepared;
• Sample code with results adjacent to overall statistics of sample;
• Statistical summaries and graphs;
• Comments on the performance of the participants;
• Comments on the samples in relation to the results (if necessary)

It is also important to include in each analyte report, the performance evaluation from previous surveys (e.g., difference history plots or Z-score history plots) for previous surveys. See Figures 2 and 3. This helps the participant laboratory to evaluate for discrimination between random and systematic errors and long-term drift, as well as assessment of the effect of corrective actions.

Figure 3 Examples of longitudinal plot summaries.

Erroneous result investigation

The use of sanctions for participating laboratories who fail proficiency testing is usually determined by the regulatory agency or legislation. The number of failures in a single challenge or multiple failures in two or more challenges is based on the regulatory agency. In addition, the regulatory agency determines what the sanction will be. This is beyond the scope of this document.

EQAS programs are established to help laboratories improve the quality of their testing, doing so through objective evaluation of performance compared with their peers. When a participating laboratory demonstrates poor performance over long periods with no evidence of improvement, a mechanism should be in place for the EQAS Organization working with the participating laboratory to develop a method to improve testing performance. The participating laboratory, with the support of the EQAS Organization should develop a plan for improvement. This can include evaluation of methodology, laboratory space, processing of specimens and additional education or training of the technical staff. Engagement of the laboratory with the EQA program is likely to be optimal if the EQA program is supporting the laboratory to improve their testing processes.

Acknowledgments

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editor (Emmanuel J. Favaloro) for the series “External Quality Assurance” published in Annals of Blood. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at: http://dx.doi.org/10.21037/aob.2020.02.01). The series “External Quality Assurance” was commissioned by the editorial office without any funding or sponsorship.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. ISO15189-Medical laboratories-Requirements for quality and competence (2012). Available online: www.iso.org
2. ISO17043-Conformity assessment-General requirements for proficiency testing. (2010). Available online: www.ISO.org
3. ISO13528-Statistical methods for use in proficiency testing by interlaboratory comparison. (2015). Available online: www.ISO.org
4. WHO manual for organizing a national external quality assessment programme for health laboratories and other testing sites. (2016). Available online: www.WHO.int/publications