Ongoing improvements in laboratory performance of coagulation factors VIII and IX: recent experience from the RCPAQAP

Introduction

Coagulation factors VIII and IX are the coagulation factors most regularly tested by haemostasis laboratories, in part because Haemophilia A (FVIII deficiency) and Haemophilia B (FIX deficiency) represent the most common bleeding disorders, and in part because of their known associations with bleeding risk. One-stage assays are widely used to measure these coagulation factors (1) with chromogenic assays less widely used (2). There are several clinical reasons to determining FVIII and FIX levels, including to identify factor deficiencies (either due to haemophilia or other reasons such as trauma), as well as for monitoring factor replacement therapies at times of interventions or in response to bleeding events, and/or apparent resistance to factor replacement that may arise in inhibitor development (1-3). Assurance of the quality of laboratory testing is essential in ensuring the performance of the test and hence the results produced (4,5). Internal quality control processes can be readily applied by provision of commercial plasma controls and is primarily a measure of reproducibility (precision) (6). External quality assurance is a supplementary process, allowing for peer group comparison of each analyte with other laboratories, and thus is primarily a measure of accuracy (6).

The Royal College of Pathologists Australasia Quality Assurance Programs (RCPAQAP) is an international QAP with over 1000 worldwide participants, including 100 laboratories enrolled in the Coagulation Factors VIII and IX Program (7). To assess the improvements in laboratory performance in the Coagulation factors program, survey data for FVIII and FIX testing over a six-year period (2013 to 2018) was analysed. Importantly, the assessment criteria during this period were changed from versus factor deficient plasma reagent to versus plasma calibrator. To ensure no adverse outcome of the change, a benefits and weaknesses analysis of this change was performed, and data was evaluated to determine whether the change had a positive impact on Coagulation Factors survey performance by laboratories.

Methods

Each of the RCPAQAP programs has an assessment criterion based upon which laboratories are assessed for their performance. For FVIII and FIX results submitted by participants from 2013 to 2015, the assessment criterion was ‘factor deficient plasma reagent’. From the period 2016 to 2018, FVIII and FIX results were assessed in regards to the plasma ‘Calibrator’ used by the laboratory for that test, since upon review by the expert panel of RCPAQAP Haemostasis Advisory Committee, this comparison was considered more relevant for factor assays, given the calibration process involved.

The acceptable range of results for RCPAQAP participants is determined by the Analytical Performance Specifications (APS), which is also set by the same expert panel of RCPAQAP Haemostasis advisory committee members. The APS consists of lower and upper limits. For both FVIII and FIX, the lower limit is set at “+ or –3 U/dL” (absolute units) when the factor level is ≤10 U/dL, and the upper limit is set at “+ or –25%” (relative units) when the factor level is >10 U/dL. Results outside these lower and upper limits are therefore flagged as being outside the APS on participant survey reports. Such assessments permit laboratories to review their performance against other (‘peer’) laboratories using the same (or even different) methods. In addition to survey reports after each run of survey, an overall review of laboratory performance is also provided to participants at the end of each calendar year.

For this study, FVIII and FIX data on participant APS as collected between 2013 to 2018 was reviewed, with 2013–2015 reflecting assessment against ‘factor deficient plasma reagent’ and data from 2016–2018 reflecting assessment against plasma ‘calibrator’. Six surveys were distributed in each year, with each comprising two samples, for a total of 12 samples/year. Samples reflect a full range of analyte concentrations (i.e., ranging from deficient to normal). Specific participant assessment based on reagent/calibrator is performed when there are ten or more users in a reagent/calibrator group. Here, the laboratory result is then compared against the median of all users of the same reagent/calibrator. For reagent/calibrator groups with less than ten users, assessment is based on the overall median.

The coefficients of variation (CVs) for each group were also compared to assess overall performance. To compare the CV’s more succinctly, data for samples containing the same approximate level of factor, as distributed to participants in different surveys, have been grouped together. In each year, five different levels of plasma with different factor levels are distributed to participants. For the purpose of this evaluation, the groups have been identified as Level A, B, C, D and E. As each of these five levels have varying amounts of factors VIII and IX, it allows for comparison of performance in each of these level groups.

In brief, performance of laboratories over this six-year period has been analysed, in part to determine if the change in assessment criteria has had a positive impact for laboratories by comparing the number of outliers by each assessment criteria for each analyte, FVIII and FIX. In addition, performance of each reagent and calibrator group has been considered to provide evidence for or against the change in assessment criteria.

Results

There were a total 72 data sets for the six-year (2013–2018) period, which overall comprised 10006 participant-reported FVIII results and 8888 FIX results. As noted in methods, FVIII and FIX results were assessed based on the factor deficient plasma reagent used by the laboratory as the assessment criteria between 2013 and 2015, but against plasma calibrator between 2016 and 2018. Figures 1-4 show the factor deficient plasma reagents and plasma calibrators used for each of FVIII and FIX, and the maximum number of users in each group for each year of assessment.

Figure 1 Factor VIII reagent plasma used by participants of the RCPAQAP Coagulation Factors program between 2013 and 2015. The number of users presented here are the maximum number of users across the survey runs in each of the reagent groups for each year. FVIII, Factor VIII; SM FVIII Deficient, Siemens Factor VIII Deficient plasma; DS STA-Deficient VIII, STA-Deficient VIII plasma; HL Deficient FVIII, Helena Deficient Factor VIII plasma; IL FVIII Deficient, Instrumentation Laboratory Factor VIII Deficient plasma; GF FVIII Deficient, Griffol Factor VIII Deficient plasma; TB FVIII Deficient, Trinity Biotech Factor VIII Deficient plasma.

Figure 2 Factor IX reagents used by participants of the RCPAQAP Coagulation Factors program between 2013 and 2015. The number of users presented here are the maximum number of users across the survey runs in each of the reagent groups for each year. FIX, Factor IX; SM FIX Deficient, Siemens Factor IX Deficient plasma; DS STA-Deficient IX, STA – Deficient Factor IX plasma; HL Deficient FIX, Helena Deficient Factor IX plasma; IL FIX Deficient, Instrumentation Laboratory Factor IX Deficient plasma; TB FIX Deficient, Trinity Biotech Factor VIII Deficient plasma.

Figure 3 Factor VIII Calibrators used by participants of the RCPAQAP Coagulation Factors program between 2016 and 2018. The number of users presented here are the maximum number of users across the survey runs in each of the calibrator groups for each year. FVIII, Factor VIII; SM Standard Human Plasma, Siemens/Dade Standard Human Plasma; IL Calibration Plasma, Instrumentation Laboratory Calibration Plasma; DS Unicalibrator, Diagnostica STAGO Unicalibrator.

Figure 4 Factor IX Calibrators used by participants of the RCPAQAP Coagulation Factors program between 2016 and 2018. The number of users presented here are the maximum number of users across the survey runs in each of the calibrator groups for each year. Abbreviations: FIX, Factor IX; SM Standard Human Plasma, Siemens/Dade Standard Human Plasma; IL Calibration Plasma, Instrumentation Laboratory Calibration Plasma; DS Unicalibrator, Diagnostica STAGO Unicalibrator.

The data presented in Figure 1 identifies that the number of users for each of the factor deficient reagents used has not shown significant variation through the years 2013 to 2015, with the exception of IL Factor VIII deficient plasma, which has had an increase in users. Similarly, Figure 2 identifies that the number of users of FIX reagents has not shown significant variation, with the exception of IL Factor IX deficient plasma. Figures 3 and 4 shows that SM Standard Human Plasma calibrator had an increase in the number of users for FVIII and FIX, DS Unicalibrator has a decrease in users for FVIII and FIX, while IL Calibration plasma users has remained reasonably consistent.

A summary of the number of responses received from participants is shown in Figure 5 and Table 1 for both assessment criteria with separation of factor deficient plasma reagent/plasma calibrator for where there are more than ten users. The numbers of outliers/acceptable results are also shown in Figure 5 and Table 1.

Figure 5 Summary of results from 2013 to 2015 for FVIII and FIX reagent groups with more than 10 users. ‘No filter’ indicates all results reported to RCPAQAP. ‘Reagent filter’ indicates all results reported to RCPAQAP as ascribable to a particular reagent. ‘Accepted results’ indicates those results within the Analytical Performance Specifications (APS). ‘Outliers’ indicates results outside the APS. Abbreviations: FVIII, Factor FVIII; FIX, Factor IX; SM FVIII Deficient, Siemens Factor VIII Deficient plasma; DS STA-Deficient VIII, STA – Deficient VIII plasma; IL FVIII Deficient, Instrumentation Laboratory Factor VIII Deficient plasma; SM FIX Deficient, Siemens Factor IX Deficient plasma; DS STA-Deficient IX, STA-Deficient Factor IX plasma; IL FIX Deficient, Instrumentation Laboratory Factor IX Deficient plasma.

Figure 6 shows the trends in the percentage of outliers in each FVIII reagent group between 2013 and 2015. The percentage of background outliers generally, when no filter was applied was <3% between 2013 to 2018. There were minor variations to percentage outliers in these years; however, no identifiable consistent trend was evident. In contrast, when the change to assessment criteria was made from reagent to calibrator, there was a more relevant identifiable trend. Percentage outliers (with reagent filter) between 2013 to 2015 was wider, ranging from 0.4–3.9%, whereas percentage outliers (calibrator filter) between 2016 to 2018 ranged from 0–2.5%.

Figure 6 Trend of participants flagged as outliers for the two assessment methods, reagents and calibrators (with more than 10 users in each group), for Factor VIII based on the RCPAQAP Analytical Performance Specifications (APS). FVIII, Factor VIII; SM FVIII Deficient, Siemens Factor VIII Deficient plasma; DS STA-Deficient VIII, STA-Deficient VIII plasma; IL FVIII Deficient, Instrumentation Laboratory Factor VIII Deficient plasma; SM Standard Human Plasma, Siemens/Dade Standard Human Plasma; IL Calibration Plasma, Instrumentation Laboratory Calibration Plasma; DS Unicalibrator, Diagnostica STAGO Unicalibrator.

Figure 7 shows the trends in percentage of outliers in FIX results with assessment based on reagent and calibrator. Similar to FVIII, the percentage of background outliers generally, when no filter was applied was <3% between 2013 to 2018. When assessment was changed from reagent to calibrator there was a more applicable trend. Percentage outliers (with reagent filter) between 2013 to 2015 were wider, ranging from 0.9–3.0%, whereas percentage outliers (calibrator filter) between 2016 to 2018 ranged from 0.3–1.7%.

Figure 7 Trend of participants flagged as outliers for the two assessment methods, reagents and calibrators (with more than 10 users in each group), for Factor IX based on the RCPAQAP Analytical Performance Specifications (APS). FIX, Factor IX; SM FIX Deficient, Siemens Factor IX Deficient plasma; DS STA-Deficient IX, STA-Deficient Factor IX plasma; IL FIX Deficient, Instrumentation Laboratory Factor IX Deficient plasma; SM Standard Human Plasma, Siemens/Dade Standard Human Plasma; IL Calibration Plasma, Instrumentation Laboratory Calibration Plasma; DS Unicalibrator, Diagnostica STAGO Unicalibrator.

From Figures 6 and 7, the overall reduction in the number of outliers for FVIII and FIX results reported by participants from 2013 to 2018 can also be seen. This seems may reflect a year by year improvement, but more importantly the percentage of outliers is higher for both FVIII and FIX (Figures 6,7) when assessment was against reagent, than when assessment was against calibrator. There were 43 laboratory results marked as outliers in 2013 when the assessment criteria was using reagent filter. By 2018, when assessment criteria were changed to using calibrator filter, the number of outliers had reduced by nearly 50% to 23. Similarly, for FIX, there were 36 laboratory results identified as outliers in 2013, which reduced by over 60% to 14 laboratories by 2018.

Comparative CVs for data returned for different years is shown in Tables 2 and 3. This data shows a clear reduction in the periods representing comparison against ‘reagent’ to against ‘calibrator’.

These findings indicate that the performance of participants in the RCPAQAP coagulation factors programs has improved over the years as well providing support to the hypothesis that the change in assessment criteria from reagent to calibrator was at least partially responsible for some of this improvement. Reduction in the number of participants flagged as outliers may also be an indication of improvements in participating laboratories assay performance.

Discussion

Measurement of coagulation Factors VIII and IX is important to assess bleeding and also potentially thrombosis risk (1-3,8). The RCPAQAP coagulation Factors VIII and IX program is designed to provide an opportunity to participating laboratories to gauge their performance when compared to other laboratories using similar or different assay principles and identify areas of improvement. A change to this program’s assessment criterion from reagent to calibrator was made from 2016 to enable the assessment criteria to be more relevant. The aim of this study was to determine if this change in assessment criteria has had a positive impact on the assessment of participants. In these six years of the coagulation factors QAP program, no changes have been made to the source of the material sent out to participating laboratories or the actual APS lower and upper limits as mentioned earlier. Hence, allowing this review to be reflective of what the assessment criteria change has had on the QAP program.

Looking at the CV’s of both reagent and calibrator (Tables 2,3) allows for evaluating the performance of both assessment criteria. The FVIII and FIX survey results’ CV’s for reagent assessment indicates more variation than that of calibrator assessment, and it is also noted that higher CV’s are seen more frequently with reagents than calibrators. Therefore, this further supports the change to calibrator assessment being more relevant.

From this investigation it is observed that performance of FVIII and FIX testing in the participating laboratories has improved over the past six years. Assessments based against calibrator rather than reagent showed reduction in the numbers of outliers in each peer group and provided improved comparability of results with reduced CVs. The decreasing trend in outliers supports the retention of the assessment criteria being based on calibrator. Based on the period of assessment (2013–2018) it can be concluded that the change in assessment criteria to reagent calibrator had no adverse bearing and in fact had a positive impact on the RCPAQAP Coagulation Factors survey performance.

In addition, this review provided an observation of the number of users of each reagent and calibrator type as shown in Figures 1 to 4. Although this information may not be as significant as the finding of improved performance due to change in assessment criteria, this data shows the trends of users of each reagent/calibrator manufacturer, which may be of interest if further data analysis on the performance of these reagents and calibrators is ever carried out.

There is agreement amongst EQA providers in the fact that EQA programs have an important role in the performance of a laboratory (6,9-11). Looking at other publications in this area of EQA, and based on the findings from this review, the need to standardise assessment criteria is evident (10). To be able to compare performance across various EQA’s, standardisation in reporting is essential. It is the responsibility of EQA providers to deliver reports that are of significance to the participating laboratory as well as having some form of global standardisation in the assessment criteria (12). The findings from this review support the requisite for harmonisation among EQA’s to enable consistent peer comparisons across EQA’s. Identifying clinically relevant assessment criteria for each laboratory test that has an EQA program will take time and will certainly be ongoing. However, the importance of creating global harmonisation amongst EQA providers and assessment criteria will be beneficial to the clinical laboratories as well as the scientific community.

Acknowledgments

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Annals of Blood for the series “External Quality Assurance”. The article has undergone external peer review.

Conflicts of Interest: Both authors have completed the ICMJE uniform disclosure form (available at: http://dx.doi.org/10.21037/aob.2020.02.05). The series “External Quality Assurance” was commissioned by the editorial office without any funding or sponsorship. EJF served as an unpaid Guest Editor of the series. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Disclaimer: The opinions expressed in this manuscript are those of the authors, and not necessarily those of the RCPAQAP or NSW Health Pathology.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Duncan E, Rodgers S. One-Stage Factor VIII Assays. Methods Mol Biol 2017;1646:247-63. [Crossref] [PubMed]
2. Rodgers S, Duncan E. Chromogenic Factor VIII Assays for Improved Diagnosis of Hemophilia A. Methods Mol Biol 2017;1646:265-76. [Crossref] [PubMed]
3. Hassan S, Fijnvandraat K, van der Bom JG, et al. Preventing or Eradicating Factor VIII Antibody Formation in Patients with Hemophilia A: What Can We Learn from Other Disorders? Semin Thromb Hemost 2018;44:531-43. [Crossref] [PubMed]
4. Bonar RA, Lippi G, Favaloro EJ. Overview of Hemostasis and Thrombosis and Contribution of Laboratory Testing to Diagnosis and Management of Hemostasis and Thrombosis Disorders. Methods Mol Biol 2017;1646:3-27. [Crossref] [PubMed]
5. Bonar R, Favaloro EJ, Adcock D. Quality in coagulation and haemostasis testing. Biochemia Medica 2010;20:184-99. [Crossref]
6. Favaloro EJ, Jennings I, Olson J, et al. Towards harmonization of external quality assessment/proficiency testing in hemostasis. Clin Chem Lab Med 2018;57:115-26. [PubMed]
7. RCPAQAP. Available online: https://rcpaqap.com.au/, last accessed 20th August, 2019.
8. Chandler WL, Ferrell C, Lee J, et al. Comparison of three methods for measuring factor VIII levels in plasma. Am J Clin Pathol 2003;120:34-9. [Crossref] [PubMed]
9. Cunningham MT. Quality assurance in hemostasis: the perspective from the College of American Pathologists proficiency testing program. Semin Thromb Hemost 2007;33:250-8. [Crossref] [PubMed]
10. Olson JD, Jennings I, Meijer P, et al. Lack of grading agreement among international hemostasis external quality assessment programs. Blood Coagul Fibrinolysis 2018;29:111-9. [Crossref] [PubMed]
11. Jennings I, Kitchen DP, Woods TA, et al. Laboratory performance in the World Federation of Hemophilia EQA programme, 2003-2008. Haemophilia 2009;15:571-7. [Crossref] [PubMed]
12. Hermans C, Dolan G, Jennings I, et al. Managing Haemophilia for Life: 5th Haemophilia Global Summit. Eur J Haematol 2015;95:1-25. [Crossref] [PubMed]