Blog post #4, February 2024
Did you know that a fraction of algorithm-based tools that are conceived go on to get externally validated and even fewer make it to implementation in clinical practice, not least because of the burden of registering it as a medical device. There are many steps to take from an initial clinical risk calculator to a finalised analytical tool. PRIMIS’ analytical tool which contains the FAMCAT2 algorithm is one fully successful and realised implementation, which has even gone through registration as a medical device. The FAMCAT2 algorithm has been developed by University of Nottingham academics (PRISM) and has high predictive accuracy to identify Familial Hypercholesterolaemia (FH). The conversion of an algorithm to a fully functioning in-practice resource is long, convoluted with many steps prone to failure if not completed successfully. A key task to a successful recreation of the clinically predicted outcome is to ensure the clinical coding involved represents the researchers’ concept as closely as possible. With FAMCAT2, the original code groups used were in a different nomenclature to the proposed final analytical tool. PRIMIS mapped the code groups over to SNOMED CT, with a final set being approved for implementation. However, it’s not enough to just build a tool using the agreed code groups; it must produce the correct, expected results, evidenced via thorough testing. Testing includes comparing results with manual calculations from the original research findings and further testing to ensure accuracy is delivered in the proprietary or third party solutions (depending on how it will be deployed). Any algorithm-based tool must fit seamlessly into day-to-day working for clinical staff, without creating unnecessary workload and provide useful, clear results that can be acted upon. FAMCAT2 produces a clear list of patients for review that have a high probability score and is designed to result in a manageable number whilst retaining the likelihood of identifying new cases accurately. A crucial stage in development is assessing the impact on patient safety. PRIMIS is thorough in this area, basing its clinical safety cases on a hazard log (a list of identified and perceived risks). Risks are analysed and mitigated where possible. PRIMIS can build implementation tools outright but often works with supplier organisations who deploy our specifications, code groups and application programming interfaces (APIs) within their own systems. APIs are securely controlled and centrally updated: ensuring consistency and accuracy in third party tools. PRIMIS worked with the Digital Research Service at the University of Nottingham on the FAMCAT2 API as a way for suppliers to access the algorithm. Getting an algorithm from concept to a working clinical practice tool is a rare feat. FAMCAT2 is a success story in that regard. PRIMIS' extensive experience in translating risk calculators into real world NHS IT systems means we have many successful past and current projects that incorporate an algorithm. One such project that we are now developing is MODUS, where PRIMIS is collaborating with researchers from the University of Nottingham to build the MODUS risk calculator within live GP clinical information systems. The calculator is designed to identify patients at higher risk of immune suppressing drug toxicity. PRIMIS is working alongside the academic team to reproduce the calculator within live GP clinical information systems with the potential to reduce the number of blood tests being routinely performed in primary care. Read more about both FAMCAT2 and MODUS on the Projects page
Did you know that a fraction of algorithm-based tools that are conceived go on to get externally validated and even fewer make it to implementation in clinical practice, not least because of the burden of registering it as a medical device. There are many steps to take from an initial clinical risk calculator to a finalised analytical tool.
PRIMIS’ analytical tool which contains the FAMCAT2 algorithm is one fully successful and realised implementation, which has even gone through registration as a medical device. The FAMCAT2 algorithm has been developed by University of Nottingham academics (PRISM) and has high predictive accuracy to identify Familial Hypercholesterolaemia (FH).
The conversion of an algorithm to a fully functioning in-practice resource is long, convoluted with many steps prone to failure if not completed successfully. A key task to a successful recreation of the clinically predicted outcome is to ensure the clinical coding involved represents the researchers’ concept as closely as possible. With FAMCAT2, the original code groups used were in a different nomenclature to the proposed final analytical tool. PRIMIS mapped the code groups over to SNOMED CT, with a final set being approved for implementation.
However, it’s not enough to just build a tool using the agreed code groups; it must produce the correct, expected results, evidenced via thorough testing. Testing includes comparing results with manual calculations from the original research findings and further testing to ensure accuracy is delivered in the proprietary or third party solutions (depending on how it will be deployed).
Any algorithm-based tool must fit seamlessly into day-to-day working for clinical staff, without creating unnecessary workload and provide useful, clear results that can be acted upon. FAMCAT2 produces a clear list of patients for review that have a high probability score and is designed to result in a manageable number whilst retaining the likelihood of identifying new cases accurately.
A crucial stage in development is assessing the impact on patient safety. PRIMIS is thorough in this area, basing its clinical safety cases on a hazard log (a list of identified and perceived risks). Risks are analysed and mitigated where possible.
PRIMIS can build implementation tools outright but often works with supplier organisations who deploy our specifications, code groups and application programming interfaces (APIs) within their own systems. APIs are securely controlled and centrally updated: ensuring consistency and accuracy in third party tools. PRIMIS worked with the Digital Research Service at the University of Nottingham on the FAMCAT2 API as a way for suppliers to access the algorithm.
Getting an algorithm from concept to a working clinical practice tool is a rare feat. FAMCAT2 is a success story in that regard.
PRIMIS' extensive experience in translating risk calculators into real world NHS IT systems means we have many successful past and current projects that incorporate an algorithm. One such project that we are now developing is MODUS, where PRIMIS is collaborating with researchers from the University of Nottingham to build the MODUS risk calculator within live GP clinical information systems. The calculator is designed to identify patients at higher risk of immune suppressing drug toxicity. PRIMIS is working alongside the academic team to reproduce the calculator within live GP clinical information systems with the potential to reduce the number of blood tests being routinely performed in primary care.
Read more about both FAMCAT2 and MODUS on the Projects page
Back to the blog home page
Applied Health Research Building University Park University of Nottingham Nottingham, NG7 2RD