Why FHIR implementations need a special kind of testing to ensure success – and how to do it
So, you’ve heard about, and maybe even started investigating, this Health IT specification called HL7® FHIR®. Maybe you’ve been playing with FHIR (yes, pun intended) now for weeks, months or even years. Perhaps you’ve attended at least one FHIR Connectathon or certainly hope to in the near future. Regardless of where you’re at with FHIR – building a system from scratch or taking advantage of one of the existing open source FHIR projects – there will be or already has been the point at which you ask yourself, “How do I know my FHIR implementation will work (interoperate) with other FHIR implementations?” In fact, “How do I know my FHIR system is really conformant to the specification?” The answer to these and related questions is “testing“. But then, of course, the next question is, “What kind of testing do I need?”
- There’s unit testing which all good developers know about. But that’s not between systems. It’s making sure your low-level functionality is behaving correctly; e.g. does this method throw the expected exception or error for these input values?
- There’s user acceptance testing which may include system to system testing. But typically, the focus of this testing is having the targeted users of the system run through numerous user interfaces and workflows to make sure navigation through the application is correct.
- Then there’s integration testing. Now we’re getting there. Integration testing is where we exercise the interfaces of the application which means between systems. For our FHIR system that means testing the FHIR RESTful API and validating the data contents.
Great! Integration testing. All we need now is another FHIR system that we know is already conformant and, not conformant.
- Conformant so that we know our system can handle correctly formatted messages. Fortunately, this one, conformant, isn’t so hard. There are many publicly available FHIR test systems out there already. We just need to have a little faith that they will always send us valid, conformant FHIR messages. Hmm. We’ll get back to that one.
- Not conformant so that we know our system can handle incorrectly formatted messages without breaking. Why is this important? Because system failure due to mishandling a non-conformant message has the same end result as failure due to mishandling a conformant one – both outcomes break the system and break interoperability. Ok, all we need is a FHIR system that will send us bad messages. Uh oh. This one isn’t so easy. Which FHIR test system author do we ask, “Would you please break your system so that you can send me bad messages? Oh, not just a couple of bad messages, a whole lot of bad messages.” We’ll definitely get back to this one.
Right now, you’re probably saying to yourself, “Hmm, this might not be as easy as I thought”, or maybe, “This sounds hard, how am I supposed to do that?” Well, thankfully the authors and contributors to the FHIR specification have an answer, the “FHIR Testing Framework”. It’s part of the specification and defines what a FHIR Test Engine is, how it will behave and how testing and test cases are defined.
Excellent! Let’s get started. Oh. Wait. Where do we find a FHIR Test Engine that implements the FHIR Testing Framework? Well, we [AEGIS] have one for you, Touchstone! Let’s take a look.
The Touchstone Project – A Developers Integration Lab Initiative from AEGIS.net, Inc.
Touchstone is the only FHIR Testing Platform that follows and conforms to the FHIR Testing Framework specification and is available as a publicly accessible, open access, cloud based, Testing as a Service (TAAS) solution. Initially deployed in October 2015, Touchstone today now has 580+ registered users across over 200 unique organizations. More than 2,500 test definitions – FHIR TestScript resources – have been deployed supporting the official FHIR versions of DSTU2 and STU3 as well as the snapshot releases for the last 8 HL7 FHIR Connectathons.
That last point regarding “TestScript” resources is important. Rather than the test engine defining the tests within its code, all test definitions are represented and executed from the TestScript resources. This means new and updated test definitions simply need to be uploaded to Touchstone and the Touchstone FHIR Test Engine executes that definition.
Why is this important?
Remember what we needed for our integration testing… a “FHIR system that we know is already conformant and, not conformant”. A FHIR Test Engine simply executes what is defined in the TestScript. Guess what, we can define a TestScript that sends not only valid but, invalid FHIR operations. This capability allows implementers to focus on building conformant FHIR systems. These systems can then be easily tested not only for happy-path and success scenarios but also negative and exception handling scenarios without relying on another implementer to ‘break’ their system.
I encourage you to explore the FHIR Testing Framework specification as well as how AEGIS’ Touchstone Testing Platform implements a FHIR Test Engine and the TestScript resource. Touchstone provides a comprehensive environment for developers, quality assurance testers, managers, etc. to execute and manage their testing scenarios. I will be continuing this exploration and presentation of these Touchstone features and capabilities. Stay tuned for more…