Before medical devices can be used by the public they need to be tested, either with actual patients or through the use of simulation. Richard Featherstone discusses the advantages and pitfalls of these two types of testing.

Simulated use human factors testing is an established requirement for many types of medical technologies, in particular those devices that are used in higher risk scenarios such as operating rooms and intensive care units.

However, a recent trend is for manufacturers to be asked by the regulators to provide human factors data based on actual use in addition to simulated use. This is a relatively new development, and seems to be coming from reviewers within the FDA who appear to be applying a ‘clinical trials’ mindset to human factors study requirements. There is only passing reference to them in guidance, and methodologies are not widely understood.

What is ‘simulated use’ human factors testing?

Testing your product during simulated use involves recreating the scenarios in which you anticipate that your device would be used. The closer you can get to reality, the more naturalistic the behaviours of the study participants. Creating a realistic environment may involve setting out a room to simulate an operating room, including lights, drapes, visual and auditory displays, and using a mannequin to simulate the patient. The newer mannequins can be programmed to simulate particular medical conditions and exhibit symptoms such as a particular breathing rate, oxygen saturation and pulse. Users will interact with the device in a naturalistic way and perform use scenarios that are designed to simulate the frequent use scenarios for your device.

Simulated use testing applies the rigours of a study methodology in a way that does not endanger patients but allows just enough realism to generate reliable, representative human factors data. However, it has the drawback that it is obvious to the user that there is no actual patient involved, and therefore no risk of harm. How much this affects user behaviour is a matter for some discussion, but it is undeniably less realistic than actual use.

What is ‘actual use’ human factors testing?

Actual use involves your intended patient using the device to receive the drug or have the procedure performed on them. Actual use of your product can obviously be done during clinical trials before it is launched. If you are asked to conduct an actual use human factors study as part of your regulatory file, it will in practice mean that you will need to gather human factors data during a clinical study. An actual study of your product once launched usually means conducting some type of post-market survey of your users.

How do the two types of study compare?

The two types of human factors study generate human factors data with distinct differences, and the methodologies involved differ too. Some of the key differences are:

• Sources of bias. Bias is introduced into simulated use studies because the participant and the study moderator both know that there is no real patient involved, and therefore there is very limited potential for harm. Bias is introduced into actual use studies when an investigator intervenes to collect human factors data, for example by the use of diary cards recording use difficulties, or electronic monitoring technologies to track device use in real time.
• Use scenarios. A simulated environment enables the study moderator to simulate very precisely a use scenario and to gather data points that lead directly from it. Because we are not putting a human being in harm’s way, we have more scope for gathering human factors data in ways that may be impossible during actual use. In simulated use studies we can recreate a massive trauma scenario quite precisely, and reproduce it exactly from one participant to the next and thus give the data some rigour.
• Linking usability to clinical outcomes. This is clearly where actual use testing wins over simulated scenarios. A clinical study is designed to generate data on clinical outcomes and involves real users using the device with real patients in a controlled way. When the product being tested has a user interface, and therefore requires the user to perform specific tasks, there is obviously a potential to link the use of the product to the clinical outcomes for the patient.

Whilst actual use studies can potentially link to clinical outcomes, simulated use studies can get part way there too. For example, if a series of ‘surrogate markers’ of clinical outcome can be defined, there is the potential to provide data that provides some link to outcomes. If a simulated use study can show that the full dose of drug was delivered in a way that could reasonably be expected to lead to clinical benefit, then that data is of some value. Or if you can show the time taken for a healthcare professional to respond to a medical crisis, and if there is a proven link between time to treat and outcome, then again simulated use human factors data provides value.

The moderator in a human factors study has a crucial role in avoiding bias, and there is a very fine line between guiding the participant through the tasks and intervening and therefore biasing the data. I have not yet met a clinician who would also regard themselves as a human factors expert, and the most practical solution is for both types of investigator to work together to a common agreed protocol.

So can actual use and simulated use studies work together?

There are some fundamental differences between human factors studies and clinical studies that make it very difficult to see how one study can satisfy the requirements for both clinical outcomes data and usability data.

For example, let’s imagine you are developing a new type of inhaler device for diabetes. You have an open label clinical study in which patients use the inhaler daily for a month, and you then collect data on certain clinical outcomes such as blood glucose levels.
On enrolment into the study, you will need to teach the patient how to use the inhaler correctly so that every patient starts off with a comparable technique and you have a baseline against which to compare their technique later in the study.

At the first follow up visit, you ask your patient to demonstrate their inhaler technique. If their technique is poor, it is going to cause bias because their technique has deviated from the baseline you established at the start of the study, plus it is not ethical to send your patient away knowing that their technique is poor and that they are likely to struggle to use the device. However, you could record their difficulties and explore the reasons for the difficulty, but you must train these out again, and send them away with an improved technique.

Now we come to the end of the study, and you bring the patient back into clinic for all the battery of clinical tests you want to run such as blood tests, quality of life and of course, an assessment of their inhaler technique. But since you have been correcting the use-related difficulties along the way, how much rigour will there be in any claimed correlation between the user’s performance with their inhaler and the clinical outcomes?

Can technology help?

For pre-launch products, the only opportunity that manufacturers have to gather actual use human factors data is to include human factors data collection during clinical trials. One of the problems presented by clinical trials is how to evaluate the use of a product when it’s being used at home unsupervised.

What should the clinician do if the patient has difficulties using the device? The ethical choice is to correct the user’s technique, but this may introduce bias. If the patient’s technique is not corrected, does this introduce bias into the clinical outcome?

There are some technologies that may help, such as remote monitoring of devices using inbuilt electronic monitors, small video cameras placed in the patient’s home, and tele-health systems that require users to record data on their product’s usage. We have used some of these technologies in our studies.  None are perfect, and none give the rigour that a simulated use study would. However, with smaller and more discrete sensors that may be invisible to users, it may be possible to collect reliable data during actual use.

The best quality human factors data can only be generated by a well-designed simulated use human factors study. Actual use studies are probably necessary only where a simulated use scenario is not feasible or would not give results that could be applied to the wider user population.

By Richard Featherstone

This article first appeared in The Ergonomist, No 533 November 2014

Back