Software Best Practices

I would very much like to introduce some Agile practices but work nearly exclusively in the medical device area. By way of explanation for today's post, a project I managed just finished its initial FDA on-site inspection. And let me tell you, those vinyl gloves can get into some pretty sensitive areas.

So here is my question: how do you balance the FDA's desire to have all requirements and hazards detailed before design begins with agility? The best I can come up with so far is to do BDUF and then do tight implementation iterations.

Any ideas?

Kevin
 

Kevin,

Your proposal is a good start - basically you're describing a stage-delivery lifecycle.

But, I suggest you really question the assumption that "FDA desire to have all the requirements and hazards detailed before design."  We've worked with many companies that had perceived restrictions - even FDA. 

I ran into this when I was in the AF. I worked on a project that people kept saying we need to do "blah blah blah"  because it was required by Military Standard 498.  Well I fooled them by reading the standard and found that it really didn't say what everybody thought it said. We were able to create a very adaptive process and while still complying with the standard.

Here's an experiment for you - google Agile +FDA.  You will find many companies that successfully convinced FDA that Agile was appropriate for their needs.  Here's an interesting article I pulled up that talks how one person dealt with the same questions. http://www.jacquette.com/articles/agilevalidation.shtml

So the key, is make sure you can show how your processes comply with FDA objectives, even though they don't follow the steps prescribed by some process wonk.

Thank you for bringing this up. I, too, work with medical devices and have found it very hard to get people to change from a waterfall process, even when that process is failing miserably. I often hear the excuse - "the FDA won't allow it". But in the FDA documents I have needed to read (Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices for one), there is no directive to use a particular process. In this document they give a definition of software validation:

"Software validation refers to establishing, by objective evidence, that the software conforms with the user needs and intended uses of the device. Software validation is a part of design validation of the finished device. It involves checking for proper operation of the software in its actual or simulated use environment, including integration into the final device where appropriate. Software validation is highly dependent upon comprehensive software testing and other verification tasks previously completed at each stage of the software development life cycle. Planning, verification, traceability, configuration, and many other aspects of good software engineering are important activities that together help to support a conclusion that software is validated."

Nothing in this conflicts with an iterative process except the phrase "... design validation of the finished device". And to handle this, I think it is a matter of viewing each deliverable in an iteration as a finished partial device. And then convincing the FDA auditor of this very thing!

One way or another, people in this field are going to have to change - managers/developers/FDA alike. Because when it comes to cutting-edge medical device software, a waterfall life-cycle just sucks.

susan:

Thank you for bringing this up. I, too, work with medical devices and have found it very hard to get people to change from a waterfall process, even when that process is failing miserably. I often hear the excuse - "the FDA won't allow it". But in the FDA documents I have needed to read (Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices for one), there is no directive to use a particular process. In this document they give a definition of software validation:

Nothing in this conflicts with an iterative process except the phrase "... design validation of the finished device". And to handle this, I think it is a matter of viewing each deliverable in an iteration as a finished partial device. And then convincing the FDA auditor of this very thing!

One way or another, people in this field are going to have to change - managers/developers/FDA alike. Because when it comes to cutting-edge medical device software, a waterfall life-cycle just sucks.

I agree, there is nothing in the guidance documents themselves that preclude a particular development cycle. I am referring to my particular recent experience of the FDA auditor asking for finished requirements, hazard analysis, etc. Guidance is one thing, but there is another phrase in the guidance to remember: "non-binding". I don't know if you have had an auditor show up at your site, but telling them they are the ones who have to change does not go over very well :)

There is another practical aspect to going tightly iterative across the whole development cycle, and that is the repetition of revising hazard and risk analysis (and lots of other) documents extensively for each iteration. Not too mention the overhead of independent design reviews, yada yada ad infinitum. The FDA definitely wants to see hazards analyzed and mitigated before design and evaluated at each gate. If anyone has suggestions on how to manage this one, I may just kiss them.

Jerry, 

The article you linked to is excellent. I will definitely be including some ideas from that in my next MD project. In particular, I like the idea of keeping the SRS open as long as possible before instituting change control. It would definitely require a well-written SOP that hits all of the required deliverables, but it seems eminently doable.

There is a clear tension here between a process guaranteed to pass FDA muster (notice I didn't say product) and one that requires some time to educate both the 510(k) reviewer and the on-site auditor. Another internal hurdle is almost guaranteed to be the company's compliance officer, a group of people notoriously change and risk averse.

I had previously done the "Agile +FDA" search and hadn't come up with a lot. If anyone else has links, please share.

Kevin:

telling them they are the ones who have to change does not go over very well

You're right that you can't just tell them to go away -- you need to work with them to show how the practices comply with the objectives.  While there are some regulators who adopt the My Way Only mindset; many more are a lot more willing to work with you to put in practical practices.

Having examples from other companies that have been approved is a great help.  One way to approach the auditor is to start by explaining how you understand the standard and asking the auditor what's wrong with your understanding.  Then explain your problems, share how other companies have addressed the issue, and how you would like to handle it.  This approach is useful because it eductes the auditors while showing them you still recognize the need to comply with the regulations. 

Good luck 

There is also some flexibility with what you call a requirement. My review of the FDA guidance says that you must show that you met your requirements but does not define a requirement.

I have one client that has taken the word "requirement" down to the number of pixels a button must have on a UI screen.

Tom Gilb would have us have a page or so of "requirements" and lots of supporting documentation about how we are going to meet those requirements.

The point is, our decisions of how to do requirements work itself will impact FDA compliance.

Good discussion. My comments are here:

Agile development in a FDA regulated setting

Bob 

Hi Bob,

I like many of your comments, but I have to take issue with a couple of them.

rnadler:

1. I just don’t see how a cost-effective quality system process implementation can be accomplished. Even if (and it’s a big if) the actual overall software development time was shorter, the extra costs incurred by the additional process controls, documentation and testing required for each iteration would far exceed those savings.
2. The last point Mr. Jacquette makes is the other issue:
   

   Take the time to explain agile methodologies to your regulatory specialists and work hard to gain their understanding and agreement, because the burden of proof will be on them and you if an FDA auditor decides to take a peek under the hood.

   This seems like a huge risk to me.

The first question becomes: Is the possibility of an improved product (features and quality) worth the additional cost? I suppose if you had a development team with extensive Agile experience you could make the argument that it would be worth it. If not, I think the ROI (return on investment) analysis would be a difficult one to make.

 

From your comments, I am assuming that you do not see a risk in using a waterfall life cycle when developing medical devices. But there is! A nice book that talks all about this is Agile & Iterative Development by Craig Larman. There are so many studies that show a waterfall life cycle fails often, and in very big ways. It is in no way risk free.

You also seem to assume that iterative processes take a lot of effort to implement and people need lots of experience to do it well. This depends on just how much and how heavy you want that process to be. I work with a group that produces embedded software for a medical device. And while we cannot convince management to fully adopt anything iterative, we do our own iteration plans every 2 weeks. We tend to deliver something every 2 weeks as well, even if it is just to other people on the team. It is a beginning. And it has had a positive affect, at least on the team. And if that is happening, it is good for the project.

One last note.. iterative processes are not new. They have been around since the 60's. The NASA space shuttle software development used 17 iterations over a 31 month period and this was developed from 1977 - 1980. This software is widely recognized as top notch.

cheers,

Susan the obviously iterative, agile proponent:) 

Hi Susan,

I think Bob was more concerned about the business risk that the FDA would not accept the process than the product or project risk that is associated with waterfall type development. The issue is that, as I understand it from my regulated clients, is that you are supposed to define what you are going to build and get that approved before you build it. This, of course, would require a bit of ingenuity for a lifecycle that does not fully define it until you build it. You could say we define it in the first two weeks of the sprint and build it in the last two weeks but then do you do a regulatory filing (not the most agile process in the world) every sprint? Probably not.

The questions I really think for agile and FDA is more along the lines of how much do we need to define it. I believe my FDA regulated clients define way too much. I could go more with a Gilb EVO approach (the first agile lifecycle?) that has a sharp but small requirements bit (needed for regulatory filing) and then incremental and evolutionary development.

Earl Beede:

The questions I really think for agile and FDA is more along the lines of how much do we need to define it. I believe my FDA regulated clients define way too much. I could go more with a Gilb EVO approach (the first agile lifecycle?) that has a sharp but small requirements bit (needed for regulatory filing) and then incremental and evolutionary development.

I was initially nodding my head as I read this, but upon further review I like it less and less. The problem is that regulatory filing is not a singularity. As defined above, only the functionality of that small requirements set would be eligible for sale. Additional functionality in the incremental development process has a high probability of being subject to additional regulatory filings. That is a ginormous paperwork hurdle, as every design change necessitates revisions to the hazard analysis, needs an independent design review, blah, blah blah. At a minimum, software updates can easily be seen by the FDA auditor as a recall of existing product (an experience I recently went through) and there is additional documentation to be done on that end as well.

I am sticking with my original thought (for now), that requirements definition be comprehensive and implementation be iterative. I would love to find compelling arguments to change my mind, but haven't come across them yet. 

Kevin 

Ah, I combined two thoughts into one sentence. Bad Earl.

First thought, define requirements and then build (in agreement with Kevin)

Second thought, be careful what you identify as requirements. A client has many design details that they label as "requirements". Makes validation not so fun. However the Gilb approach (see how I combined ideas) would have us identify the key drivers of the project (a few pages) as the requirements and the rest of the details (design?) of how we tend to implement those key ideas.

The trick is 1) fully define your requirements up front and 2) make sure they are truly requirements (not design/solution requirements). Many projects I have seen in the FDA regulatory space do neither.

 Kevin,

Kevin:

I am sticking with my original thought (for now), that requirements definition be comprehensive and implementation be iterative. I would love to find compelling arguments to change my mind, but haven't come across them yet. 

I have never been in an environment where up-front, comprehensive requirements have worked well. Does this actually work for you? I realize that this is what most FDA auditors will want to see. And I know the labor involved in making changes to all those millions of documents that normally go with this process. But someone has to champion a change.

The book I mentioned, Agile & Iterative Development (Craig Larman), has a chapter entitled 'Evidence'. He shows how the DoD transitioned from recommending a waterfall lifecycle to recommending an evolutionary lifecycle (MIL-498). Under the software requirements analysis section of this doc, it says:

If a system is developed in multiple builds, its requirements may not be fully defined until the final build. The developer's planning should identify the subset of each system's requirements to be defined in each build and the subset to be implemented in each build.

There is even a section in this standard entitled "Removing the Waterfall Bias". 

I mention this because if the DoD is doing this, you know the FDA will follow. We may as well help them along by coming up with processes and tools that help streamline an iterative lifecycle.

Susan 

I tend to agree with Kevin. His definition of a 'ginormous' effort is what my original thinking was regarding the cost effectiveness of an iterative process. I think we're both talking about the 'Making Agile Heavyweight' approach from Adrian's presentation (pdf).Even after hearing that Frank has had success with Agile solutions and FDA regulators I still have not changed my mind regarding being cautious with the FDA. I'd love to see more evidence that people are having success. I should note that my experience is based on class II 510(k) device submissions. There are of course many other types of FDA regulatory interactions, so we should make sure we're not comparing apples to oranges.

Susan's points are well taken. The Waterfall development approach has been around a long time and has many known pitfalls. I think one of the major challenges facing Agile is simply buy in, and not just by the FDA. Any software engineer that has worked for more than one company understands the concept of software development culture and what a profound affect it has on how you work and interact with others. One of the clear advantages of Agile is that it is inclusive by design. All stake holders must be fully engaged in order for the process to be effective. In my mind this is just expanding that culture to include those outside of the actual software development activities. As Susan has found -- 'cannot convince management' -- this can be a difficult thing to do. Not only do you have to rally internal resources, but getting consistent and reliable customer feedback can also be a challenge.

Susan:

I mention this because if the DoD is doing this, you know the FDA will follow.

I wouldn't hold by breath waiting for the FDA to follow DoD practices! :-)

Bob

Great thread - I am not sure if it is still active, but here are my 2 cents on this:

To me whether to use waterfall or agile highly depends on what you are doing. Waterfall is good for well-deffined, known-effort tasks, like painting your backyard fence. Not so good for any innovative, not-tried-before development efforts, which is unfortunately the type of effort FDA is trying to regulate. Never, ever in my experience I have seen a case where ALL requirements are striclty defined in the begining and nothing changes.

 But to me the biggest flaw and business risk involved with the watefall model lays in its inherent structure: Simplified, it looks like: Requirements - Design - Testing - Verification (make sure we build the thing right) - Validation (make sure we bild the RIGHT THING). Only after the enormous effort is spent, we are support to make sure we built the right thing? And what is the chance that we did not (given that we "froze" requirements in the beginning, maybe never talked to the customer exepb occasionally throwing paperwork over the wall back and forth)?

In my experience putting the emphasis on the paperwork and not on the product and customers actually impedes the quality. Unfortunately it is right that FDA is stuck with the waterfall model and most likely will frown upon any other, so we will be seeing more low quality products acompanied with lots of paper. And lots of trees will be cut for this paper.

 Ivan

Hi Ivan,

iivanov:

In my experience putting the emphasis on the paperwork and not on the product and customers actually impedes the quality. Unfortunately it is right that FDA is stuck with the waterfall model and most likely will frown upon any other, so we will be seeing more low quality products acompanied with lots of paper. And lots of trees will be cut for this paper.

I heartily agree with your comments. But there is some good news about this. I have a colleague who attended the 9th 'Software Design for Medical Devices' conference. There were 2 speakers from the FDA there, Richard Chapman and Brian Fitzgerald, and they were asked multiple questions about the FDA's stand on agile processes. Both of them said the FDA is open to these processes and realize that the current approach to software validation is not working. They still do have concerns that an agile process would not produce the required paperwork :(, but they were more than happy to discuss this. The 10th annual is coming up: http://www.iqpcevents.com/ShowEvent.aspx?id=79066&details=79098, if you want to see what they talk about.

 cheers, Susan

Good points. Agile is a great risk mitigation process. Design and code reviews, as well as, test driven development can also be used in waterfall type processes.

The original paper defining the waterfall process is at the link below.

Winston Royce described several processes at an IEEE conference in 1970.  The interesting thing is that he says the waterfall doesn't work and describes several other processes that would be better.  Many steps are quite agile including large involvement by the customer and iterations. He even advocates building the system twice.

http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf

I don't know of anyone that has tried the strict waterfall model. He advocates documentation to improve quality, particulalry for a dedicated qa staff. He's talking about very large development teams too.   

talmans:

Winston Royce described several processes at an IEEE conference in 1970.  The interesting thing is that he says the waterfall doesn't work and describes several other processes that would be better.   

It has befuddled me for some time now, how the heck waterfall got such a following when the original guy (Winston Royce) said it really wasn't a very good process. If only....

In my experience no one ever really used waterfall but it seems that all non-agile processes have been lumped together under the waterfall name.  There've been many iterative processes over the years.  I find it interesting that each iteration of an agile project is really a mini-waterfall dev cycle.

I did work at a company that tried, very hard, to use a waterfall lifecycle. All the requirements were established, through many, many meetings, and we tried to lock them down. And then the engineering department tried really hard to design an architecture (that we hoped would last for years), and then we designed, then coded and finally tested. But, of course, the requirements never did stay the same. So there was a constant battle between marketing/sales people who wanted new/different stuff and the engineering group who was trying to lock everything down so we could do our work and meet our schedules. At the time, I think I thought that the marketing people were just flaky, etc. And a frustrating thing would happen.. the engineering department would slap together a demo version of the stuff (under pressure from marketing/sales), and this would then get into the field and then this ugly prototype would become a real version. It left a bad taste in my mouth about doing quick prototypes. I actually had to overcome this when I first  started learning about iterative processes. The idea of doing quick prototypes scared me. But I finally realized that quick prototypes are just fine, in fact, they are essential, if they are produced at the same level of quality as the production code should be. The key, in my mind, to any iterative process: extremely high quality every step of the way (although, this applies to all processes I suppose). I look back on those days and realize what a waste of so many people's energy it was to try and force a waterfall approach.

I've had similar experiences. I've done the traditional processes with quality evolutionary prototypes and with quick ugly ones.  This came with all the frustrating negotiations with marketing too.  Those were mostly about accommodating changing requirements and still meeting the marketing target date.  Adding to the frustration is asking endless questions to get more precise quality requirements and marketing doesn't have the answer to those yet.  Prototypes really make this easier and show early progress.

I understand your point and I've had similar thoughts. But I don't look at it as forcing a waterfall approach. All the steps in an agile process are available in the traditional processes.  In agile they're built in and in the others you have to choose.  In agile, the schedule vs. quality tradeoff is negotiated up front and it’s a prototyping risk mitigation process from the start which helps.