Engineering Software: Verification and Validation

[McDermott1711]

Hi there,
Clause 7.6 of ISO 9001:2008, says:"... When used in the monitoring and measurement of specified requirements, the ability of computer software to satisfy the intended application shall be confirmed. This shall be undertaken prior to initial use and reconfirmed as necessary. NOTE Confirmation of the ability of computer software to satisfy the intended application would typically include its verification and configuration management to maintain its suitability for use."

On the Other hand, in ASME V&V 10-2006, we have guidance for Verification and Validation in Computational Solid Mechanics.

Is there anybody out there that has an experience in preparation of a procedure for V&V of computer packages which are used in engineering design process, e.g. Caesar II, Ansys, etc.?

 

[IRstuff]

Well, you might start by comparing the item under test (IUT) with its specifications. You then might, for something like an FEA, compare its outputs on canonical problem sets that are run on other, proven software. The latter falls under the broader category of regression testing.

As with image processing, one would use the equivalent of a "Lena" image or images. Lena is a headshot picture of a model wearing a hat. I'd guess she's been featured in hundreds of papers on image processing. Likewise, one might start off with simple beams that have analytical solutions and go from there.

TTFN

FAQ731-376
Chinese prisoner wins Nobel Peace Prize

 

[McDermott1711]

Thank you IRstuff for your reply. Would you please tell me more about Item Under Test (IUT)? Considering a broad range of application for engineering software, are you going to say that every item should be tested physically and the result of, say FEM for example, be compared to it? If i'm modeling a storage tank and making an FE analysis for it how can i make a stress (or strain) test on it before the construction begins?

 

[GregLocock]

You can't. But you can compare your FEA results for your previous design with tests on the previous design. If you haven't ever done that you aren't doing FE /A/ you are making pretty cartoons for management.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[McDermott1711]

But Greg, isn't it somehow paradoxical? What about validity of the analysis for previous design, if it's the first one?
Did anybody implement the content of ASME V&V 10 for his/her company?

 

[IRstuff]

Nothing is ever the "first" one. FEA was developed to address problems that couldn't be done by hand. Nonetheless, there were lots of hardware built, and tested. And, those results can be compared with what the FEA results would have been.

TTFN

FAQ731-376
Chinese prisoner wins Nobel Peace Prize

 

[corus]

It's not common practice to verify proprietary software as it's assumed that they are from a 'reliable' provider (such as ANSYS, Abaqus etc.) and that they have their own QA standards to have checked the software. This doesn't mean that the odd bug occurs but it'd be nigh impossible to find these under some system. Validating the results is another matter, as you're really checking the modelling assumptions, input data etc. You're really down to establishing a level of confidence in the model (and the modeller for that matter), and you can only really do that with actual physical measurements. If that's not possible you can only compare the results from the FE model with theoretical values where these can be taken from the model. Alternatively you can compare the results with other FE models that take a different approach to the problem.

 

[IRstuff]

OK, I thought the OP was developing the software. The criteria for V & V as a PE are considerably less. You need to show due diligence, so a reasonable smattering of design examples coupled with a customer list of the software would seem to be apropos to me.

TTFN

FAQ731-376
Chinese prisoner wins Nobel Peace Prize

 

[transmissiontowers]

If the work is done for a Nuclear Power Plant, the NRC has some stringent QA/QC requirements that the FEA software has to meet and the software has to have documentation on file with the NRC that it meets their requirements. Each new version of the software must be re-validated with the NRC.

_____________________________________
I have been called "A storehouse of worthless information" many times.

 

[GregLocock]

Corus - I disagree. We check each new release of software that is used for sign-off against the results for the previous release. That's why I'm currently using 2008 software!

This would apply to vehicle dynamics software, crash, and probably general FEA.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[McDermott1711]

IRstuff says: "OK, I thought the OP was developing the software. The criteria for V & V as a PE are considerably less."

From ASME V&V 10-2006:"...Verification and validation (V&V) are the processes by which evidence is generated, and credibility is thereby established, that computer models have adequate accuracy
and level of detail for their intended use."
So, I think "ASME V&V 10"'s expression is not only for softeware developers. Am i right?

 

[IRstuff]

No it's not. But no one expects a lone PE, let alone a raft of them to validate software to the level that a developer would, particularly with proprietary software, since you would have no clue where the potential weak points of the software might be. As I posted earlier, you have to do due diligence, but it's not your responsibility to do formal software qualification, although you still need to have proven some level of applicability and validity of the software for your intended application.

Otherwise, of course, we'd be talking about thousands of companies running essentially the same identical tests on the same software, which seems a bit pointless. Running the same test 5000 times does not make the software 5000 times better.

TTFN

FAQ731-376
Chinese prisoner wins Nobel Peace Prize

 

[SomptingGuy]

ISO 9001 means that we (vendors) have regression testing in place. Purchasers require 9001 but still do their own tests. Why?

- Steve

 

[IRstuff]

It's a question of liability. If a PE is sued resulting from errors in calculations, he generally cannot claim that the software was at fault if he has no proof of due diligence validation testing.

TTFN
Eng-Tips Forum Policies

 

[SomptingGuy]

So customer demands 9001 compliance from vendor and then:

a) Does their own (redundant) testing.
b) Moans that vendor can't respond quickly to change requests.

- Steve

 

[GregLocock]

Every warranty on MSC software I've ever seen disclaims ANY repsonsibility for mathematical or programming errors and their consequences. So we have to check, I guess.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[IRstuff]

I thought I had posted a further comment, but it appears to have gotten waylaid.

Even if the software company does its own due diligence regression testing, that does not ensure that it's applicable to every other application, or even an application that's only numerically different.

The once was math processor bug on the Pentium that was undetectable for 99.9% of the people using the chip. Likewise, one version of popular math program had a rounding bug in the 5th, 9th, and 11th decimal places, but only if the preceeding digits had a certain pattern. This is why one has always be wary of software results, and do as much testing of your own to ensure that your particular problems are correctly solved.

TTFN
faq731-376

 

[McDermott1711]

From my point of view, this discussion should be divided in two different way (if only Solid Mechanic software are targeted):
1- V&V of software for its declared capabilities.
2- V&V of model, B.C.s and loadings, which depend on capabilities of user, and not software.
BTW, my previous post also has gotten waylaid,it seems nobody already engaged with ASME V&V 10, seriously.