While I started writing this in response to https://fossbazaar.org/content/quality-matters, but seemd to warrant a separate post.
FOSS users are becoming increasingly apathetic regarding the proactive management of software obtained for nominal cost. The recent Debian example comes to mind, where for an extended period of time, OpenSSL within it had been modified with a code checking tool. Such modification removed a programmatic element important to the generation of the key, such that the total possible key combinations were effectively reduced to a fraction of the total unbroken possibilities. This problem existed for nearly two years, with countless users depending on the code, using vendor solutions to test for the same things, and yet this went undetected.
Our government is embracing FOSS publicly, yet I have heard horror stories. They do not understand the management requirements of software delivered without a vendor, yet they have the same expectations.
http://www.gcn.com/online/vol1_no1/47320-1.html?page=1
Without a defined and active process for the ongoing and diligent public management of software, our government could be stepping into FOSS unprepared. If their motivation is cost, they will under staff the management resources that should be diligently testing all software. While I hope that they are going to staff for increased management requirements in the use of FOSS, there is no assurance either way. What makes the situation even more difficult is that there is no clear process or method for the government to implement that will offer a high degree of quality in FOSS investments.
What is lacking is not the desire to check. The responsibilities tied to the development of a FOSS project no longer ends when the project is compiled. Quality assurance and validation steps are so critical to the ongoing build process that the community needs to be part of it. Commercial vendors do not release code until it has survived a series of tests. Commercial vendors have liabilities to protect their investment, and do so through structured testing and processes, since their money is better spent in quality assurance than in remediation and legal actions afterwards.
FOSS needs a repeatable, measurable, verifiable and public checklist of testing and processes performed by the community in a "trusted" manner to safeguard the code that we all depend on. A public forum allows all of us to check an application, see which tests have been performed and which have not, allows us to contribute to the process, and qualify the contributions of others.
While the code is transparent, who has the skills and ability to look at it with the depth and creativity required these days? We need to make the management and ongoing qualification of open source software a community effort. By having the community actively involved in all pieces of quality assurance, we will have a greater understanding for the complexity in certifying code for distribution, and we will be able to verify that such work has been done.
The answer is not just to engage professional services, or use open source software that is financially backed by a large vendor. Since we lack transparency into the detailed, complex and ever changing process for testing software components, we are better to choose commercial solutions with contracts that put liability on the vendor. Additionally, mitigating the unknown risk of the use of FOSS with service contracts undermines some of the core principles of FOSS. If our only solution is to engage services, our freedoms in the use of FOSS are being undermined due to our inability to use the community to grasp, understand, constrain and manage the problem.
NIST sponsors http://cwe.mitre.org, the Common Weakness Enumeration. It is a database for identifying and describing in a common language, programmatic and architectural weaknesses within software, hardware and operating systems. It provides a reasonable starting point from which to build processes upon.
http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2008-0166
http://cwe.mitre.org/data/definitions/310.html
If we use a public and transparent process for the certification of FOSS, continuing in the spirit of how the code was developed, the strength of the community can actively participate in the management of unknown risk in software. We as a community can impose basic qualitative requirements on software packages. This community involvement in the validation of the code is a natural progression of the popularity and ubiquitous nature of FOSS in our computing lives.
In conclusion, the solution for software quality assurance is in the control of the user community. We need a public process to define, manage and implement validation processes, as well as a community effort to invite an ongoing process to post those results. If our professional services suppliers are worthy of their role of managing open source usage, they should be actively posting their tests, their reviews, their reports. If they do not have the requisite skills to help us manage this problem, we need a better process, and better providers, to help us manage this challenge and it is not going to get any easier soon. We owe this to ourselves, the success and health of our financially strained businesses worldwide, and our national and international security to get this right.
Ernest Park