Quality Matters

FOSSBazaar is no longer being updated. The information on this site is preserved for your convenience but may be out of date. Please visit Linux Foundation's Open Compliance Program for current information and activities.

andrew's picture

A question that frequently comes up when talking with those who are new to FOSS is, "But how can I be sure as to the quality of the code?" And I would suggest that the answer to which, as with proprietary software, is largely the application of common sense. Due diligence in the form of seeking out case studies, soliciting the experiences of peers and the collation of other empirical evidence. Nothing new or difficult here.

It may be that a vendor features in the equation, and in which case they should be able to provide some level of assurance also. However, the absence of a vendor should not dissuade you from doing what you would have done previously. But you must also consider the ability of your organisation, or its contracted support provider, to maintain the code.

Where FOSS does differ, is in what it allows you to do that is simply not possible with (most) proprietary software. At least not without, if you are lucky, signing an NDA and handing over a lot of cash... You can inspect the code. You can speak direct to core developers. You can observe in real-time the evolution of the software. In short, the FOSS paradigm brings about an unprecedented transparency, that enables to you inform decision making in ways that not very long ago would have been unimaginable.

Despite these benefits there is still room for improvement. There is work that can be done to help build further confidence in FOSS, and as such will serve to accelerate its adoption.

Due to the fact that most FOSS is developed in a very public manner, with open access to RCS repositories and mailing lists Etc, FOSS code and the associated communities have become a prime target for researchers. And much work is being done to develop analysis methods, tools and metrics that will give some indication of, for example, a FOSS project's health. Such iniatives include: QualiPSo, SQO:OSS, QualOSS and FLOSSMetrics. And what needs to happen now, is for the the research and FOSS communities to work closer together in order to effect mutually beneficial outcomes.

As ever, I'd be interested to hear the thoughts of others. E.g. For you, what builds confidence? Which metrics, if any, matter? Would you prefer to have the tools to collect your own metrics, or would access to a service which does this for you be preferable?

stormy's picture


I think one of the things that builds confidence is being able to see who else is using it and how many people are using it. That's why case studies, word of mouth and efforts like Open Source Census, http://osscensus.org.

You are much more likely to think a restaurant with lots of people in it has better food than the empty one across the street.

You are much more likely to take your car to the mechanic that all your friends recommended.

You are much more likely to use software that the rest of the world has evaluated and decided it's good to use.

ernest.park's picture

Quality - a community responsibility

I was writing this as I saw this post . . .


I agree that processes are in place, and I agree the code is transparent (everybody gets to look), and that the community "should" come together on this. 

However, I still recall the recent Debian OpenSSL example where the crypto key possibilities were reduced by 90% percent thanks to the use of code review tools, yet nobody caught this weakness for nearly two years.

Now that everyone has a cell phone, how come 911 volumes for a given incident not increased at a rate equal to the increase in the number of people with phones in any emergency situation?

Apathy. In a group situation, if there is no moral or assigned direct responsibility to react, more and more people assume someone else will.


While not directly related, it is an interesting piece. Many of us use Firefox, yet how many of us have to tools and skills to actually look at the source for weaknesses, code quality and vulnerabilities? We assume that someone else will do this. We are using the same logic that people on the highway use when they don't call 911 for an accident - we start to believe that in the large group of people passing by, someone else will call. In the same way, we assume that because the FOSS code is available, some one will look, that whomever looks will be competent, and such review by someone else, or the fear of it, will be sufficient to keep our code safe. For many, they are waiting until someone tells them that there is a code problem before they react, and at that point, they only react as they are instructed.

FOSS is built around independance and community participation. We need more committers and less leechers in order for FOSS to work. Freedom in any form comes at a price. I believe that we as a community should be willing to pay the price of responsibility for the quality and security of FOSS in exchange for its availability.

The community does not work if no responsibility and commitment is tied to the process. We can look at the code, but do we? For qualitative community review to work, there must be a membership of committed users, responsible either due to their kind nature or because their employer told them to be. This community with a collective responsibility for the health and quality of open source software will work to improve FOSS because they have a vested interest in doing so, and because they understand the reality that if they don't do it, nobody will.


stormy's picture

I think it's unrealistic,

I think it's unrealistic, and not necessary, to expect everyone to look at code. You need lots of users, a percentage of them to submit bugs, and developers to fix them.

I always call 911 when I see an accident. Most of the time I'm told the cops already know. But I still call because sometimes I'm first, and that's important. There will always be users that submit bugs even though others may have submitted them and even though they don't know how to fix them. They are part of the community. They don't have to look at code to support open source software and freedom.

ernest.park's picture

Freedom, and quality, come at a price

While some still go for the cell at the sign of a problem, 911 calls relative to the increased availability of cell phones per incident observer has gone down. This means that while people still call, proportionately, less people are calling relative to those that can. This is apathy.

There is a restaurant in my town that sells hot dogs really cheap - free relative to the cost of a hot dog and the fuel to cook it. A soda costs more than the dog. In the summer, there are lines out the door. Is the food good? No. Is it likely to make you sick? Probably. Still, the crowds come. 

I don't expect everyone to look at code, but I would like to have the open source community accept more public and centralized methods of code review processes, and have people qualified to look at code accept that responsibility to document their results on a regular basis. This is work that is being done every day. I propose that we centralize the results and conform them to standards, perhaps along the lines of the Common Weakness Enumeration ( http://cwe.mitre.org )


We need to transparently monitor the process of qualitative review.  I am not suggesting that you to do any more than you normally would. If you feel that your role is to report bugs, continue to do so. However, there are people that privately or on behalf of corporations are paid to do analysis and testing of code. Some of this work is done on commercial software, in secrecy. Right now, it seems that much of the work for OSS is also secret, disparate and disjoined. There are respected templates for testing and validating software for qualitative and structural measures. I suggest that the work already being done by those qualified to do so follow a community supervised regimen. 

Ronald Reagan

Farewell Address to the Nation, Oval Office, January 11, 1989

"If they persist, pull the plug. It's still trust but verify. It's still play, but cut the cards. It's still watch closely. And don't be afraid to see what you see."


andrew's picture

Quality, freedom and time to fix, compared with proprietary s/w

I agree that the Debian OpenSSL incident was unfortunate, and that, as always, it is likely that more could have been done to prevent this occurring. However, taking vulnerabilities as an example, has anyone done a comparison, of say Debian, with proprietary operating systems? And not just the number of incidents, but the average time to fix also. And how often are such major vulnerabilities discovered on a wider scale, in FOSS compared with proprietary software?

My gut feeling, is that whilst FOSS might not always have the formal methods of quality assurance in place that you speak of, its large user base combined with open nature and passionate peer-reviewed development processes, lead to the production of code that is of a similar or perhaps even better quality than much that is proprietary in nature. However, I'm not suggesting that we all become apathetic to quality in FOSS and just hope someone else takes care of this. And would suggest that more research is needed, whilst agreeing that there is likely more that can be be done. That said, I would be wary of trying to fit frameworks used in connection with proprietary software to FOSS development.