Measuring quality
in a complex software system
Measuring quality
in a complex software system
“Some things tend to degrade with time.”
Degrade de·grade verb /diˈgrād/
To deviate so as to no longer function as well or satisfy the same quality attributes.
“The buildup of harmful elements is usually inevitable.”
Inherent complexity in software, team dynamics, industry shifts, business pressure and so on.
Technical debt ∈ Harmful elements
Technical debt tech·ni·cal debt noun /ˈteknikəl det/
A debt that is incurred as a result of software engineering malpractices that must be paid at some point in the future in the form of time, effort or defects.
“Technical debt can seriously impact complexity, and hence quality.”
Dan Sturtevant, Technical Debt in Large Systems, http://sdm.mit.edu/news/news_articles/webinar_050613/sturtevant_050613.pdf
In many paradigms, modularity (low coupling, high cohesion)
is a desirable quality attribute.
How can we measure technical debt?
Five measures of technical debt
1. Lines of code
2. Cyclomatic complexity
3. Direct dependencies
4. Propagation cost
5. Core size
The Armory
1. Static analysis
Generates metrics from a system without having to execute it.
2. Network manipulation
Allows us to generate inter-component architectural metrics.
3. Design Structure Matrix (DSM)
Allows us to visualize structural complexity and hence quality.
The static analysis tool gives us three things.
Lines of code
Cyclomatic complexity
Inter-component dependencies
We then take those inter-component dependencies and build a DSM.
We then build a visibility DSM by finding our initial matrix's transitive closure.
Alan MacCormack et al, "Exploring the Structure of Complex Software Designs: An Empirical Study of Open Source and Proprietary Code", 2006.
gfx module in Firefox 20.
The fan-in and fan-out data gives us core size.
Now we have all five architectural metrics.
So we apply this technique to Firefox.
| Mean | Median | Stdev | Min | Max | |
| loc | 2,885,799 | 2,686,348 | 595,408 | 2,066,655 | 4,012,434 |
| files | 13,266 | 11,907 | 2,844 | 9,478 | 18,460 |
Descriptive statistics for the set of Firefox releases (n=23)
lines of code
cyclomatic complexity
direct dependencies
propagation cost
core size
21% of files in Firefox are highly interconnected.
Making any change to a randomly selected file can, on average, directly impact 10 files.
Making any change to a randomly selected file can indirectly impact over 2,300 files.
The switch to the rapid release cycle from version 5.0 onwards had a positive impact on quality.
| Mean | Median | Stdev | Min | Max | |
| loc | 4,767,974 | 5,157,958 | 1,951,894 | 896,204 | 7,331,584 |
| files | 29,020 | 31,132 | 12,349 | 5,009 | 46,620 |
Descriptive statistics for the set of Chromium releases (n=23)
| Mean | Median | Stdev | Min | Max | |
| loc | 2,885,799 | 2,686,348 | 595,408 | 2,066,655 | 4,012,434 |
| files | 13,266 | 11,907 | 2,844 | 9,478 | 18,460 |
Descriptive statistics for the set of Firefox releases (n=23)
Processing the Chromium data took much longer
On average, it was an order of magnitude longer per release.
A day vs a couple of hours (2.5GHz i7, 500GB SSD, 16GB memory).
lines of code
direct dependencies
propagation cost
core size
33% of files in Chromium are highly interconnected.
Making any change to a randomly selected file can, on average, directly impact 14 files.
Making any change to a randomly selected file can indirectly impact over 8,500 files.
Recall for Firefox
21% of files in Firefox are highly interconnected.
Making any change to a randomly selected file can, on average, directly impact 10 files.
Making any change to a randomly selected file can indirectly impact over 2,300 files.
Thank you. Hope this was useful.
alialmossawi
I had some free time on the plane...
This was an ill-conceived idea, meant to be used as an analogy.
March 14, 2013
-->