Measuring quality
in a complex software system

Ali Almossawi ·  ali@mozilla.com
Links, MIT Media Lab, July 2013

Some things tend to degrade with time.

Degrade de·grade verb /diˈgrād/  

To deviate so as to no longer function as well or satisfy the same quality attributes.

The buildup of harmful elements is usually inevitable.

Inherent complexity in software, team dynamics, industry shifts, business pressure and so on.

Technical debt  ∈  Harmful elements

Technical debt tech·ni·cal debt noun /ˈteknikəl det/  

A debt that is incurred as a result of software engineering malpractices that must be paid at some point in the future in the form of time, effort or defects.

Technical debt can seriously impact complexity, and hence quality.

Dan Sturtevant, Technical Debt in Large Systems, http://sdm.mit.edu/news/news_articles/webinar_050613/sturtevant_050613.pdf

In many paradigms, modularity (low coupling, high cohesion)
is a desirable quality attribute.

How can we measure technical debt?

Five measures of technical debt

1. Lines of code

2. Cyclomatic complexity

3. Direct dependencies

4. Propagation cost

5. Core size

The Armory

1. Static analysis

Generates metrics from a system without having to execute it.

2. Network manipulation

Allows us to generate inter-component architectural metrics.

3. Design Structure Matrix (DSM)

Allows us to visualize structural complexity and hence quality.

The static analysis tool gives us three things.

Lines of code

Cyclomatic complexity

Inter-component dependencies

We then take those inter-component dependencies and build a DSM.

We then build a visibility DSM by finding our initial matrix's transitive closure.

Alan MacCormack et al, "Exploring the Structure of Complex Software Designs: An Empirical Study of Open Source and Proprietary Code", 2006.

gfx module in Firefox 20.

The fan-in and fan-out data gives us core size.

Now we have all five architectural metrics.

So we apply this technique to Firefox.

Mean Median Stdev Min Max
loc 2,885,799 2,686,348 595,408 2,066,655 4,012,434
files 13,266 11,907 2,844 9,478 18,460

Descriptive statistics for the set of Firefox releases (n=23)

lines of code

cyclomatic complexity

direct dependencies

propagation cost

core size

21% of files in Firefox are highly interconnected.

Making any change to a randomly selected file can, on average, directly impact 10 files.

Making any change to a randomly selected file can indirectly impact over 2,300 files.

The switch to the rapid release cycle from version 5.0 onwards had a positive impact on quality.

Mean Median Stdev Min Max
loc 4,767,974 5,157,958 1,951,894 896,204 7,331,584
files 29,020 31,132 12,349 5,009 46,620

Descriptive statistics for the set of Chromium releases (n=23)

Mean Median Stdev Min Max
loc 2,885,799 2,686,348 595,408 2,066,655 4,012,434
files 13,266 11,907 2,844 9,478 18,460

Descriptive statistics for the set of Firefox releases (n=23)

Processing the Chromium data took much longer

On average, it was an order of magnitude longer per release.

A day vs a couple of hours (2.5GHz i7, 500GB SSD, 16GB memory).

lines of code

direct dependencies

propagation cost

core size

33% of files in Chromium are highly interconnected.

Making any change to a randomly selected file can, on average, directly impact 14 files.

Making any change to a randomly selected file can indirectly impact over 8,500 files.

Recall for Firefox

21% of files in Firefox are highly interconnected.

Making any change to a randomly selected file can, on average, directly impact 10 files.

Making any change to a randomly selected file can indirectly impact over 2,300 files.

Thank you. Hope this was useful.

alialmossawi

I had some free time on the plane...

This was an ill-conceived idea, meant to be used as an analogy.




March 14, 2013

-->