Three Empirical Evaluations of UML
Reviewed by Greg Wilson / 2011-08-17
Keywords: Modeling
Like a lot of people (particularly those with engineering backgrounds), I was very excited when UML first appeared in the mid-1990s. "At last," I thought, "Programmers will have something like blueprints and circuit diagrams to help them design software." I knew they'd be better than flowcharts: after all, they were object-oriented, which back then was pretty much a synonym for "good". But, like a lot of people, I read about UML more than I ever actually used it. It never seemed to help me analyze things the way circuit diagrams had when I was an undergrad, and it wasn't much use as a communication tool either: every diagram I drew had to be accompanied by several paragraphs of explanatory text, and my colleagues seemed to be able to understand the latter perfectly well when accompanied by a few freehand block-and-arrow sketches.
But anecdotes don't prove anything—to do that, you need to look at many people, in many contexts, over an extended period. User experience designers have been doing this for years; one of the most hopeful developments in empirical software engineering is the way that people are using those techniques to find out where and when UML is actually useful, and, where it's not, why not. Three fairly recent studies of this kind are cited below—we look forward to many more, and to a next generation of software design notations based on their findings.
Dzidek2008 W.J. Dzidek, E. Arisholm, and L.C. Briand: "A Realistic Empirical Evaluation of the Costs and Benefits of UML in Software Maintenance". IEEE Transactions on Software Engineering, 34(3), 2008, 10.1109/tse.2008.15.
The Unified Modeling Language (UML) is the de facto standard for object-oriented software analysis and design modeling. However, few empirical studies exist which investigate the costs and evaluate the benefits of using UML in realistic contexts. Such studies are needed so that the software industry can make informed decisions regarding the extent to which they should adopt UML in their development practices. This is the first controlled experiment that investigates the costs of maintaining and the benefits of using UML documentation during the maintenance and evolution of a real nontrivial system, using professional developers as subjects, working with a state-of-the-art UML tool during an extended period of time. The subjects in the control group had no UML documentation. In this experiment, the subjects in the UML group had, on average, a practically and statistically significant 54 percent increase in the functional correctness of changes (ρ = 0.03) and an insignificant 7 percent overall improvement in design quality (ρ = 0.22), though a much larger improvement was observed on the first change task (56 percent), at the expense of an insignificant 14 percent increase in development time caused by the overhead of updating the UML documentation (ρ = 0.35).
DeLucia2009 Andrea De Lucia, Carmine Gravino, Rocco Oliveto, and Genoveffa Tortora: "An experimental comparison of ER and UML class diagrams for data modelling". Empirical Software Engineering, 15(5), 2009, 10.1007/s10664-009-9127-7.
We present the results of three sets of controlled experiments aimed at analysing whether UML class diagrams are more comprehensible than ER diagrams during data models maintenance. In particular, we considered the support given by the two notations in the comprehension and interpretation of data models, comprehension of the change to perform to meet a change request, and detection of defects contained in a data model. The experiments involved university students with different levels of ability and experience. The results demonstrate that using UML class diagrams subjects achieved better comprehension levels. With regard to the support given by the two notations during maintenance activities the results demonstrate that the two notations give the same support, while in general UML class diagrams provide a better support with respect to ER diagrams during verification activities.
CruzLemus2009 José A. Cruz-Lemus, Marcela Genero, M. Esperanza Manso, Sandro Morasca, and Mario Piattini: "Assessing the understandability of UML statechart diagrams with composite states—A family of empirical studies". Empirical Software Engineering, 14(6), 2009, 10.1007/s10664-009-9106-z.
The main goal of this work is to present a family of empirical studies that we have carried out to investigate whether the use of composite states may improve the understandability of UML statechart diagrams derived from class diagrams. Our hypotheses derive from conventional wisdom, which says that hierarchical modeling mechanisms are helpful in mastering the complexity of a software system. In our research, we have carried out three empirical studies, consisting of five experiments in total. The studies differed somewhat as regards the size of the UML statechart models, though their size and the complexity of the models were chosen so that they could be analyzed by the subjects within a limited time period. The studies also differed with respect to the type of subjects (students vs. professionals), the familiarity of the subjects with the domains of the diagrams, and other factors. To integrate the results obtained from each of the five experiments, we performed a meta-analysis study which allowed us to take into account the differences between studies and to obtain the overall effect that the use of composite states has on the understandability of UML statechart diagrams throughout all the experiments. The results obtained are not completely conclusive. They cast doubts on the usefulness of composite states for a better understanding and memorizing of UML statechart diagrams. Composite states seem only to be helpful for acquiring knowledge from the diagrams. At any rate, it should be noted that these results are affected by the previous experience of the subjects on modeling, as well as by the size and complexity of the UML statechart diagrams we used, so care should be taken when generalizing our results.