An Empirical Study of Metric-based Comparisons of Software Libraries

Reviewed by Samridhi Vaid / 2021-11-26
Keywords: Development Practices, Metrics

Selecting the best software libraries for a task is a crucial step in programming, but often difficult and confusing because there are gazillions of Q&A websites with conflicting or out-of-date advice. These websites do not provide comprehensive descriptions of libraries' features and usually do not provide comparative data on different libraries.

Lopez2018 tackled this problem by implementing and assessing a metric-based comparison of software libraries. They calculated nine software metrics for 34 open-source Java libraries belonging to ten popular domains, such as testing, databases, and cryptography. They then surveyed 61 participants of varied backgrounds to determine how useful the metric-based comparison was. They found that:

  • Developers found a metric-based comparison of libraries useful.
  • Performance, Popularity, and Security were the most popular metrics when choosing a libraries.
  • Metrics related to the documentation and usability of a library were highly desired.

While there are good reasons to be skeptical about software metrics in general (see this post for discussion), Lopez2018 used data extracted from multiple sources (open source repositories, issue-tracking systems, and Stack Overflow) and combined several metrics for comparing the libraries, which makes their approach and findings more useful than earlier work in this area.

Lopez2018 Fernando López de la Mora and Sarah Nadi: "An Empirical Study of Metric-based Comparisons of Software Libraries". Proc. International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE), 2018, 10.1145/3273934.3273937.

Software libraries provide a set of reusable functionality, which helps developers write code in a systematic and timely manner. However, selecting the appropriate library to use is often not a trivial task. AIMS: In this paper, we investigate the usefulness of software metrics in helping developers choose libraries.Different developers care about different aspects of a library and two developers looking for a library in a given domain may not necessarily choose the same library. Thus, instead of directly recommending a library to use, we provide developers with a metric-based comparison of libraries in the same domain to empower them with the information they need to make an informed decision. METHOD:We use software data analytics from several sources of information to create quantifiable metric-based comparisons of software libraries. For evaluation, we select 34 open-source Java libraries from 10 popular domains and extract nine metrics related to these libraries. We then conduct a survey of 61 developers to evaluate whether our proposed metric-based comparison is useful, and to understand which metrics developers care about. RESULTS: Our results show that developers find that the proposed technique provides useful information when selecting libraries. We observe that developers care the most about metrics related to the popularity, security, and performance of libraries. We also find that the usefulness of some metrics may vary according to the domain. CONCLUSIONS:Our survey results showed that our proposed technique is useful.We are currently building a public website for metric-based library comparisons, while incorporating the feedback we obtained from our survey participants.