From the Blogosphere
Why Are There So Many Hurdles to Efficient SAM Benchmarking?
When dealing with Software Analysis and Measurement benchmarking, people's behavior generally falls into two categories
By: Lev Lesokhin
Jul. 9, 2013 04:40 PM
Two opposite sides
As often, there is no sensible middle ground.
All of the above reasons make perfect sense and one has to be well aware of such situations. However, this is no ground for dismissing any possibility of comparison altogether.
How so? Because the Metrics required to answer the Question one has to ask themselves to measure the level of achievement of each Goal can differ from one technology stack to another, or evolve from one release of the measurement platform to another without invalidating the results. This is also true about the number and nature of the Questions one has to ask themselves to measure the level of achievement of each Goal.
Then, with a measurement model using a compliance ratio to consolidate and aggregate (as opposed to raw count of non-conformity), this statistical processing acts as a built-in clutch.
From one technology stack to another
At the Goal level, even if the number of contributing Questions differ: the difference is inherent to the technology and is therefore normal and acceptable.
For example, there is no possibility to over-complexify COBOL code with coding and architectural practices related to Object-oriented capabilities, while there are many ways with JEE. This comparison might seem unfair to JEE that is assessed with more rules and technical criteria than COBOL, but this is inherent to their respective capabilities and a fair assessment of transferability or changeability risk must take this difference into account.
As this assessment result will guide a decision for resource allocation, it is critical to know that in this JEE component, there is not only some regular algorithmic or SQL or XYZ complexity, but there is an additional load of complexity due to excessive polymorphism or XYZ.
From one release to another
Why not rely on a raw count of violations?
From one context to another
To expect fairness may seem naïve, but a lot of the measurement process already relies on some amount of fairness. For example, what are the true boundaries of the application you're measuring? Nowadays, with cross-app principles, the definition is blurred and it could be easy to omit part of the application to hide some facts from management.
One could also define a target Architecture that would work in their best interest, but that would mean entering into the system some flawed configuration data that anyone can review.
One could vet the libraries to prevent security-related vulnerabilities, improving assessment results, but that would also mean entering some flawed configuration data.
At the end of the day
Although, this is no reason to dismiss benchmarking altogether. Not hiding the differences and their impact is important though.
As Dr. Bill Curtis stated during the CISQ (http://www.it-cisq.org/) Seminar in Berlin on June 19th when explaining the key factors to conduct a clever productivity analysis - always inspect data, investigate outliers and extreme values, and always question the results too.
In other words, always use your brain.
Latest AJAXWorld RIA Stories
Subscribe to the World's Most Powerful Newsletters
Subscribe to Our Rss Feeds & Get Your SYS-CON News Live!
SYS-CON Featured Whitepapers
Most Read This Week