11. Quality Model Configuration¶
As described in Quality Concept, MQC handles the imported data in such a way that quality can be structured and computed out of it. The configuration of the quality model including the definition of measurement functions to calculate measures for quality properties as well as the definition of base measures and derived measures have to be done in Excel. You can use the Quality Model Excel file provided by MQC as base for your configuration.
The quality model has to be imported (open the Load
button) and the
chosen configuration will be applied by MQC.
If you do not have a proper Quality Model Excel file that can be adapted, you
can create one with the help of MQC. From the Export
button to export the quality model currently used
by MQC (which is the Default Quality Model, if no other quality model was
imported before). This provides you an Excel file that can be used as template
and where you can define your own project’s quality model.
If your imported quality model is changed, you can easily reload an adapted
quality model via the Reload
button. Hovering over the Reload
button
provides a tooltip, which helps you detecting name and filepath of the
currently imported Quality Model file.
The following sections provide information on how to configure the quality model and the intended quality structures in MQC.
11.1. Base Measures and Default Values¶
The “Base Measure” sheet of the quality model is used to define all base measures that you expect to be imported from a certain data source.
As described later in Derived Measures and Quality Properties, these base measures are used to calculate other measures, so-called derived measures, and – most important – quality measures, which are the base to compute quality.
For configuring base measures, MQC expects the following structure:
data source name
(default) measurement name
base measure name
variable name
default value
The data source is the tool, which is used to provide data to be imported into MQC. Examples are MXAM and MTest.
Typically, a measurement means the set of operations done to determine the value for a measure. For example this can be the name of a specific guideline document, which is used for the static analysis of a model, or the kind of test environment like MiL, SiL or PiL.
The measurement name configured in the “Base Measure” sheet is used as default, if a data source does not provide a measurement name. If a data file contains a measurement name, the default is ignored and MQC uses the name provided by the data source.
Additionally, the “Base Measure” sheet provides you the possibility to define a default value per base measure variable. Please remember that base measures of the same type may be grouped (refer to Measures and Measurements), which results in a base measure containing multiple variable-value pairs.
Without default values, in case there are values missing for variables inside such a base measure group for a certain revision, a measurement function using these missing base measure variables could not be calculated. Therefore, MQC uses a configured default in case the data source has provided values for some base measure variables of a base measure group but not for all.
As an example, please imagine the base measure group FindingCount
, which
contains the variables Canceled
, Aborted
, Review
, Failed
,
Unrepaired
, Warning
, Info
, Repaired
, Passed
and
Ignored
. If for a certain revision the data source does not provide values
for e.g. Repaired
and Passed
, MQC uses the default values for the
related base measure variables as defined in the “Base Measure” sheet (see
Figure 11.1). Hence, a
corresponding measurement function can be calculated.
Note
Please note, that in the “Base Measure” sheet of the quality model all expected base measure variables have to be defined. This is to assure that missing values can be substituted with the default values.
In addition, MQC uses default values only if a base measure variable inside a base measure group is missing! If the complete base measure is missing for a revision, no default values will be used!
In that case, a base measure is completely missing in the current revision but values exist for any of the previous revisions, data propagation may be used to gain a full set of available information.
11.2. Derived Measures¶
Derived measures are computed either from base measures or from other derived measures as explained in Derived Measures.
This may be used to shorten the definition of measurement functions, which compute quality measures for quality properties (see Quality Properties).
If you would like to configure derived measures, this can be done using the “Derived Measure” sheet of the Quality Model Excel file.
A fullqualified derived measure name consists of four parts (same structure as for base measures):
data source name
measurement name
derived measure name
variable name
With that you are able to define groups of derived measures, each group containing multiple variables. The configuration of data source name and measurement name is optional. If nothing is set here, MQC takes data source and measurement from the base and/or derived measures used to calculate the derived measure.
Figure 11.2 shows the definition
of several derived measures that are computed from various variables of the
base measures FindingCount
and GuidelineCount
provided by the data
source MXAM. For example, the derived measure GuidelinesCalc.Passed
is
calculated by the following measure function:
[GuidelineCount.Passed] + log(1 + [FindingCount.Passed], 2)
Note
As shown in the above example, base and derived measures used in a measure
function must be set in square brackets [
and ]
!
You can define measure functions using a short notation for measure names only consisting of base / derived measure name and variable name (see Figure 11.2). MQC then checks for matching measures that were imported and replaces the used measure names by the fullqualified names.
If the quality model is exported (choose the Export
button from the
dialog), the “Derived Measure” sheet of the
resulting Excle file contains the fullqualified names as shown in
Figure 11.3).
If a measure function contains a base measure that exists for different
measurements, for example if FindingCount.Passed
was measured multiple
times by checking compliance according to different guideline documents, MQC
calculates the corresponding derived measure for each of those measurements.
The same applies to measure functions containing derived measures calculated
for multiple measurements.
Note
If a derived measure is calculated for multiple measurements, the configured data source and measurement names will be replaced by data source and measurement from the base and/or derived measures used to calculate the derived measure.
To calculate a derived measure for a selected measurement only, please, use fullqualified measure names when defining the measure function (as shown in Figure 11.3).
11.3. Quality Properties¶
The “Quality Model” sheet is used to define the lowest level of computable quality namely quality properties. For each quality property a measurement function has to be defined that is used to calculate a quality measure between 0 and 1 by using base and derived measures.
Derived measures (see Derived Measures) can be used instead of base measures to simplify quality computation, hence, to shorten the formula of a particular measurement function.
Figure 11.4 shows an example
for a measurement function, where all base measure variables for the base
measures FindingCount
and GuidelineCount
were replaced by the
corresponding derived measures.
Note
As shown in the above example, base and derived measures used in a
measurement function must be set in square brackets [
and ]
!
As for measure functions to calculate derived measures, you can define quality property measurement functions using a short notation for measure names consisting of base / derived measure name and variable name. MQC then checks for matching measures that were imported respectively calculated and replaces the used measure names by the fullqualified names.
If a measurement function uses a base or derived measure that exists for
different measurements, for example if GuidelinesCalc.Passed
was calculated
multiple times, MQC calculates the corresponding quality property for each of
the measurements.
Note
If a quality property is calculated for multiple measurements, the configured
quality property name will be extended by the name of the measurement, e.g.
Guideline Compliance_GuidelineAnalysis
, whereas Guideline Compliance
is the defined quality property name and GuidelineAnalysis
is the name
of the measurement.
To calculate a quality property for a selected measurement only, please, use fullqualified measure names when defining the measurement function.
Additionally as explained in Quality Model Structure, quality in MQC is aggregated on different levels (as shown in Figure 6.4). MQC allows to assign quality properties with their measurement functions to sub-characteristics. Those sub-chracteristics again can be assigned to characteristics (see Figure 11.5). This enables the user to group quality properties in order to define different quality aspects, which may be analysed separately.
The following sunburst visualization (Figure 11.6, lower right) illustrates the computation of the configured quality model.
The outer ring of the sunburst consists of all quality properties defined by the quality model. Please note, each quality property is already aggregated over the artifacts (see Figure 6.6 in section Quality Aggregation).
The next ring shows the sub-characteristics defined by the quality model. A sub-characteristic is calculated using the average of all quality properties assigned to that sub-characteristic.
The third ring of the sunburst chart shows the configured characteristics. Each characteristic is calculated using the average of all quality properties assigned to all sub-characteristics belonging to that particular characteristic.
Finally, the inner circle shows the overall quality.
Note
Per default, MQC uses all quality properties to compute the separate levels of the quality model, even if the measure value for a certain quality property cannot be calculated. This may be the case if all or a few measures, which are used to compute a quality property measurement function, are not available within a certain revision, i.e. not yet measured.
Missing quality properties always are treated as 0 when calculating the overall quality of a project.
MQC allows to exclude missing quality properties for the calculation. To
achive this, open the Exclude Empty Values
check box.
This may be used to compute quality only for those parts of a project,
which are already available, respectively to compute quality only based on
those measures already provided.
In the example shown in Figure 11.6 all quality properties defined by the quality model are treated equally, means they have the same weight when computing the overall quality of a project. Nevertheless, MQC allows to give some quality properties a higher importance than others, which is explained in the following section (Quality Property Weights).
11.4. Quality Property Weights¶
MQC supports to define the relative importance of quality properties. In this way the weight of the process steps is mapped into the quality model. Herewith the result of the quality calculation is representing the process more realistically, similar to the definition of Artifact Weights.
In our last example (see Figure 11.6) all defined quality properties had the same weight of 1. Now we use different weights per quality property (Figure 11.7):
After replacing the quality model (via the Load
button in the Configuration section of the menu), the sunburst visualization
adapts as well as the quality measures calculated for
sub-characteristics, characteristics and the overall quality of the project.
The size of each quality property tile (outer ring of the sunburst visualization) adjusts according to the weight of the quality property configured in the quality model.
Each level of the quality model now uses the weighted average for quality computation.
Finally, Figure 11.8 shows a different overall quality (inner circle of the sunburst chart) compared to the example with equal weights as shown in Figure 11.6.
11.5. Context Categories¶
By using context categories you can define, which data (on the level of base measures) is expected for a certain artifact. In that way, data never provided and therefore normally stated as “missing” - and probably resulting in bad quality - is treated as “excluded” data. This means it is ignored in availability and quality calculation.
Configuration of context categories is done in two steps:
assigning data to (or explicitly removing from) a context category is done via the “Context Category” sheet within the quality model configuration file, because this is directly related to the base measures defined here
assigning context categories to artifacts is done in the project structure configuration file, where the expected artifacts are defined
For a detailed explanation how to configure context categories please refer to chapter Context Categories.
11.6. Quality Model Kit¶
To help you defining a proper quality model, MQC provides predefined
tool-specific quality property measurement functions as part of the
installation under the following path:
C:\Users\...\AppData\Local\TIBCO\Spotfire\10.3.3\Modules\MES.MQC.CoreExtension_X.X.X.X\Resources\QualityModel\
.
This folder contains Quality Model suggestions for all Data Sources supported
by MQC. You will find examples, which allow you to easily assemble your own
project-specific quality model (including expected base and derived measures,
as well as proposed measurement functions) depending on the data sources you
are using within your project.
The following sections will give you examples based on MES data sources, which may help you to define your own measurement functions.
11.7. Quality Model for MES data sources¶
11.7.1. Quality Model for MXAM¶
In section Quality Properties there were already examples shown, how to define derived measures and quality properties, as well as a measurement function using measures provided by MXAM.
The defined derived measures are calculated using both, guideline results
(GuidelineCount
) as well as check findings (FindingCount
). Because of
the fact that the amount of check results typically is much higher than the
number of the guideline results, we use a log
function to reduce the impact
of the check findings.
GuidelinesCalc.Passed = [GuidelineCount.Passed] + log(1 + [FindingCount.Passed], 2)
The MXAM quality model contains derived measures for all possible results, i.e.
Failed
, Info
, Warning
and so forth.
Those derived measures are used in the measurement function for the quality
property Guideline Compliance
, which is defined as follows:
(1.0 * ([GuidelinesCalc.Passed] + [GuidelinesCalc.Info] + [GuidelinesCalc.Repaired]) +
0.8 * [GuidelinesCalc.Warning] +
0.5 * [GuidelinesCalc.Unrepaired] +
0.2 * [GuidelinesCalc.Failed] +
0.1 * [GuidelinesCalc.Review] +
0.0 * [GuidelinesCalc.AbortedCanceled]) /
([GuidelinesCalc.Passed] + [GuidelinesCalc.Info] + [GuidelinesCalc.Repaired] +
[GuidelinesCalc.Warning] + [GuidelinesCalc.Unrepaired] + [GuidelinesCalc.Failed] +
[GuidelinesCalc.Review] + [GuidelinesCalc.AbortedCanceled])
As you can see, the measurement function calculates the average of all findings, whereas each finding gets an individual weight.
There are two additonal MXAM derived measures defined.
IssueCount.Value
which is the sum of all “bad” findings, i.e.
FindingCount.Canceled
FindingCount.Aborted
FindingCount.Review
FindingCount.Failed
FindingCount.Unrepaired
FindingCount.Warning
SevereIssueCount.Value
which is the sum of all “bad” findings not
considering FindingCount.Warning
.
Note
Do not remove IssueCount.Value
and SevereIssueCount.Value
! The
calculation of those values is needed for correct displaying the MXAM tool
page (see MXAM Tool Page).
11.7.2. Quality Model for M-XRAY¶
The MXRAY quality model contains quality properties based on base measures provided by the data source M-XRAY.
As an example, the measurement function for Local Complexity
is defined as
follows:
(1.0 * [Local Complexity.Good] +
0.8 * [Local Complexity.Acceptable] +
0.2 * [Local Complexity.Bad]) /
([Local Complexity.Good] + [Local Complexity.Acceptable] + [Local Complexity.Bad])
The same pattern may be used for Modelling Depth
, Level
, Inports
,
Outports
as well as for %Elementary Inputs Unused (globally)
.
11.7.3. Quality Model for MTest¶
The MTest quality properties are defined by comparing current and expected
results, which leads to quite simple measurement functions, e.g. the quality
property Requirements with Reviewed Testability
is calculated as follows:
[Requirements with Reviewed Testability.Reached] / [Requirements with Reviewed Testability.Total]
Following this pattern, the MTest quality model additionally defines quality properties for:
Requirements Compliance
Testable Requirements with Assessments
Testable Requirements with Test Sequences
Test Sequence Work Progress
Reviewed Test Sequences
Test Sequence Compliance
Assessment Work Progress
Assessments Compliance
andReviewed Assessments
.
11.8. Measurement function definition¶
This section aims to give you an overview on how to define your own quality properties or adapt your measurement functions.
Therefore you can use a bunch of mathematical functions provided within MQC.
Apart from the four basic arithmetic operations (+
, -
, *
and
/
), MQC allows to use: Abs
, Exp
, Ceiling
, Floor
, Round
,
Ln
, Log
, Mod
, Power
, Product
, Sqrt
, Min
, Max
,
Avg
, WeightedAverage
, Case... when
and IF... else
.
In the following some of these functions are explained with their possible usage within a measurement function.
We already have seen
- Log (n,2)
:
Given the case you only have at your disposal certain base measure values, for that your quality property calculation tends to converge to 1 using basic arithmetic operations. Here you might consider to use Log function with basis 2 (please see the derived measure calculation in the example above Quality Model for MXAM.)
Given the case that you want to avoid negative values in your measurement function, you can handle this at any time of its calculation using the
Abs
-function, that returns the absolute value of the argument.Abs([GuidelinesPassedCalc] - [GuidelinesInfoCalc])
or
Case... when
: case when ([GuidelinesPassedCalc] - [GuidelinesInfoCalc] < 0) then 1 when ([GuidelinesPassedCalc] - [GuidelinesInfoCalc] > 0) then ([GuidelinesPassedCalc] - [GuidelinesInfoCalc]) else 0 endPower
Returns Arg1 raised to the Arg2 power. Power(Arg1, Arg2)Exp
Another useful application of mathematical functions could be the case of using theExp (argument)
function, which returns e raised to the power of the argument. A linear function, such as1.0 - ([ModelClones.Clones] / [ModelClones.Subsystems])
might not be feasible to accurately adapt the Quality Property of ModelClones to your needs. Therefore, you might want to use this function:
1.0 / Exp( 2 * [ModelClones.Clones] / [ModelClones.Subsystems])
Ceiling
Floor
Round
Ln
Log
Mod
Power
Product
Sqrt
Min
Max
Avg
This function calculates the average (mean) value of the input parameters: Avg(parameter 1, parameter 2, …, parameter n)WeightedAverage
Case... when
IF... else