8. Data Sources

To import data from reports in MQC, the data has to be provided as files. These files can be located in the local file system, a network path or a git repository.

../_images/MQC_QuickStart_ImportFilesDirectories.png

Figure 8.1 Data sources in MQC. By clicking on “Add Data Source” a new data source can be configured.

A new data source can be added through a browser in the data source dialog. The type of the data source has to be selected first and then the respective directory, files or git repository can be added. You can also have multiple data sources added to your project.

../_images/MQC_DataSource_AddBrowser.png

Figure 8.2 The Browser to add a new Data Source

Each data source is constantly monitored for updates and changes. Whenever e.g. a new report appears in a data source (new file in directory or new commit in version control system), it will be detected by MQC. Either the user is notified about the changes or the project is automatically updated (refer to Refresh Data).

Note

Use the Force Refresh button (see Figure 8.1) to not just apply changes to a project but to re-import all data anew. This may be useful for instance in case of the commit history of a version control system has been changed.

8.1. Local File System / Network / Sample Projects

The local file system and network types function similarly. A directory, and therefore all subdirectories and files within, or a single file can be selected as the data source. At any point, the allowed file extensions that can be read by MQC are displayed below.

Navigating through the file system works like a windows explorer. Double-clicking a directory opens it. The current path is displayed at the top of the browser and can be used to navigate upwards in the file system. You can also add the path manually.

../_images/MQC_DataSource_Local.png

Figure 8.3 Local File System selected. Add/Edit the path manually.

../_images/MQC_DataSource_Network.png

Figure 8.4 Network selected. The path can be used to navigation upwards by click.

Note

The local file system of the server is not accessible when using the web player.

You have also direct access to the MQC Showcase source files under Sample Projects.

../_images/MQC_DataSource_SampleProject.png

Figure 8.5 Sample Project selected.

8.2. Version Control Systems

MQC also allows version control systems to be used as a data source. Currently only Git is supported.

8.2.1. Git (Beta)

After selecting Git as the data source type, all previously configured git repositories in the project are available for selection.

By clicking on “Manage repositories” git repositories can be added and already configured repositories can be deleted or modified.

Note

If git is not installed, locally (if using the MQC desktop client) or on the server (if using the web player), an error is shown.

Git for windows can be found here: https://git-scm.com/downloads

../_images/MQC_DataSource_Git.png

Figure 8.6 Git selected. No repositories are configured yet.

Git repositories can be added by ssh or https urls.

An rsa private key can be provided for authentication of ssh urls if it is not configured as the default ssh key for the current user.

For an https url the authentication has to be done by entering the username and password, if it is not already stored in the windows credential store.

When a repository is saved, it is validated. If the authentication fails, an error message is shown.

In case you have multiple repositories with a similar structure, you can configure them as one data source by adding a list of repository urls to the “Add new repositories” dialog and use one configuration (branches, commits, time range, …) for all repositories afterwards.

../_images/MQC_DataSource_GitRepositories.png

Figure 8.7 Git Repositories dialog to add, delete and modify git repository(ies) configurations for data sources.

You can select a branch of a configured git repository(ies). For further configurations of the data source, the selected branch of the repository(ies) should be checked out first.

Note

Only branches that exist in all repositories can be selected, when multiple Git Repositories are given.

The selected branch for a git repository(ies) can be manually checked out using the Fetch Repositories button.

../_images/MQC_DataSource_GITDialog.png

Figure 8.8 Git Dialog to select repository(ies), branch, commit filters and relevant File Paths for adding a new data source.

The selected branch can then be added as a data source so that it’s entire commit history can be used by MQC.

In case that multiple repositories are added, they are considered as one data source.

../_images/MQC_DataSource_MultipleGitRepos.png

Figure 8.9 Multiple Git repositories as one Data Source.

Since your commit history might contain commits not relevant for your MQC quality analysis, you have the possibility of filtering these commits in the Commit Filters section of this dialog.

../_images/MQC_DataSource_GITFilterTargets.png

Figure 8.10 Configuration options to add a commit filter.

The Commit Filters section provides three possible targets to filter, namely, Message, Author and Tag. You can further choose to either include or exclude the commits matching this filter by clicking on the toggle button below Apply. The green colored + stands for include and the red colored - stands for exclude. Next you need to add the suitable regular expression in RegEx text box and the filtered commits matching all the configured filters can be seen on the Filtered Commits panel at the right side. All the filters to the same target are connected with an OR whereas different targets are connected with an AND.

For example, the configured filters as seen in the image below would be interpreted as, All commits from the Author "Jenkins Slave" AND with the Message "Static Analysis".

If there is another filter for Message then it would be, All commits from the Author "Jenkins Slave" AND with (Message "Static Analysis" OR Message "Dynamic Analysis").

../_images/MQC_DataSource_GITFilterOR.png

Figure 8.11 Commit Filters with “AND” between different targets and “OR” between the same targets.

Additionally, MQC offers the possibility to filter by time. This allows to consider only commits of a specific time frame.

../_images/MQC_DataSource_GITFilterTimeFrame.png

Figure 8.12 Time Restriction Commit Filter to consider only commits that were added after a specific date.

Note

If no time filter is added, but a project structure configuration has been loaded, MQC automatically sets the project start and end dates as internal time restriction commit filter to limit the number of commits to be recognized especially for very large repositories.

In that case and if a commit appears outside the configured project time frame, you will be notified about such commits within the Notifications panel. After adapting the project structure configuration, a data refresh is necessary to fetch the relevant commits from the repository.

Once you have filtered the relevant commits, you might also want to select specific folders or files relevant to your analysis. MQC provides two possibilities to do this.

../_images/MQC_DataSource_GITFileRegEx.png

Figure 8.13 Selection menu to choose the folder or file path of relevant data by file system or add patterns to match relevant folder or file paths.

If you select the Regular Expressions option in the drop down, then you can add multiple regular expressions as done in the Commit Filters section.

../_images/MQC_DataSource_GITFileRegExIncludeExclude.png

Figure 8.14 Regular Expression pattern matching for selecting data source paths.

If you select Browser Selection option, one file or folder can be chosen as the data source, similar to the selection for local or network.

../_images/MQC_DataSource_GITSelectRoot.png

Figure 8.15 Browser Selection for choosing data source path.

Note

Browser Selection is available only when there is one git repository given. By multiple repositories only Regular Expressions could be applied.

Finally, with the checkbox shown in Figure 8.16 you can define which date and time is used as the report date-time:

  • Git commit date (checked)

  • Report creation date time as contained in the report itself (unchecked)

../_images/MQC_DataSource_GIT_CommitTime_ReportDate.png

Figure 8.16 Enable checkbox to use commit time as report date.

If a git data source has been added, a monitoring service periodically checks if a new commit is made on the remote. The default for the check interval is set to 10 seconds. In case of multiple repositories, the supervision timer is stepwise increased for performance reasons based on the number of repositories. I.e. If you have configured more than 100 repositories, the check happens only every 60 seconds.

When data is refreshed, the git repository is updated and all remote changes are fetched and applied to the MQC project.