Introduction

“If it is not in the TCMS then we do not do it”, and the converse: “If we do it then it is in the TCMS”.

The above motto has been the paradigm driving the development of Kiwi TCMS. The development team has created a canonical source of information about testing that is useful to both managers and QE Associates.

Kiwi TCMS provides:

  • Managers with a source of information on planning, cases and execution status.
  • Reproducibility across planning, cases and execution.
  • Audit traceability.
  • Increased productivity - Associates are able to identify and work on gaps in product coverage.

Data organization within Kiwi TCMS

In order to successfully model your testing workflow with Kiwi TCMS you need to understand how internal data structures relate to one another. The image below represents a high level overview. More information can be found in section Database Schema.

Models relations

A Test Plan is a high-level container object which is used to describe testing activities. The most important scalar properties are the document text and the product which is being tested. The most important aggregate properties are the list of test cases and test runs (aka test executions).

A Test Case describes how to perform a specific scenario. This object is used to record scenarios in the database.

For every distinct product build that is being tested you have to create a Test Run artifact. The Test Run holds a collection of Test Case Run objects, information about the testing environment (via tags), who the default tester is, start/finish timestamps, etc.

A Test Case Run object is the container that links together the actual testing scenario with its execution status, additional comments and bugs found during testing! A Test Case Run object has a 1-to-1 relation to the Test Case for which it holds status. Thus a Test Case Run object represents the actual Test Case that will be/was executed against a particular product build and environment combination!

Explanation of additional data entities

In order to fully understand how Kiwi TCMS works and how it can be tailored to various testing workflows you need to be familiar with the different data entities within Kiwi. For more information please read Explanation of editable entities.

Testing workflow example

Based on the information above we propose the following way to organize testing activities within your team:

  1. Create Test Plan for individual areas you’d like to test, for example
performance testing or cross-browser testing;
  1. Add all necessary Test Cases to your Test Plan;
  2. When a new product build is available:
    • Create Test Run for that build including the Test Cases that need to be executed.
    • If you don’t want to execute all Test Cases you can leave their status as IDLE or remove them from the Test Run!
    • In case you test in multiple environments then create a Test Run for each environment/build combination. Annotate the Test Run with tags to be able to search for it later!
    • QA engineers will execute the Test Runs and update statuses as appropriate.

Question: How do I show all executions for a given Test Case?

Answer: Open the Test Case view, click Case Runs tab and expand the widget. All executions from all Test Plans are shown.

Question: When did a Test Case started to fail?

Answer: Navigate to the page which shows all executions of that test case as instructed above. Find the failing result. Here you have timestamp information and link to the Test Run (with possible product build and environment) which recorded the failure.

Question: How do I show all test executions for a particular Product/Plan/Build/Environment ?

Answer: Navigate to Search::Search Test Runs and filter Test Runs by desired properties. If you want to filter runs from a particular Test Plan make sure to specify the plan name in the Plan field. You can also filter Test Runs by tags if necessary.

Question: Is this the first time that Test Case is failing on envrionment X/Y/Z ?

Answer 1: Open Case Runs tab for that particular Test Case and examine the results by hand.

Answer 2: Perform the search described above and use the product version and environment information to answer this question. This is the preferred way to do this!

Answer 3: Go to REPORTING -> Testing report, select product/versions and select the By Plan & Build Per Plan View option. You can see your Test Plans with Runs grouped by builds and an execution status table on the right of each Test Run. WARNING: This is not recommended because the reporting views query the entire database and are very inefficient at the moment!

Getting to know the TCMS home page

The home page is the principal section of the TCMS. It provides quick access to assigned Test Runs and Test Plans. This section explains the features accessible from the home page.

The TCMS home page

  1. Navigation bar
  2. Help menu
  3. Personal menu
  4. Your Test Run dashboard
  5. Your Test Plan dashboard

Navigation bar

The navigation bar consists of the Main menu, Help menu and Personal menu. When you click over a menu item, a sub-menu appears. To navigate to a previous screen, click on the breadcrumb located below the menu bar.

The TCMS menu bar and breadcrumbs

Personal dashboard

The home screen contains a list of Test Runs associated with the user and a graphical display showing their completion status. To access a Test Run, click on its name.

The home screen also contains a list of Test Plans associated with the user. To access a Test Plan, click on its name.

Help menu

The Help menu provides additional links to help pages:

  • Report an issue with Kiwi TCMS
  • User guide
  • Administration guide
  • XML-RPC service API document
  • Kiwi TCMS version

Note

The items in the Help menu are configurable via the HELP_MENU_ITEMS setting in product.py.