Test Management and Traceability

This guide demonstrates how to integrate test reports into your documentation with full traceability between test specifications, test cases, source code, and test results.

Overview

A comprehensive testing approach includes:

• Test Specifications: Define what needs to be tested

• Test Cases: Actual test implementations in code

• Source Code: The implementation being tested

• Traceability: Links between all these elements

• Test Reports: Results from test execution

By using Sphinx-Needs for test specifications, sphinx-codelinks for code tracing, and sphinx-test-reports for test results, we create a complete traceable testing ecosystem.

Workflow

        graph LR
   REQ[Requirements] --> TS[Test Specifications]
   TS --> TC[Test Cases in Code]
   TC --> IMPL[Implementation Code]
   TC --> RUN[Test Execution]
   RUN --> REPORT[Test Reports]
   REPORT --> DOC[Documentation]

   style REQ fill:#FFB300
   style TS fill:#A6BDD7
   style TC fill:#A6BDD7
   style IMPL fill:#fa8638
   style REPORT fill:#4aac73
   style DOC fill:#F6768E
    

Step 1: Define Test Specifications

Create test specifications using the test-spec directive to define what needs to be tested:

Test Specification: Factorial Function Test Specification TS_FACTORIAL

status: open tags: factorial, math links incoming: T_FACT_001, T_FACT_002, T_FACT_003

Test the factorial function for: Negative numbers (should return 1) Zero (should return 1) Positive numbers (should return correct factorial) The factorial function is critical for mathematical operations and must handle edge cases correctly.

Test Specification: Prime Number Check Test Specification TS_PRIME

status: open tags: prime, math links incoming: T_PRIME_001, T_PRIME_002, T_PRIME_003

Test the IsPrime function for: Negative numbers (should return false) Trivial cases (0, 1, 2, 3) Positive numbers (both prime and composite) Prime number detection is used in cryptographic operations and must be accurate.

Step 2: Annotate Test Cases in Code

Add sphinx-codelinks annotations to your test cases. The annotation format is e.g.:

// @<description>, <need_id>, <need_type>
TEST(TestSuite, TestName) {
    // test implementation
}

Example from sample1_unittest.cpp:

Annotated test case with GTest properties

// @Test negative factorial values, T_FACT_001, test, [TS_FACTORIAL, IMPL_2]
TEST(FactorialTest, Negative) {
  // Link this test to documentation via GTest property
  RecordProperty("need_id", "T_FACT_001");
  RecordProperty("requirement", "REQ_MATH_001");
  RecordProperty("test_spec", "TS_FACTORIAL");
  
  // This test is named "Negative", and belongs to the "FactorialTest"
  // test case.
  EXPECT_EQ(1, Factorial(-5));
  EXPECT_EQ(1, Factorial(-1));
  EXPECT_GT(Factorial(-10), 0);

The annotation @Test negative factorial values, T_FACT_001, test creates a traceable link between the test code and the documentation.

GTest Properties (RecordProperty):

• need_id: Links the test execution result to the test case need (T_FACT_001)

• requirement: Links to the requirement being tested (REQ_MATH_001)

• test_spec: Links to the test specification (TS_FACTORIAL)

These properties are included in the XML test report and enable automatic linking between test results and documentation needs.

Step 3: Link Test Cases to Specifications

Test cases are automatically discovered from source code using sphinx-codelinks. The test files contain @ annotations that define test needs:

// @Test negative factorial values, T_FACT_001, test
TEST(FactorialTest, Negative) {
  RecordProperty("need_id", "T_FACT_001");
  RecordProperty("requirement", "REQ_MATH_001");
  RecordProperty("test_spec", "TS_FACTORIAL");
  // ... test implementation
}

These annotations are parsed by sphinx-codelinks and automatically create test needs. Below are all test cases discovered from the test files:

Test Case: Test negative factorial values T_FACT_001

remote-url: src/sample1_unittest.cpp#L76; links outgoing: TS_FACTORIAL, IMPL_2

Test Case: Test factorial of zero T_FACT_002

remote-url: src/sample1_unittest.cpp#L106; links outgoing: TS_FACTORIAL, IMPL_2

Test Case: Test factorial of positive numbers T_FACT_003

remote-url: src/sample1_unittest.cpp#L116; links outgoing: TS_FACTORIAL, IMPL_2

Test Case: Test prime check for negative values T_PRIME_001

remote-url: src/sample1_unittest.cpp#L131; links outgoing: TS_PRIME, IMPL_3

Test Case: Test prime check for trivial cases T_PRIME_002

remote-url: src/sample1_unittest.cpp#L145; links outgoing: TS_PRIME, IMPL_3

Test Case: Test prime check for positive values T_PRIME_003

remote-url: src/sample1_unittest.cpp#L158; links outgoing: TS_PRIME, IMPL_3

The test cases link to:

• Test specifications (e.g., TS_FACTORIAL) via the :spec: field

• Implementation (e.g., IMPL_2) via the :implements: field

• Requirements (e.g., REQ_MATH_001) via GTest properties

Step 4: Show Code Traceability

Use sphinx-codelinks to display where test cases are implemented:

.. src-trace:: Test Case Implementation
    :project: x-as-code-cpp
    :directory: src

This shows all annotated code locations, creating bidirectional links between documentation and source code.

Step 5: Integrate Test Reports

After running tests, integrate the test results using sphinx-test-reports:

# Build and run tests
cd src
cmake -S . -B build
cmake --build build
cd build
./eac_test --gtest_output=xml:test-results.xml

The test results XML file contains properties that link back to the test case needs, enabling complete traceability.

Traceability Matrix

Test Specifications to Test Cases

View which test cases implement each test specification:

ID	Title	Type	Status	Spec	Implements
DEBT_001	Technical Debt: Legacy API Endpoints	spec	open
T_FACT_001	Test negative factorial values	test
T_FACT_002	Test factorial of zero	test
T_FACT_003	Test factorial of positive numbers	test
T_PRIME_001	Test prime check for negative values	test
T_PRIME_002	Test prime check for trivial cases	test
T_PRIME_003	Test prime check for positive values	test
TS_FACTORIAL	Factorial Function Test Specification	test-spec	open
TS_PRIME	Prime Number Check Test Specification	test-spec	open

Test Cases to Implementation

View the connection between test cases and the code they test:

Coverage Analysis

To generate code coverage reports with line-level detail:

# Run tests with coverage
./scripts/test_with_coverage.sh

# View coverage report
open src/build/coverage_html/index.html

The coverage report shows:

• Line Coverage: Which lines of code were executed during tests

• Function Coverage: Which functions were called

• Branch Coverage: Which conditional branches were taken

CI/CD Integration

Integrate testing into your CI/CD pipeline:

# .github/workflows/test.yml
- name: Build C++ Project
  run: |
    cd src
    cmake -S . -B build
    cmake --build build

- name: Run Tests
  run: |
    cd src/build
    ./eac_test --gtest_output=xml:test-results.xml

- name: Upload Test Results
  uses: actions/upload-artifact@v4
  with:
    name: test-results
    path: src/build/test-results.xml

Complete Traceability Flow

The complete flow from requirements to test results:

This visualization shows:

• Requirements drive test specifications

• Test specifications lead to test cases

• Test cases verify implementations

• Everything is traceable and documented

Summary

By combining:

• Sphinx-Needs: For requirements and test specifications

• Sphinx-Codelinks: For linking documentation to source code

• Sphinx-Test-Reports: For integrating test execution results

You create a fully traceable testing ecosystem where:

✅ Every test links to its specification ✅ Every test links to the code it verifies ✅ Test results are automatically integrated ✅ Coverage is measured and reported ✅ Traceability is bidirectional and complete

Additional Resources

• GoogleTest Documentation

• Sphinx-Test-Reports

• Sphinx-Codelinks

• Code Coverage with lcov