Mainframe Test Automation Glossary and FAQ

Shift-left test automation concepts for COBOL and z/OS applications

Mainframe Test Automation Glossary and FAQ


Automated testing Using testing tools and scripts to automate the execution of test cases, making the testing process more efficient and repeatable; a core DevOps concept.
Assertion A statement or a piece of code that evaluates whether a particular condition or expectation about the behavior of the software is true, typically in the form of a boolean expression. Test4z includes an Assert API.
Black box testing Focuses on the functional requirements without considering the internal structure or implementation details; based solely on external interfaces, inputs, and outputs, treating the software as a "black box". Test4z supports black box testing.
Code coverage A metric used to measure the extent to which source code has been exercised by a test suite (as compared to the total lines, branches, statements, or conditions in the codebase); ensures critical functionalities and use cases are tested thoroughly, including edge cases and error conditions; one of the 6 Core Principles of unit testing.
Continuous testing (CT) Closely aligned with Continuous Integration (CI), CT is an approach to software testing that emphasizes running tests at every stage of the SDLC to ensure changes to the codebase do not introduce defects and that the application remains in a releasable state.
Functional testing Verifies the software functions as expected by testing its features against specified requirements; non-functional testing, by contrast, encompasses aspects other than specific behaviors of the system including performance, usability, reliability, and security testing.
Gray box testing Combines elements of black box testing and white box testing; partial understanding enables testers to create more targeted and efficient tests while still maintaining a level of independence; used when white box testing is not feasible due to time or resource constraints. Test4z supports gray box testing.
Integration testing A type of functional testing; verifies the interactions between different units, modules, or systems, ensuring they work together seamlessly; follows unit testing, where individual units (such as classes, functions, or modules) are tested in isolation; focuses on communication between components, data flow, error handling, and boundary conditions.
Isolation Testing a component, module, or system in isolation from its dependencies; keeps the focus on functionality without being distracted by other behaviors; mocking is a common technique to achieve isolation. One of the 6 Core Principles of unit testing.
Manual testing
Testers execute test cases by hand, interacting with the software application as an end user would, to identify defects, assess functionality, and ensure that the application meets specified requirements.
Mocks Objects that simulate the behavior of real objects in a controlled way to isolate the code being tested from its dependencies; ensures the focus remains on the specific functionality being tested; Test4z includes a Mock API.
Regression testing
A type of functional testing that ensures changes to the codebase have not adversely affected existing functionalities by re-running previously executed test cases; can be conducted at various levels including unit, integration, system, and acceptance testing.
Shift-left In the context of testing, shifting-left implies bringing testing activities, such as unit testing, integration testing, and some aspects of system testing, earlier in the development process; before DevOps, testing was conducted after the development phase was completed, leading to potential delays in identifying and fixing issues.
Six Core Principles of Unit Testing The gold standard, these principles serve as the basis for successful unit testing: 1) keep tests small, 2) run tests in isolation, 3) make maintenance easy, 4) swift and repeatable execution with a feedback loop, 5) keep tests independent, 6) align devs and managers on code coverage; Learn more
Spy Test double or test object used to observe and verify the behavior of a component under test; used primarily in unit testing and integration testing to monitor interactions between the component being tested and its dependencies; aides in the creation of thorough and effective unit and integration tests; Test4z includes a Spy API.
System testing Evaluates the entire system as a whole to ensure all components work together according to the specified requirements; encompasses functional, non-functional, regression, and user acceptance testing.
Test data management (TDM) Creating, managing, and provisioning data for software testing purposes; aims to ensure that test environments have the necessary and realistic data required for testing scenarios without compromising sensitive or confidential information.
Test harness A set of tools, libraries, and infrastructure that automate the testing of software components or systems; typically includes scripts or cases, data, drivers, mocks, and reporting/logging mechanisms. Test4z includes a Harness API.
Test4z A component of the Broadcom Mainframe DevOps Suite, Test4z is a solution for unit testing and, more broadly, functional test automation. A code-based, CLI-enabled solution, Test4z equips mainframe developers to adopt shift-left practices using their tools of choice. APIs act as an abstraction layer to simplify test case authoring and a runtime executes tests and captures code coverage. Learn more
Testing frameworks Also known as testing libraries; pre-defined sets of guidelines, rules, and reusable components that provide a structure for designing and implementing automated tests; often include tools, libraries, and conventions that simplify the process of writing, organizing, and executing tests.
Testing pyramid Metaphor that illustrates both the granularity and volume of tests by type; unit tests are the foundation of the pyramid, integration tests are typically the middle layer and end-to-end tests are at the top.
Unhappy path Testing scenarios where the system or application is subjected to inputs or conditions that are unexpected, abnormal, or erroneous; ensures the system handles errors, exceptions, and edge cases gracefully and appropriately.
Unit testing A type of functional testing; tests individual units or components of a software application in isolation to ensure each part of the code works as intended (e.g., COBOL paragraphs); critical practice in software development that contributes to improved code quality, faster development cycles, and greater confidence in the reliability and maintainability of application.
Unit Test for Mainframe Visual Studio Code extension that facilitates the use of the Test4z APIs and runtime by developers; developers can also use the CLI and/or connect directly using ISPF via SSH. Component of the Code4z developer experience.
White box testing Also known as ‘clear box’ testing, this method involves the internal structure and implementation details of the software being tested by examining the code logic, data flows, control flows, and program paths. Test4z supports white box testing.


Frequently Asked Questions

Why automate testing for mainframe applications?

Why do you recommend unit testing as the focal point for automation?

Other tools have been around. Why aren’t they working?

What about COBOL Check?

Is any additional tooling required for Test4z?

What development environments are supported by Test4z?

We have a massive codebase. How can test automation make an impact?

Will Test4z fit into my budget?

How do I get support?

How do I learn more?

Speak with an Advisor

Our expert advisors can answer your questions and help fast track your VS Code road to modernization.

Talk to a Modernization Advisor