# Reporting The Magento Functional Testing Framework provides two types of reporting: - Inline reporting that you can view in the terminal as you run [`mftf`][mftf] or [`codecept`][codecept] CLI commands. - HTML reports that you can view using the [Allure Framework][] after a test run completes. When you run a test, MFTF copies all reporting artifacts to the `dev/tests/acceptance/tests/_output` subdirectory in the Magento root directory. The directory contains: - `allure-results/` that is a directory generated and served by the Allure Framework. - `failed` that is a text file containing relative paths to failed tests after the last test run. The paths are relative to `dev/tests/acceptance/`. - `.html` and `.png` files that are screenshots of fails in HTML and PNG formats. To cleanup the `_output/` directory, remove them manually. The `mftf` tool logs output continuously to the `dev/tests/acceptance/mftf.log` file. ## Command line MFTF reports about its progress during test run when you run the `mftf` CLI tool with [`run:test`][] or [`run:group`][] commands. The report can contain three main parts: - Pre-run checks: - Environment check, such as PHP warnings, etc. - XML test validation like deprecation warnings such as missing required components in XML tests. - Codeception report which is the progress report for each test. - Total results of the test run such as number of tests, assertions, and failures. To manage the level of verbosity, use `-v` or `--verbose` flag in the `mftf` commands. To enable verbosity using the `codecept` commands, refer to the Codeception [Console Commands][codeception]. The following sections demonstrate an example interpretation of a complete log separated into chunks. ### Pre-run check report First, MFTF returns `DEPRECATION` warnings alerting you that required test components are missing in XML test definitions. ```terminal DEPRECATION: Test AdminFilteringCategoryProductsUsingScopeSelectorTest is missing required annotations.{"testName":"AdminFilteringCategoryProductsUsingScopeSelectorTest","missingAnnotations":"stories"} DEPRECATION: Test AdminAbleToShipPartiallyInvoicedItemsTest is missing required annotations.{"testName":"AdminAbleToShipPartiallyInvoicedItemsTest","missingAnnotations":"stories"} DEPRECATION: Test AdminRemoveProductWeeeAttributeOptionTest is missing required annotations.{"testName":"AdminRemoveProductWeeeAttributeOptionTest","missingAnnotations":"stories"} Generate Tests Command Run ``` `Generate Tests Command Run` indicates that test generation is finished and tests are able to be executed. ### Test execution report A test execution report is generated by Codeception and includes configuration information, scenario execution steps, and PASSED/FAIL verdict after each test completion. #### General information The general information can be useful for MFTF contributors, but can be ignored by a test writer. Let's consider the general part of the following test execution report: ```terminal Generate Tests Command Run Codeception PHP Testing Framework v4.1.4 Powered by PHPUnit 9.1.3 by Sebastian Bergmann and contributors. Running with seed: Magento\FunctionalTestingFramework.functional Tests (2) ------------------------ Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer ``` After the test generation command (mentioned in the previous section), MFTF delegates control to the `vendor/codeception` tool, which is the `Codeception PHP Testing Framework` of version `4.1.4` that uses `PHPUnit` of version `9.1.3`. The tool runs `2 Tests` using the configuration defined in the `functional` suite under the `Magento\FunctionalTestingFramework` namespace. The corresponding configuration file is `acceptance/tests/functional.suite.yml`. It enables `Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer,..` #### Passed tests The next chunk of the log reports about test execution of the first test: ```terminal AdminLoginSuccessfulTestCest: Admin login successful test Signature: Magento\AcceptanceTest\_default\Backend\AdminLoginSuccessfulTestCest:AdminLoginSuccessfulTest Test: tests/functional/Magento/_generated/default/AdminLoginSuccessfulTestCest.php:AdminLoginSuccessfulTest Scenario -- [loginAsAdmin] AdminLoginActionGroup [navigateToAdmin] am on page "/admin/admin" [fillUsername] fill field "#username","admin" [fillPassword] fill field "#login","123123q" [clickLogin] click ".actions .action-primary" [clickLoginWaitForPageLoad] wait for page load 30 [clickDontAllowButtonIfVisible] conditional click ".modal-popup .action-secondary",".modal-popup .action-secondary",true [closeAdminNotification] close admin notification [assertLoggedIn] AssertAdminSuccessLoginActionGroup [waitForAdminAccountTextVisible] wait for element visible ".page-header .admin-user-account-text",60 [assertAdminAccountTextElement] see element ".page-header .admin-user-account-text" [logoutFromAdmin] AdminLogoutActionGroup [amOnLogoutPage] am on page "/admin/admin/auth/logout/" PASSED ``` The running test is `AdminLoginSuccessfulTestCest`, which is `Admin login successful test` (this text is generated from the test name but with the `Cest` part excluded). Its test signature is `Magento\AcceptanceTest\_default\Backend\AdminLoginSuccessfulTestCest:AdminLoginSuccessfulTest` that matches a `className:methodName` format using namespaces. A path to the corresponding `Test` is `tests/functional/Magento/_generated/default/AdminLoginSuccessfulTestCest.php:AdminLoginSuccessfulTest` (relative to the `acceptance/` directory). `Scenario` lists the tests steps as they run during test execution, ending with the successful test verdict `PASSED`. It means that all test steps were processed as expected. #### Failed tests The second test fails with the following report: ```terminal AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test Signature: Magento\AcceptanceTest\_default\Backend\AdminMenuNavigationWithSecretKeysTestCest:AdminMenuNavigationWithSecretKeysTest Test: tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest Scenario -- [enableUrlSecretKeys] magento cli "config:set admin/security/use_form_key 1",60 Value was saved. [cleanInvalidatedCaches1] magento cli "cache:clean config full_page",60 Cleaned cache types: config full_page [loginAsAdmin] AdminLoginActionGroup [navigateToAdmin] am on page "/admin/admin" [fillUsername] fill field "#username","admin" [fillPassword] fill field "#login","123123q" [clickLogin] click ".actions .action-primary" [clickLoginWaitForPageLoad] wait for page load 30 [clickDontAllowButtonIfVisible] conditional click ".modal-popup .action-secondary",".modal-popup .action-secondary",true [closeAdminNotification] close admin notification [clickStoresMenuOption1] click "#menu-magento-backend-stores" [waitForStoresMenu1] wait for loading mask to disappear [clickStoresConfigurationMenuOption1] click "#nav li[data-ui-id='menu-magento-config-system-config']" [waitForConfigurationPageLoad1] wait for page load 60 [seeCurrentUrlMatchesConfigPath1] see current url matches "~\/admin\/system_config\/~" [clickCatalogMenuOption] click "#something" [saveScreenshot] save screenshot [disableUrlSecretKeys] magento cli "config:set admin/security/use_form_key 0",60 Value was saved. [cleanInvalidatedCaches2] magento cli "cache:clean config full_page",60 Cleaned cache types: config full_page [logout] AdminLogoutActionGroup [amOnPage] am on page "/admin/admin/auth/logout/" FAIL -------------------------------------------------------------------------------- ``` The general test details and scenario has the same format as in the Passed test. ```terminal [clickCatalogMenuOption] click "#something" [saveScreenshot] save screenshot ``` When a test step fails, MFTF always saves a screenshot of the web page with the failing state immediately after the failure occurs. `[saveScreenshot] save screenshot` follows the failing test step `[clickCatalogMenuOption] click "#something"` in our case. A screenshot of the fail goes at the `acceptance/tests/_output` directory in both PNG and HTML formats: - `Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.html` - `Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.png` The file name encodes: - `Magento` namespace - with the `AcceptanceTest` test type - generated as a part of the `_default` suite - defined at the `Magento_Backend` module - implemented in the `AdminMenuNavigationWithSecretKeysTestCest` PHP class - with the `AdminMenuNavigationWithSecretKeysTest` test name - and execution status `fail` Actions after `saveScreenshot` are run as a part of the [`after`][] hook of the test. ### Test result report After MFTF completed test execution, it generates a general report about test results along with detailed information about each fail. ```terminal Time: 02:07.534, Memory: 150.50 MB There was 1 failure: --------- ``` MFTF reports that the test run took 02:07.534 using 150.50 MB of system RAM. And, finally, that there was `1 failure`. Next, the report provides details about the test failure. ```terminal --------- 1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test Test tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest Step Click "#something" Fail CSS or XPath element with '#something' was not found. Scenario Steps: 27. // Exiting Action Group [logout] AdminLogoutActionGroup 26. $I->amOnPage("/admin/admin/auth/logout/") at tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:55 25. // Entering Action Group [logout] AdminLogoutActionGroup 24. // Cleaned cache types: config full_page 23. $I->magentoCLI("cache:clean config full_page",60) at tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:52 22. // Value was saved. ``` - `1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test` - the failed Codeception test is *AdminMenuNavigationWithSecretKeysTestCest*. It references to the PHP class that implemented the failed test. - `Test tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest` - the test is implemented in the *AdminMenuNavigationWithSecretKeysTest* test method of the *tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php* file under `/dev/tests/acceptance/`. It matches the corresponding test defined in XML that is *AdminMenuNavigationWithSecretKeysTest* defined in `...` - `Step Click "#something"` - the failing test step is the *click* action with the *#something* selector. It would correspond the `` test step in the XML defined tests. Finally, the report finishes with fairly self-descriptive lines. ```terminal FAILURES! Tests: 2, Assertions: 2, Failures: 1. ``` MFTF encountered failures due to the last test run, that included *2* tests with *2* assertions. *1* assertion fails. ## Allure Each time you run tests, MFTF appends an XML file with results at the `tests/_output/allure-results/` directory. The official [Allure Test Report][] documentation is well-covered, so we'll list only the CLI commands that you would need for your day-to-day work.
The following commands are relative to the Magento installation directory.
To generate the HTML Allure report in a temporary folder and open the report in your default web browser: ```bash allure serve dev/tests/acceptance/tests/_output/allure-results/ ``` To generate a report to the `allure-report/` at the current directory: ```bash allure generate dev/tests/acceptance/tests/_output/allure-result ``` To generate a report to a particular directory, use the `-o` option: ```bash allure generate dev/tests/acceptance/tests/_output/allure-result -o dev/tests/acceptance/tests/_output/allure-report ``` To launch the generated report in a web browser: ```bash allure open dev/tests/acceptance/tests/_output/allure-report ```
By default, Allure generates reports in the `allure-report/` at the current directory. For example, if you run the command without `-o` flag while you are in the `magento2/` directory, Allure will generate a report at the `magento2/allure-report/` directory.
```bash allure generate dev/tests/acceptance/tests/_output/allure-result ``` Example of file structure after the command run: ```terminal magento2 ├── allure-report ├── app ├── bin ├── dev ├── ... ``` And if you run the `open` command with no arguments while you are in the same directory (`magento2/`): ```bash allure open ``` Allure would attempt to open a generated report at the `magento2/allure-report/` directory. To clean up existing reports before generation (for example after getting new results), use the `--clean` flag: ```bash allure generate dev/tests/acceptance/tests/_output/allure-result --clean ``` Refer to the [Reporting section][] for more Allure CLI details. [`after`]: test.md#after-tag [`run:group`]: commands/mftf.md#rungroup [`run:test`]: commands/mftf.md#runtest [Allure Framework]: https://docs.qameta.io/allure/ [Allure Test Report]: http://allure.qatools.ru/ [codecept]: commands/codeception.md [codeception]: https://codeception.com/docs/reference/Commands [mftf]: commands/mftf.md [report an issue]: https://github.com/magento/magento2-functional-testing-framework/blob/master/.github/CONTRIBUTING.md#report-an-issue [Reporting section]: https://docs.qameta.io/allure/#_reporting