From 2cc00271b134c4e62fcfa8bb73a90a415e1f6eec Mon Sep 17 00:00:00 2001 From: John Hill Date: Thu, 9 Nov 2023 08:04:52 -0800 Subject: [PATCH] [CI] Add CI Troubleshooting doc (#6988) * document how to bust cache * Update TESTING.md Co-authored-by: David 'Epper' Marshall * add approapriate link * Update missing troublehshooting doc and combine code-cov docs * comments --------- Co-authored-by: David 'Epper' Marshall --- README.md | 2 ++ TESTING.md | 83 +++++++++++++++++++++++++++++++++++++++++++++++---- e2e/README.md | 16 +++------- 3 files changed, 83 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 6f8a1f07a4..90d1bdf058 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,8 @@ Each test suite generates a report in CircleCI. For a complete overview of testi Our code coverage is generated during the runtime of our unit, e2e, and visual tests. The combination of those reports is published to [codecov.io](https://app.codecov.io/gh/nasa/openmct/) +For more on the specifics of our code coverage setup, [see](TESTING.md#code-coverage) + # Glossary Certain terms are used throughout Open MCT with consistent meanings diff --git a/TESTING.md b/TESTING.md index 734f90deb0..e5c6e04da4 100644 --- a/TESTING.md +++ b/TESTING.md @@ -37,14 +37,85 @@ Documentation located [here](./e2e/README.md) ## Code Coverage -* 100% statement coverage is achievable and desirable. +It's up to the individual developer as to whether they want to add line coverage in the form of a unit test or e2e test. -Codecov.io will combine each of the above commands with [Codecov.io Flags](https://docs.codecov.com/docs/flags). Effectively, this allows us to combine multiple reports which are run at various stages of our CI Pipeline or run as part of a parallel process. +Line Code Coverage is generated by our unit tests and e2e tests, then combined by ([Codecov.io Flags](https://docs.codecov.com/docs/flags)), and finally reported in GitHub PRs by Codecov.io's PR Bot. This workflow gives a comprehensive (if flawed) view of line coverage. + +### Karma-istanbul + +Line coverage is generated by our `karma-coverage-istanbul-reporter` package as defined in our `karma.conf.js` file: + +```js + coverageIstanbulReporter: { + fixWebpackSourcePaths: true, + skipFilesWithNoCoverage: true, + dir: 'coverage/unit', //Sets coverage file to be consumed by codecov.io + reports: ['lcovonly'] + }, +``` + +Once the file is generated, it can be published to codecov with + +```json + "cov:unit:publish": "codecov --disable=gcov -f ./coverage/unit/lcov.info -F unit", +``` + +### e2e +The e2e line coverage is a bit more complex than the karma implementation. This is the general sequence of events: + +1. Each e2e suite will start webpack with the ```npm run start:coverage``` command with config `webpack.coverage.js` and the `babel-plugin-istanbul` plugin to generate code coverage during e2e test execution using our custom [baseFixture](./baseFixtures.js). +1. During testcase execution, each e2e shard will generate its piece of the larger coverage suite. **This coverage file is not merged**. The raw coverage file is stored in a `.nyc_report` directory. +1. [nyc](https://github.com/istanbuljs/nyc) converts this directory into a `lcov` file with the following command `npm run cov:e2e:report` +1. Most of the tests are run in the '@stable' configuration and focus on chrome/ubuntu at a single resolution. This coverage is published to codecov with `npm run cov:e2e:stable:publish`. +1. The rest of our coverage only appears when run against `@unstable` tests, persistent datastore (couchdb), non-ubuntu machines, and non-chrome browsers with the `npm run cov:e2e:full:publish` flag. Since this happens about once a day, we have leveraged codecov.io's carryforward flag to report on lines covered outside of each commit on an individual PR. -This e2e coverage is combined with our unit test report to give a comprehensive (if flawed) view of line coverage. ### Limitations in our code coverage reporting +Our code coverage implementation has some known limitations: +- [Variability](https://github.com/nasa/openmct/issues/5811) +- [Accuracy](https://github.com/nasa/openmct/issues/7015) +- [Vue instrumentation gaps](https://github.com/nasa/openmct/issues/4973) -Our code coverage implementation has two known limitations: -- [Variability and accuracy](https://github.com/nasa/openmct/issues/5811) -- [Vue instrumentation](https://github.com/nasa/openmct/issues/4973) +## Troubleshooting CI +The following is an evolving guide to troubleshoot CI and PR issues. + +### Github Checks failing +There are a few reasons that your GitHub PR could be failing beyond simple failed tests. +* Required Checks. We're leveraging required checks in GitHub so that we can quickly and precisely control what becomes and informational failure vs a hard requirement. The only way to determine the difference between a required vs information check is check for the `(Required)` emblem next to the step details in GitHub Checks. +* Not all required checks are run per commit. You may need to manually trigger addition GitHub checks with a `pr: