[CI] Add CI Troubleshooting doc (#6988)

* document how to bust cache

* Update TESTING.md

Co-authored-by: David 'Epper' Marshall <epper.marshall@gmail.com>

* add approapriate link

* Update missing troublehshooting doc and combine code-cov docs

* comments

---------

Co-authored-by: David 'Epper' Marshall <epper.marshall@gmail.com>

README.md

@@ -127,6 +127,8 @@ Each test suite generates a report in CircleCI. For a complete overview of testi
 
 Our code coverage is generated during the runtime of our unit, e2e, and visual tests. The combination of those reports is published to [codecov.io](https://app.codecov.io/gh/nasa/openmct/)
 
+For more on the specifics of our code coverage setup, [see](TESTING.md#code-coverage)
+
 # Glossary
 
 Certain terms are used throughout Open MCT with consistent meanings

TESTING.md

@@ -37,14 +37,85 @@ Documentation located [here](./e2e/README.md)
 
 ## Code Coverage
 
-* 100% statement coverage is achievable and desirable.
+It's up to the individual developer as to whether they want to add line coverage in the form of a unit test or e2e test.
 
-Codecov.io will combine each of the above commands with [Codecov.io Flags](https://docs.codecov.com/docs/flags). Effectively, this allows us to combine multiple reports which are run at various stages of our CI Pipeline or run as part of a parallel process.
+Line Code Coverage is generated by our unit tests and e2e tests, then combined by ([Codecov.io Flags](https://docs.codecov.com/docs/flags)), and finally reported in GitHub PRs by Codecov.io's PR Bot. This workflow gives a comprehensive (if flawed) view of line coverage.
+
+### Karma-istanbul
+
+Line coverage is generated by our `karma-coverage-istanbul-reporter` package as defined in our `karma.conf.js` file:
+
+```js
+    coverageIstanbulReporter: {
+      fixWebpackSourcePaths: true,
+      skipFilesWithNoCoverage: true,
+      dir: 'coverage/unit', //Sets coverage file to be consumed by codecov.io
+      reports: ['lcovonly']
+    },
+```
+
+Once the file is generated, it can be published to codecov with
+
+```json
+    "cov:unit:publish": "codecov --disable=gcov -f ./coverage/unit/lcov.info -F unit",
+```
+
+### e2e
+The e2e line coverage is a bit more complex than the karma implementation. This is the general sequence of events:
+
+1. Each e2e suite will start webpack with the ```npm run start:coverage``` command with config `webpack.coverage.js` and the `babel-plugin-istanbul` plugin to generate code coverage during e2e test execution using our custom [baseFixture](./baseFixtures.js). 
+1. During testcase execution, each e2e shard will generate its piece of the larger coverage suite. **This coverage file is not merged**. The raw coverage file is stored in a `.nyc_report` directory.
+1. [nyc](https://github.com/istanbuljs/nyc) converts this directory into a `lcov` file with the following command `npm run cov:e2e:report`
+1. Most of the tests are run in the '@stable' configuration and focus on chrome/ubuntu at a single resolution. This coverage is published to codecov with `npm run cov:e2e:stable:publish`.
+1. The rest of our coverage only appears when run against `@unstable` tests, persistent datastore (couchdb), non-ubuntu machines, and non-chrome browsers with the `npm run cov:e2e:full:publish` flag. Since this happens about once a day, we have leveraged codecov.io's carryforward flag to report on lines covered outside of each commit on an individual PR.
 
-This e2e coverage is combined with our unit test report to give a comprehensive (if flawed) view of line coverage.
 
 ### Limitations in our code coverage reporting
+Our code coverage implementation has some known limitations:
+- [Variability](https://github.com/nasa/openmct/issues/5811)
+- [Accuracy](https://github.com/nasa/openmct/issues/7015)
+- [Vue instrumentation gaps](https://github.com/nasa/openmct/issues/4973)
 
-Our code coverage implementation has two known limitations:
-- [Variability and accuracy](https://github.com/nasa/openmct/issues/5811)
-- [Vue instrumentation](https://github.com/nasa/openmct/issues/4973)
+## Troubleshooting CI
+The following is an evolving guide to troubleshoot CI and PR issues.
+
+### Github Checks failing
+There are a few reasons that your GitHub PR could be failing beyond simple failed tests.
+* Required Checks. We're leveraging required checks in GitHub so that we can quickly and precisely control what becomes and informational failure vs a hard requirement. The only way to determine the difference between a required vs information check is check for the `(Required)` emblem next to the step details in GitHub Checks.
+* Not all required checks are run per commit. You may need to manually trigger addition GitHub checks with a `pr:<label>` label added to your PR.
+
+### Flaky tests
+There are two ways to know if a test on your branch is historically flaky: 
+1. `deploysentinel`'s PR comment bot to give an accurate and historical view of e2e flakiness. Check your PR for a view of the test failures and flakes (with link to the failing test). Note: only a 7 day window of flake is available.
+2. (CircleCI's test insights feature)[https://circleci.com/blog/introducing-test-insights-with-flaky-test-detection/] collects historical data about the individual test results for both unit and e2e tests. Note: only a 14 day window of flake is available.
+
+### Local=Pass and CI=Fail
+Although rare, it is possible that your test can pass locally but fail in CI.
+
+#### Busting Cache
+In certain circumstances, the CircleCI cache can become stale. In order to bust the cache, we've implemented a runtime boolean parameter in Circle CI creatively name BUST_CACHE. To execute:
+1. Navigate to the branch in Circle CI believed to have stale cache.
+1. Click on the 'Trigger Pipeline' button.
+1. Add Parameter -> Parameter Type = boolean , Name = BUST_CACHE ,Value = true
+1. Click 'Trigger Pipeline'
+
+#### Run tests in the same container as CI
+
+In extreme cases, tests can fail due to the constraints of running within a container. To execute tests in exactly the same way as run in CircleCI. 
+
+```sh
+// Replace {X.X.X} with the current Playwright version 
+// from our package.json or circleCI configuration file
+docker run --rm --network host --cpus="2" -v $(pwd):/work/ -w /work/ -it mcr.microsoft.com/playwright:v{X.X.X}-focal /bin/bash
+npm install
+```
+
+At this point, you're running inside the same container and with 2 cpu cores. You can specify the unit tests:
+```sh
+npm run test
+```
+or e2e tests:
+
+```sh
+npx playwright test --config=e2e/playwright-ci.config.js --project=chrome --grep <the testcase name>
+```

e2e/README.md

@@ -490,15 +490,7 @@ Our e2e code coverage is captured and combined with our unit test coverage. For
 
 #### Generating e2e code coverage
 
-Code coverage is collected during test execution using our custom [baseFixture](./baseFixtures.js). The raw coverage files are stored in a `.nyc_report` directory to be converted into a lcov file with the following [nyc](https://github.com/istanbuljs/nyc) command:
-
-```npm run cov:e2e:report```
-
-At this point, the nyc linecov report can be published to [codecov.io](https://about.codecov.io/) with the following command:
-
-```npm run cov:e2e:stable:publish``` for the stable suite running in ubuntu.
-or
-```npm run cov:e2e:full:publish``` for the full suite running against all available platforms.
+Please read more about our code coverage [here](../TESTING.md#code-coverage)
 
 ## Other
 
@@ -548,10 +540,10 @@ A single e2e test in Open MCT is extended to run:
 - How is Open MCT extending default Playwright functionality?
 - What about Component Testing?
 
-### Troubleshooting
+### e2e Troubleshooting
+
+Please follow the general guide troubleshooting in [the general troubleshooting doc](../TESTING.md#troubleshooting-ci)
 
-- Why is my test failing on CI and not locally?
-- How can I view the failing tests on CI?
 - Tests won't start because 'Error: <http://localhost:8080/># is already used...'
 This error will appear when running the tests locally. Sometimes, the webserver is left in an orphaned state and needs to be cleaned up. To clear up the orphaned webserver, execute the following from your Terminal:
 ```lsof -n -i4TCP:8080 | awk '{print$2}' | tail -1 | xargs kill -9```