Open MCT Testing is iterating and improving at a rapid pace. This document serves to capture and index existing testing documentation and house documentation which no other obvious location as our testing evolves.
* Unit Test specs should reside alongside the source code they test, not in a separate directory.
* Unit test specs for plugins should be defined at the plugin level. Start with one test spec per plugin named pluginSpec.js, and as this test spec grows too big, break it up into multiple test specs that logically group related tests.
* Unit tests for API or for utility functions and classes may be defined at a per-source file level.
* Wherever possible only use and mock public API, builtin functions, and UI in your test specs. Do not directly invoke any private functions. ie. only call or mock functions and objects exposed by openmct.* (eg. openmct.telemetry, openmct.objectView, etc.), and builtin browser functions (fetch, requestAnimationFrame, setTimeout, etc.).
* Where builtin functions have been mocked, be sure to clear them between tests.
* If you’re testing a view, you do not need to test the whole application UI, you can just fetch the view provider using the public API and render the view into an element that you have created.
* You do not need to test that the view switcher works, there should be separate tests for that.
* You do not need to test that telemetry providers work, you can mock openmct.telemetry.request() to feed test data to the view.
* Use your best judgement when deciding on appropriate scope.
* Automated tests for plugins should start by actually installing the plugin being tested, and then test that installing the plugin adds the desired features and behavior to Open MCT, observing the above rules.
* All variables used in a test spec, including any instances of the Open MCT API should be declared inside of an appropriate block scope (not at the root level of the source file), and should be initialized in the relevant beforeEach block. `beforeEach` is preferable to `beforeAll` to avoid leaking of state between tests.
* A `afterEach` or `afterAll` should be used to do any clean up necessary to prevent leakage of state between test specs. This can happen when functions on `window` are wrapped, or when the URL is changed. [A convenience function](https://github.com/nasa/openmct/blob/master/src/utils/testing.js#L59) is provided for resetting the URL and clearing builtin spies between tests.
Line Code Coverage is generated by our unit tests and e2e tests, then combined by ([Codecov.io Flags](https://docs.codecov.com/docs/flags)), and finally reported in GitHub PRs by Codecov.io's PR Bot. This workflow gives a comprehensive (if flawed) view of line coverage.
### Karma-istanbul
Line coverage is generated by our `karma-coverage-istanbul-reporter` package as defined in our `karma.conf.js` file:
```js
coverageIstanbulReporter: {
fixWebpackSourcePaths: true,
skipFilesWithNoCoverage: true,
dir: 'coverage/unit', //Sets coverage file to be consumed by codecov.io
reports: ['lcovonly']
},
```
Once the file is generated, it can be published to codecov with
1. Each e2e suite will start webpack with the ```npm run start:coverage``` command with config `webpack.coverage.mjs` and the `babel-plugin-istanbul` plugin to generate code coverage during e2e test execution using our custom [baseFixture](./baseFixtures.js).
1. During testcase execution, each e2e shard will generate its piece of the larger coverage suite. **This coverage file is not merged**. The raw coverage file is stored in a `.nyc_report` directory.
1. [nyc](https://github.com/istanbuljs/nyc) converts this directory into a `lcov` file with the following command `npm run cov:e2e:report`
1. Most of the tests focus on chrome/ubuntu at a single resolution. This coverage is published to codecov with `npm run cov:e2e:ci:publish`.
1. The rest of our coverage only appears when run against persistent datastore (couchdb), non-ubuntu machines, and non-chrome browsers with the `npm run cov:e2e:full:publish` flag. Since this happens about once a day, we have leveraged codecov.io's carryforward flag to report on lines covered outside of each commit on an individual PR.
* Required Checks. We're leveraging required checks in GitHub so that we can quickly and precisely control what becomes and informational failure vs a hard requirement. The only way to determine the difference between a required vs information check is check for the `(Required)` emblem next to the step details in GitHub Checks.
* Not all required checks are run per commit. You may need to manually trigger addition GitHub checks with a `pr:<label>` label added to your PR.
[CircleCI's test insights feature](https://circleci.com/blog/introducing-test-insights-with-flaky-test-detection/) collects historical data about the individual test results for both unit and e2e tests. Note: only a 14 day window of flake is available.
It's possible that you're running with dependencies or a local environment which is out of sync with the branch you're working on. Make sure to execute the following: