UPSTREAM.TSF.TT
UPSTREAM.TSF.TT-CHANGES | Reviewed: ✔ | Score: 0.0
Supported Requests:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TRUSTABLE-SOFTWARE | This release of XYZ is Trustable. | 0.05 | ✔ Item Reviewed ✔ Link Reviewed |
Supporting Items:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TA-FIXES | Known bugs or misbehaviours are analysed and triaged, and critical fixes or mitigations are implemented or applied. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-UPDATES | XYZ components, configurations and tools are updated under specified change and configuration management controls. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
References:
None
Guidance
We expect that XYZ will need to be modified many times during its useful/production lifetime, and therefore we need to be sure that we can make changes without breaking it. In practice this means being able to deal with updates to dependencies and tools, as well as updates to XYZ itself.
Note that this implies that we need to be able to:
- verify that updated XYZ still satisfies its expectations (see below), and
- understand the behaviour of upstream/suppliers in delivering updates (e.g. frequency of planned updates, responsiveness for unplanned updates such as security fixes).
We need to consider the maturity of XYZ, since new software is likely to contain more undiscovered faults/bugs and thus require more changes. To support this we need to be able to understand, quantify and analyse changes made to XYZ (and its dependencies) on an ongoing basis, and to assess the XYZ approach to bugs and breaking changes.
We also need to be able to make modifications to any/all third-party components of XYZ and dependencies of XYZ, unless we are completely confident that suppliers/upstream will satisfy our needs throughout XYZ's production lifecycle.
UPSTREAM.TSF.TT-CONFIDENCE | Reviewed: ✔ | Score: 0.0625
Supported Requests:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TRUSTABLE-SOFTWARE | This release of XYZ is Trustable. | 0.05 | ✔ Item Reviewed ✔ Link Reviewed |
Supporting Items:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TA-CONFIDENCE | Confidence in XYZ is measured based on results of analysis | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-METHODOLOGIES | Manual methodologies applied for XYZ by contributors, and their results, are managed according to specified objectives. | 0.12 | ✔ Item Reviewed ✔ Link Reviewed |
References:
None
Guidance
Our overall objective is to deliver releases of XYZ that meet our expectations and do not cause harm. By collecting and assessing evidence for all of the factors above, we aim to assess (ideally measure) confidence in each release candidate, to support go/nogo decision-making, In assessing confidence we need to consider various categories of evidence including:
- subjective (e.g. provenance, reviews and approvals)
- binary (e.g. test pass/fail)
- stochastic (e.g. scheduling test results over time)
- empirical (e.g. advance warning signal monitoring data from production deployments)
UPSTREAM.TSF.TT-CONSTRUCTION | Reviewed: ✔ | Score: 0.05
Supported Requests:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TRUSTABLE-SOFTWARE | This release of XYZ is Trustable. | 0.05 | ✔ Item Reviewed ✔ Link Reviewed |
Supporting Items:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TA-ITERATIONS | All constructed iterations of XYZ include source code, build and usage instructions, tests, results, and attestations. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-RELEASES | Construction of XYZ releases is fully repeatable and the results are fully reproducible, with any exceptions documented and justified. | 0.15 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-TESTS | All tests for XYZ, and its build and test environments, are constructed from controlled/mirrored sources and are reproducible, with any exceptions documented | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
References:
None
Guidance
Where possible we prefer to build, configure and install XYZ from source code, because this reduces (but does not eliminate) the possibility of supply chain tampering.
When constructing XYZ we aspire to a set of best practices including:
- reproducible builds
- construction from a given set of input source files and build instructions leads to a specific fileset
- re-running the build leads to exactly the same fileset, bit-for-bit
- ensuring that all XYZ dependencies are known and controlled (no reliance on external/internet resources, or unique/golden/blessed build server); and
- automated build, configuration and deployment of XYZ based on declarative instructions, kept in version control (e.g. no engineer laptop in the loop for production releases)
Some of these constraints may be relaxed during XYZ development/engineering phases, but they must all be fully applied for production releases. Note that when we receive only binaries, without source code, we must rely much more heavily on Provenance; who supplied the binaries, and how can we trust their agenda, processes, timescales and deliveries?
UPSTREAM.TSF.TT-EXPECTATIONS | Reviewed: ✔ | Score: 0.18
Supported Requests:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TRUSTABLE-SOFTWARE | This release of XYZ is Trustable. | 0.05 | ✔ Item Reviewed ✔ Link Reviewed |
Supporting Items:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TA-BEHAVIOURS | Expected or required behaviours for XYZ are identified, specified, verified and validated based on analysis. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-CONSTRAINTS | Constraints on adaptation and deployment of XYZ are specified. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-INDICATORS | Advance warning indicators for misbehaviours are identified, and monitoring mechanisms are specified, verified and validated based on analysis. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-MISBEHAVIOURS | Prohibited misbehaviours for XYZ are identified, and mitigations are specified, verified and validated based on analysis. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.ECLIPSE.ECLIPSE-PROJECT_README | An Eclipse projects must have a README file with information about the project. | 0.90 | ✔ Item Reviewed ✔ Link Reviewed |
References:
None
Guidance
While most modern software is developed without a formal requirement/specification process, we need to be clear about what we expect from XYZ, communicate these expectations, and verify that they are met.
In most (almost all?) cases, we need to verify our expectations by tests. These tests must be automated and applied for every candidate release of XYZ.
It is not sufficient to demonstrate that the software does what we expect. We also need to analyse the potential risks in our scenario, identify unacceptable and/or dangerous misbehaviours and verify that they are absent, prevented or mitigated.
In most cases it is not sufficient to demonstrate behaviours and mitigations only in a factory/laboratory environment. We also need to establish methods for monitoring critical behaviours and misbehaviours in production, and methods for taking appropriate action based on advance warning indicators of potential misbehaviour.
Consequently, when defining expectations, mitigations, and warning indicators, the scope and its assumptions about environment and usage must be specified and stated explicitly, so users understand the context of the claims.
UPSTREAM.TSF.TT-PROVENANCE | Reviewed: ✔ | Score: 0.0
Supported Requests:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TRUSTABLE-SOFTWARE | This release of XYZ is Trustable. | 0.05 | ✔ Item Reviewed ✔ Link Reviewed |
Supporting Items:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TA-INPUTS | All inputs to XYZ are assessed, to identify potential risks and issues | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-SUPPLY_CHAIN | All sources for XYZ and tools are mirrored in our controlled environment | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
References:
None
Guidance
Anything that can affect the output of the XYZ project is considered to be an input. This will include:
- Software components used to implement specific features
- Software tools used for construction and verification
- Infrastructure that supports development and release processes
Ideally we want all XYZ contributors to be expert, motivated, reliable, transparent and ethical. Unfortunately this is not always achievable in practice:
- Given the scale, complexity and evolution of modern software systems it is impossible for engineers to be expert in all topics.
- Even the most competent engineers have bad days.
- Many engineers are unable to share information due to commercial secrecy agreements.
- Individuals and teams may be motivated or manipulated by external pressures, e.g. money and politics.
We can and should, however, consider who produced XYZ and its components, their motivations and practices, the assertions they make, supporting evidence for these assertions, and feedback from users of XYZ if available.
Similarly, we want existing software used to create XYZ to be well documented, actively maintained, thoroughly tested, bug-free and well suited to its use in XYZ. In practice, we will rarely have all of this, but we can at least evaluate the software components of XYZ, and the tools used to construct it.
UPSTREAM.TSF.TT-RESULTS | Reviewed: ✔ | Score: 0.0
Supported Requests:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TRUSTABLE-SOFTWARE | This release of XYZ is Trustable. | 0.05 | ✔ Item Reviewed ✔ Link Reviewed |
Supporting Items:
| Item | Summary | Score | Status |
|---|---|---|---|
| UPSTREAM.TSF.TA-ANALYSIS | Collected test and monitoring data for XYZ is analysed using verified methods to validate expected behaviours and identify new misbehaviours. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-DATA | Test and monitoring data from development and production are appropriately collected and retained. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
| UPSTREAM.TSF.TA-VALIDATION | Tests exercise both stressed and representative conditions, validating behaviour through systematic, scheduled repetition. | 0.00 | ✔ Item Reviewed ✔ Link Reviewed |
References:
None
Guidance
We need to perform tests to verify expected behaviours and properties of XYZ in advance of every candidate release.
We also need to validate these tests, to confirm that they do actually test for the expected behaviours or properties, and that test failures are properly detected. Usually this can be done by inducting software faults to exercise the tests and checking that the results record the expected failure.
Similarly we need to verify that prevention measures and mitigations continue to work for each XYZ candidate release, and in production.
All these validated test results and monitored advance warning signal data from production must be collected and analysed on an ongoing basis. We need to notice when results or trends in advance warning signals change, and react appropriately.