Dev.to Security 🔐 Cybersecurity 👁 0 📖 12 min read

Beyond Green CI: Building a More Trustworthy DevSecOps Pipeline

How real container images, security checks, and CI failures helped improve the InfraPulse security pipeline. In Part 1 — 12 Green CI Jobs, 422 Security Findings, I shared an important lesson: even when all required CI

Beyond Green CI: Building a More Trustworthy DevSecOps Pipeline

How real container images, security checks, and CI failures helped improve the InfraPulse security pipeline.

In Part 1 — 12 Green CI Jobs, 422 Security Findings, I shared an important lesson: even when all required CI checks pass, security scanners can still report hundreds of findings that need further investigation.

In Part 2 — From 422 Security Findings to Auditable Evidence, I explored how InfraPulse evolved to produce more reliable evidence about software components, artifact integrity, and the software supply chain.

Now, in Part 3, a new challenge emerged:

How can we make sure these security checks still work when we move beyond controlled tests and start analyzing real container images?

That became the focus of the next stage of InfraPulse, an open-source monitoring and observability platform I am building to explore software engineering, DevOps, DevSecOps, and Site Reliability Engineering (SRE) through practical implementation.

The goal was to improve the security pipeline by connecting previously verified evidence to real container image analysis.

This article covers what changed, the challenges I encountered, and the lessons learned along the way.

1. The Challenge: Verification Is Not the Same as Trust

Imagine an application running inside a Docker container.

That container includes the application itself, operating system files, libraries, and other dependencies required to run.

Before using the image, we might want to answer a few questions:

  • Are we analyzing the files we actually expected?
  • Does the dependency inventory match the contents of the image?
  • Was the information used for comparison properly verified?
  • Can we confidently say that every software component was identified?

These questions sound similar, but they address different concerns.

For example, a valid digital signature can help verify that an artifact has not changed and that it was signed using a key accepted by a configured trust policy.

However, it does not automatically prove that every dependency was discovered or establish the real-world identity of the person or system that built the image.

Similarly, two inventories may match without either one containing every component.

The first important lesson was that successful verification must come with clearly defined limits.

2. How the InfraPulse Security Pipeline Evolved

InfraPulse already had mechanisms for validating security evidence, including digital signatures, artifact information, and software component inventories.

One of these inventories is called an SBOM (Software Bill of Materials).

Think of an SBOM as an ingredient list for software. It records known components and dependencies associated with an application.

The next step was to compare this information with the contents of real container images using the OCI format, a standard widely used in the container ecosystem.

A broader view of the security pipeline

Before exploring the implementation details, it helps to understand how the different security checks fit together.

The following diagram provides a high-level overview of the InfraPulse security pipeline, from source code and CI checks to container analysis, evidence verification, and reporting.

Figure 1 — InfraPulse Final Security Pipeline

Figure 1 — InfraPulse Security Pipeline Overview. Conceptual overview of code quality checks, container image analysis, security evidence, signature verification, and OCI inventory reconciliation. The pipeline preserves explicit coverage limitations rather than treating successful CI execution as proof of complete security.

The diagram highlights an important idea: security is not a single check performed at the end of development.

It involves multiple activities, including code quality, testing, vulnerability analysis, artifact integrity, and evidence verification.

Each activity contributes useful information, but no individual check can guarantee that an application is completely secure.

How the verification process works

The integration implemented in this stage follows five main steps:

  1. Receive the evidence package and independently defined trust requirements.
  2. Verify signatures, expected identities, and supporting information.
  3. Preserve the exact SBOM content that successfully passed verification.
  4. Compare that verified content against the inventory extracted from the container image.
  5. Report the results while keeping any limitations visible.

The most important detail is step three.

Imagine a system verifies a file and then opens the same file path again a few seconds later.

If the file changes between those operations, the system might end up processing something different from what it originally verified.

To avoid that inconsistency, InfraPulse passes forward the exact data snapshot that was verified.

The principle is straightforward:

Verifying an artifact is not enough. We also need to ensure that the data we use afterward is the data that passed verification.

This integration was delivered through Pull Request #15.

The post-merge GitHub Actions workflow completed successfully.

An important verification limitation

The implementation reuses the existing signed-evidence verifier instead of introducing a separate signature-verification mechanism.

This reduces duplicated logic and helps maintain consistent verification behavior.

However, the historical signing materials used in an earlier execution were not preserved with the real-image artifacts.

As a result, that exact historical signed execution cannot be reproduced.

The integration was exercised through automated tests, including mocked verifier interactions and real Cosign signing and verification in CI using disposable keys.

These tests provide evidence that the implemented components work under the tested conditions.

They do not reproduce the original historical signature chain.

This distinction matters because testing an integration and reproducing a specific historical verification are not the same thing.

3. Real Container Images Reveal Problems That Test Files May Miss

During development, small and controlled test files are extremely useful.

They make it easier to reproduce specific situations, such as invalid paths, malformed archives, or unexpected file structures.

But real container images are more complicated.

In this stage of the project, I worked with preserved archives from two images:

  • Backend: the application and its dependencies.
  • Nginx: the component used to serve the frontend.

These images contained variations of TAR archive metadata that the earlier tests did not fully represent.

The image layer reader had to be extended to support the observed formats, including USTAR, a GNU TAR header variant, and selected PAX metadata.

Why does this matter?

A container image is not just one compressed file.

It contains layers that describe changes to a filesystem. Understanding the resulting content requires interpreting those layers and their metadata correctly.

If a tool encounters an unfamiliar format, guessing how to interpret it can produce misleading results.

For this reason, the implementation follows a simple rule:

If the reader encounters an unsupported structure, it rejects the input instead of silently accepting it.

This approach is known as fail-closed.

It does not make the reader compatible with every possible container image format. Instead, it helps prevent unsupported data from being treated as successfully analyzed.

Some valid archive variations, such as compressed layers or unsupported PAX extensions, may still require additional implementation work.

What did the real-image validation show?

The validation used image archives preserved from an earlier GitHub Actions execution.

Result Backend Nginx
Layers processed 12 12
Inventory entries 11,095 2,214
Files with content hashes 9,228 1,264

In total, 24 layers were processed, with checks against recorded file hashes, image identifiers, and layer information.

These figures come from the project's validation report and should be confirmed against the retained execution evidence before publication.

The result demonstrated that the reader could process those specific real images and their observed archive formats.

It did not prove universal OCI compatibility or complete detection of every software component.

Figure 2 — Backend and Nginx Image Validation

Figure 2 — Real OCI Image Validation. InfraPulse analyzed two preserved container images: Backend and Nginx. Each image contained 12 processed layers, generating a total of 13,309 inventory entries and 10,492 files with content hashes. The validation included image identity and integrity checks, along with inventory comparison against the verified SBOM. The reported status remained COVERAGE_PARTIAL, indicating that successful validation does not guarantee universal OCI compatibility or complete identification of all software components.

4. Security Also Means Controlling Resource Usage

Another important lesson was that security tools need protection too.

Imagine receiving an unexpectedly large archive or one containing an excessive number of filesystem entries.

Even without malicious code, processing that archive could consume too much memory, overload the CPU, or interrupt a service.

To reduce that risk, I introduced explicit limits into the OCI layer reader.

Resource Implemented limit
Bytes per layer 128 MiB
Total layer bytes 256 MiB
Inventory entries 100,000
Number of layers 64
Local PAX metadata 1 MiB

Additional restrictions cover file path length and depth.

These values are specific defensive choices made for this implementation, not universal requirements of the OCI specification.

One limitation worth understanding

These controls protect the reader, but they do not automatically protect every step before it.

For example, a downloader could load an oversized archive into memory before the reader has an opportunity to reject it.

A production-oriented solution therefore needs resource controls throughout the process, including:

  • Downloading artifacts.
  • Extracting archives.
  • Parsing image layers.
  • Building inventories.
  • Generating reports.

This is an example of defense in depth: using multiple layers of protection rather than relying on a single security check.

The same principle applies to other systems that process external files, including backup tools, document processors, and software deployment pipelines.

5. When a CI Failure Is Actually Good News

During the implementation, GitHub Actions detected a Python formatting issue.

The check was performed by Ruff, a tool used to enforce code quality and formatting rules.

The initial pipeline run failed.

This was not a security vulnerability or a major architectural problem. The code simply did not meet one of the project's established quality requirements.

After correcting the formatting issue, the required checks were executed again and passed.

The workflow after the pull request was merged also completed successfully.

This reinforced an important principle:

The goal of CI is not to stay green at all costs. Its purpose is to catch problems before they become part of the codebase.

A failure that is detected, understood, and corrected shows that a quality control is doing its job.

The same reasoning applies to security checks.

We need to distinguish between situations such as:

  • A scanner completed successfully and reported vulnerabilities.
  • A scanner failed before completing its analysis.
  • A required security check never ran.
  • An informational security audit reported findings without blocking the pipeline.

These results mean different things.

Treating them as equivalent can create a false sense of security.

Figure 3 — GitHub Actions CI Failure and Recovery

Figure 3 — GitHub Actions Validation. The initial Ruff formatting failure was corrected before merge, followed by successful required checks and a successful post-merge workflow. Passing CI does not establish that the application is free from vulnerabilities.

6. The Most Important Result Was Not a Green CI Pipeline

By the end of this implementation stage, InfraPulse could validate the expected image information, process the supported layers, and compare image inventories against previously verified data.

Even so, the final result retained an explicit status:

COVERAGE_PARTIAL

In other words, partial coverage.

Why keep that status after so many checks succeeded?

Because three concepts need to remain separate.

Integrity: Does the data match the expected values?

Authenticity: What can we establish about the identity or authority behind an artifact or signature?

Completeness: Have all relevant software components actually been identified?

A tool may successfully verify file integrity without proving who originally built the image.

Likewise, it may compare two inventories without proving that either inventory is complete.

That is why InfraPulse does not turn a successful comparison into a claim of complete software coverage.

The project also does not claim to have achieved a particular SLSA level.

What about the 422 security findings from Part 1?

Those findings remain important context.

The 422 records were scanner-reported occurrences across different tools and targets. They were not necessarily 422 unique, confirmed, exploitable vulnerabilities.

This stage of the project did not demonstrate that all those findings had been eliminated.

A green CI pipeline does not mean zero vulnerabilities. It means the configured required checks passed under the conditions they evaluated.

Understanding that distinction is essential when interpreting security results.

It also explains why maintaining the COVERAGE_PARTIAL status is useful.

Instead of presenting a stronger guarantee than the available evidence supports, the pipeline makes its coverage limitations explicit.

7. Five Lessons I Learned Along the Way

This stage of InfraPulse brought several lessons that apply beyond this particular project.

1. Use the data that was actually verified

When an artifact passes validation, avoid later using another version that could have changed.

This applies to SBOMs, configuration files, deployment manifests, and other security-sensitive data.

2. Keep trust requirements independent

Trusted keys, expected image identifiers, and verification policies should not be defined solely by the artifact being examined.

Otherwise, an artifact could effectively set its own acceptance rules.

3. Combine controlled tests with real artifacts

Small test fixtures help reproduce specific problems.

Real artifacts reveal compatibility issues and unexpected details that controlled tests may miss.

Both approaches are valuable, but they answer different questions.

4. Protect the entire processing flow

A parser limit cannot prevent excessive memory usage if the downloader has already loaded a huge file.

Resource controls need to cover every relevant stage.

5. Never claim more than the evidence supports

A valid signature, a matching inventory, a successful scanner execution, and a green CI result are all useful.

But they do not prove the same thing.

Reliable engineering means understanding what each result confirms—and what remains uncertain.

That was probably the most valuable lesson from this implementation.

8. What Still Needs Improvement

The planned integration was completed, but several opportunities for improvement remain.

One is extending image format compatibility while maintaining safe parsing behavior.

Another is improving the ability to reproduce earlier security validations.

Different categories of evidence currently have different retention periods:

Artifact Retention
Preserved image archives 1 day
Sanitized reports 7 days
Unsigned CI evidence 30 days

This means a report may remain available after the original files needed to reproduce its results have already been removed.

Additionally, the disposable signing materials used in an earlier historical verification were not preserved, so that exact signed execution cannot be replayed.

This revealed another important distinction:

Keeping a report is not necessarily the same as preserving everything required to reproduce it.

Evidence retention and reproducibility are related, but they are not identical.

A project may retain summary reports for weeks while no longer having access to the exact image archives or signatures used in an earlier verification.

This is something worth considering when designing CI/CD pipelines that need to support audits or repeatable investigations.

Beyond the security pipeline, future areas of exploration include service-level indicators, actionable alerts, failure recovery, and repeatable operational procedures.

These are potential improvements, not features being claimed as completed here.

Figure 4 — Evidence Retention and Reproducibility

Figure 4 — Evidence Retention and Reproducibility. Image archives, sanitized reports, and unsigned CI evidence have different retention periods. Preserving a report does not necessarily preserve all materials required to reproduce the original verification.

9. Conclusion

This stage of InfraPulse demonstrated that implementing security controls is only part of the challenge.

We also need to verify that those controls work with real files, handle unexpected inputs safely, and produce results that can be interpreted correctly.

Throughout this implementation, I:

  • Connected verified security evidence to OCI image inventory analysis.
  • Extended the reader to handle formats encountered in real images.
  • Introduced resource limits to reduce processing risks.
  • Fixed an issue detected by CI before merging.
  • Completed the integration with the required checks passing.

But the most important lesson was understanding that a reliable system must recognize its own limits.

A green pipeline, a valid signature, or a well-structured report is not enough on its own.

We need to understand what each result actually proves.

InfraPulse continues to evolve as an open-source project focused on practical software engineering, DevOps, DevSecOps, and SRE.

GitHub: https://github.com/jacivaldocarvalho/jc-infrapulse

There are still challenges to explore, solutions to test, and lessons to share.

Until next time!

References and Tools

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.