Skip to main content

Compliance

Starter+Enterprise

The Compliance module runs your cloud account against industry-standard security benchmarks and produces a structured pass/fail report for every control. Powered by the Calabi Compliance Engine, it continuously evaluates your cloud configuration against CIS, NIST, and AWS-published frameworks — giving security and compliance teams a single pane of glass for audit readiness.

Supported Benchmarks​

BenchmarkStandardControlsUpdate cadence
CIS AWS Foundations Benchmark v3.0CIS58Quarterly
CIS AWS Foundations Benchmark v1.4CIS53Quarterly
NIST 800-53 Revision 5NIST447Annual
Cloud Foundational Security Best Practices v1.0AWS210Monthly
PCI DSS v3.2.1PCI165Annual
SOC 2 Type IIAICPA98Annual
HIPAA Security RuleHHS71Annual
Benchmark versioning

When a benchmark publishes a new version, Calabi releases an updated module within 30 days. Historical scan results are retained so you can compare your posture across benchmark versions.

How Benchmark Dashboards Work​

The Calabi Compliance Engine evaluates each control by querying live AWS resource data and applying the control's logic. No agents are installed in your account — all evaluation is performed through read-only API calls via the Calabi Cloud Operations IAM role.

Running a Compliance Scan​

  1. Navigate to Cloud Operations > Compliance.
  2. Select a benchmark from the list (e.g., CIS AWS Foundations Benchmark v3.0).
  3. Click Run Now to start an immediate scan.
  4. Scans typically complete in 2–5 minutes depending on the number of resources in your account.
  5. Results appear inline as controls complete — you do not need to wait for the full scan.

To set up automatic recurring scans, see Scheduling Compliance Scans below.

Reading a Compliance Report​

Summary panel​

The top of each benchmark report shows:

FieldDescription
Overall scorePercentage of controls that passed (PASS / total evaluated)
Critical controls failedCount of controls marked Critical that failed
High controls failedCount of controls marked High that failed
Last scanTimestamp of the most recent completed scan
TrendScore change vs. the previous scan

Control table​

Below the summary panel, each control is listed in a table with the following columns:

ColumnDescription
IDBenchmark control ID (e.g., 1.4, CIS.1.4)
TitleHuman-readable control name
SeverityCritical / High / Medium / Low
StatusPASS / FAIL / SKIP / ERROR
Affected resourcesCount of resources that failed this control
RemediationLink to the remediation guide

Expanding a control​

Click any control row to expand it and see:

  • The full control description and rationale
  • A list of every affected resource (with ARN, region, and relevant attributes)
  • Step-by-step remediation guidance
  • A direct link to the relevant AWS documentation

Example: CIS 1.4 — Ensure no root account access key exists​

Control:   CIS AWS Foundations 1.4
Severity:  Critical
Status:    FAIL
Affected:  1 resource

Resource:  arn:aws:iam::123456789012:root
Issue:     Root account has an active access key (AKIA...)
Remediation:
  1. Sign in as root.
  2. Navigate to IAM > Security credentials.
  3. Under "Access keys", delete all root access keys.
  4. Verify that programmatic access is delegated to IAM users or roles.

Reference: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html

Scheduling Compliance Scans​

Automated scans ensure your compliance posture is always current without manual intervention.

  1. Open Cloud Operations > Compliance.
  2. Select a benchmark.
  3. Click Schedule.
  4. Choose a frequency: Daily, Weekly, or Monthly.
  5. Set the time and time zone.
  6. Optionally enable Alert on score drop: if the overall score drops by more than N percentage points between scans, an alert is sent to your configured notification channels.
  7. Click Save Schedule.

Scheduled scan results are stored and accessible in the Scan History tab for the last 90 days.

Exporting Compliance Reports​

Reports can be exported for auditors and compliance reviews:

FormatContentsHow to export
PDFFormatted report with benchmark summary, control table, and resource detailsClick Export PDF on the benchmark results page
CSVRaw control-level data: control ID, title, severity, status, affected resource ARNsClick Export CSV on the benchmark results page

PDF exports include your organization name, the scan timestamp, and a Calabi-generated report ID that can be used as an audit reference.

Benchmark Details​

CIS AWS Foundations Benchmark​

The CIS AWS Foundations Benchmark is the most widely adopted prescriptive set of AWS security configuration guidelines. It is organized into five sections:

SectionFocus areaControl count (v3.0)
1Identity and Access Management21
2Storage7
3Logging9
4Monitoring15
5Networking6

Level 1 controls are recommended as minimum baseline configurations. Level 2 controls are more restrictive and suited to environments that require a higher security posture.

NIST 800-53 Revision 5​

NIST SP 800-53 Rev. 5 defines security and privacy controls for federal information systems and organizations. It is commonly used as the basis for FedRAMP, StateRAMP, and DoD compliance programmes. Calabi maps each NIST control to the corresponding AWS resource check where automation is possible; controls that require organizational policy evidence are flagged as manual.

Cloud Foundational Security Best Practices​

AWS FSBP is a cloud-curated set of controls that covers services not addressed by CIS, including ECS, EKS, Kinesis, managed search services, and Secrets Manager. It is updated more frequently than CIS and reflects AWS's current recommended defaults.

  • Security Posture — Continuous security checks outside the benchmark scan cycle
  • Configure — Connect an cloud account and set up alert channels
  • Query Resources — Write ad-hoc SQL for custom compliance checks