---
title: Comprehensive Guide to AWS CloudTrail: Centralized Audit Logging, API Call Tracing, and Security Monitoring Best Practices
created: 2026-04-02
updated: 2026-04-02
type: summary
tags: ["audit", "logging", "notes", "security", "telegram"]
external: http://go/ui/posts/2330/
---

AWS has a centralized audit trail system built around **CloudTrail**, with a few other services complementing it. Here's how it all fits together:

## AWS CloudTrail — The Core Audit Service

CloudTrail records **every API call** made in your AWS account, regardless of how it was triggered — AWS Console (UI), CLI, SDK, CDK, Terraform, or a Lambda function. It captures:

- **Who** made the call (IAM user, role, service)
- **What** action was taken (e.g., `ec2:DescribeInstances`, `s3:GetObject`)
- **When** (timestamp)
- **Where from** (source IP, user agent)
- **What changed** (request/response parameters)

### Where to find CloudTrail logs

**Console:** `CloudTrail → Event history` — gives you a 90-day searchable view with no setup needed, great for quick lookups.

**S3:** If you've created a Trail, logs are delivered to an S3 bucket (usually within ~15 min). Path pattern:
```
s3://your-bucket/AWSLogs/<account-id>/CloudTrail/<region>/YYYY/MM/DD/
```

**CloudWatch Logs:** You can configure a Trail to also stream to a CloudWatch Log Group, which lets you set metric filters and alarms on suspicious activity.

---

## Tracing IAM Role Assumption

This is a common audit scenario. When someone does `sts:AssumeRole`, CloudTrail logs two things:

**1. The AssumeRole call itself** — logged in the *calling account* or the account where the principal lives:
```json
{
  "eventName": "AssumeRole",
  "userIdentity": {
    "type": "IAMUser",
    "userName": "john.doe"
  },
  "requestParameters": {
    "roleArn": "arn:aws:iam::123456789:role/DevOpsRole",
    "roleSessionName": "john-session"
  }
}
```

**2. All subsequent API calls made with that role** — these show up with `userIdentity.type = "AssumedRole"` and include the `sessionContext` showing both the role ARN and the session name:
```json
{
  "eventName": "DescribeInstances",
  "userIdentity": {
    "type": "AssumedRole",
    "arn": "arn:aws:iam::123456789:assumed-role/DevOpsRole/john-session",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "userName": "DevOpsRole"
      }
    }
  }
}
```

So you can correlate: find the `AssumeRole` event → grab the `roleSessionName` → search all events with that session name to reconstruct exactly what that person did.

---

## UI vs Code — How to Tell Them Apart

CloudTrail logs the `userAgent` field which reveals the source:

| Source | userAgent example |
|---|---|
| AWS Console | `AWS Console` or `signin.amazonaws.com` |
| AWS CLI | `aws-cli/2.x.x` |
| Terraform | `APN/1.0 HashiCorp/1.0 Terraform/1.x` |
| Python SDK (boto3) | `Boto3/1.x.x Python/3.x` |
| CDK | `aws-cdk/...` |

---

## Querying Logs Efficiently

**CloudTrail Lake** — the modern approach. It's a managed SQL query engine over your CloudTrail events (no S3 parsing needed):
```sql
SELECT eventTime, userIdentity.arn, eventName, sourceIPAddress
FROM <event-data-store-id>
WHERE eventName = 'AssumeRole'
  AND eventTime > '2026-03-01 00:00:00'
ORDER BY eventTime DESC
```

**Athena over S3** — if you're storing logs in S3, you can point Athena at them with a Glue table and query with SQL. Good for long-term/historical analysis.

**CloudWatch Logs Insights** — good for real-time querying if you've streamed CloudTrail to a Log Group:
```
fields @timestamp, eventName, userIdentity.arn, sourceIPAddress
| filter eventName = "AssumeRole"
| sort @timestamp desc
```

---

## Other Complementary Services

| Service | What it adds |
|---|---|
| **AWS Config** | Records *resource state changes* over time (what did this S3 bucket's config look like last Tuesday?) |
| **GuardDuty** | Threat detection on top of CloudTrail — flags anomalies like unusual role assumptions or API calls from unexpected geos |
| **IAM Access Analyzer** | Shows what external entities have access to your resources |
| **Security Hub** | Aggregates findings from GuardDuty, Config, Inspector into one dashboard |
| **S3 Access Logs / Data Events** | CloudTrail *management events* are on by default; S3 *data events* (GetObject, PutObject) need to be explicitly enabled in your Trail — they're high volume |

---

## Key Things to Know for a Staff-Level Context

A few nuances worth being aware of:

- **Data events are opt-in** — by default CloudTrail only logs *management plane* calls (create/delete/modify resources). To audit S3 object reads or Lambda invocations, you must enable data events on the Trail, which significantly increases cost/volume.
- **Cross-account role assumptions** — if Account A assumes a role in Account B, the event appears in *Account B's* CloudTrail (where the role lives). You need an org-level Trail or log aggregation to get the full picture.
- **CloudTrail Lake vs Athena** — Lake is simpler (no schema management, faster setup) but costs more per query; Athena on S3 is cheaper at scale but needs Glue table setup and partition management.
- **Log integrity** — CloudTrail supports log file validation (SHA-256 digest files), which is important for forensic/compliance use cases to prove logs haven't been tampered with.
