Data, Storage & Key Management

Data, Storage & Key Management — Full Guide

Unit 3 of 8. The comprehensive full-coverage note for Data, Storage & Key Management (Key Vault, BYOK, TDE/Always Encrypted, immutable storage, DB protection). Companion notes: quick recall in the cheat-sheet section · the highest-yield concepts in the High-ROI entry · date-sensitive facts, glossary, and the objective index in 99-Appendix. Prerequisite: Identity, Access & Privileged Access — Azure RBAC, role assignments, managed identities, service principals, PIM, Conditional Access. Referenced here, not re-taught.

Part A — The encryption foundation (read this before anything else)

Every data-protection feature in this unit is an instance of one pattern: envelope encryption. A fast symmetric Data Encryption Key (DEK) encrypts the actual bytes; a Key Encryption Key (KEK) wraps (envelopes) the DEK. Rotating or revoking the KEK is cheap (you only re-wrap the small DEK); re-encrypting all the data is not. Only three knobs vary:

1. Who owns the KEK — Microsoft (MMK / platform-managed key) vs you (CMK / BYOK).
2. Where the data is in its lifecycle — at rest, in transit, or in use.
3. How many independent layers — single (service) vs double (infrastructure) encryption.

A.1 Microsoft-managed vs customer-managed keys (CMK / BYOK)

• Microsoft-managed keys (MMK / platform-managed keys, PMK) — the default, zero-config option; Microsoft handles rotation per its compliance obligations. You have no audit trail of, or control over, that rotation.
• Customer-managed keys (CMK) — you generate and own the KEK in Azure Key Vault or Managed HSM. Benefits: full key-lifecycle control (create, rotate, revoke, delete), auditability, separation of duties between key custodians and data owners, and a kill-switch — revoking the key makes the DEK un-unwrappable, so the data becomes inaccessible while remaining encrypted.
• BYOK (Bring Your Own Key) is a subset of CMK: you generate the key in an on-premises HSM and securely import it into Key Vault / Managed HSM (the key material is transferred wrapped, never exposed in the clear). Once imported it behaves like any other CMK.

A.2 Single vs double (infrastructure) encryption

• Service-level encryption (SSE) is the always-on first layer (AES-256, FIPS 140-2). Its KEK may be MMK or CMK.
• Infrastructure-level encryption is an optional second layer using a different algorithm and a separate, always-Microsoft-managed key. The two at-rest layers use separate keys from independent key hierarchies, managed by different operators — for Azure Storage you control the service-level key while Microsoft controls the infrastructure-level key. It defends against the scenario where one algorithm or key is compromised.

Part B — Azure Key Vault & Managed HSM

Key Vault is the keystone of this unit: it stores the keys (CMK/BYOK for Storage, SQL TDE, disks, Cosmos), the secrets (connection strings, passwords, API keys), and the certificates that every other control consumes.

B.1 Choosing the vault — Key Vault vs Managed HSM

B.2 The two planes — the central, most-tested concept

Every Key Vault operation falls into exactly one of two authorization planes:

• Control (management) plane (management.azure.com) — manage the vault resource: create/delete, configure networking, set the permission model, read properties. Always Azure RBAC, scoped at management group → subscription → resource group → vault.
• Data plane (<vault>.vault.azure.net, or <hsm>.managedhsm.azure.net) — actually use the contents: get/wrap/unwrap keys, read/write secrets, manage certificates.

The Key Vault data plane has two authorization models — you pick one per vault:

1. Azure RBAC (recommended) — data-plane roles such as Key Vault Administrator, Key Vault Secrets User, Key Vault Crypto Officer, Key Vault Certificates Officer. Centralized, auditable, supports deny assignments and PIM just-in-time elevation. Crucially, only Owner / User Access Administrator can grant permissions — separating who administers from who uses.
2. Vault access policy (legacy) — assign granular get/list/set/delete permissions per object type directly to a principal at vault scope. No deny, no per-object scope.

B.3 Managing keys, secrets, and certificates

• Keys — RSA / RSA-HSM / EC / oct-HSM. Operations: encrypt/decrypt, wrap/unwrap, sign/verify. Consumed as TDE protectors and Storage/Disk/Cosmos CMKs. Keys are automatically versioned.
• Secrets — arbitrary values up to 25 KB (connection strings, passwords, API keys). Apps fetch them via managed identity — never embed secrets in code or config (cross-ref the workload-identity pattern in Identity, Access & Privileged Access).
• Certificates — managed lifecycle with an issuer (an integrated CA such as DigiCert/GlobalSign, or self-signed); supports auto-enrollment and auto-renewal.

B.4 Recovery — soft-delete & purge protection

• Soft-delete is always on for vaults created after 2019-09-01 and cannot be disabled once enabled. Retention is 7–90 days (default 90), chosen at vault creation and immutable thereafter. Deleted vaults/objects are recoverable; vault names are globally unique, so a soft-deleted name cannot be reused until purged or expired. Note: recovering a soft-deleted vault does not restore integrated services (RBAC assignments, Event Grid subscriptions) — recreate them.
• Purge protection is opt-in, requires soft delete first, and once enabled cannot be disabled or overridden by anyone — including Microsoft. No purge is possible until the retention period elapses. This is the anti-ransomware control for keys, and it is frequently required: Storage CMK requires purge protection to prevent data loss. The tradeoff is irreversibility — it also blocks legitimate immediate reuse of a name.
• Purge (permanent delete during retention) requires the Key Vault Purge Operator role or subscription-owner privileges.
• Managed HSM uses the same model; its soft-delete retention is 7–90 days (default 90), set at HSM creation and not changeable, and soft-deleted Managed HSM resources continue to incur billing until purged.
• Enforce both with Azure Policy (“Key vaults should have soft delete enabled” / “…deletion protection enabled”, deployIfNotExists).

B.5 Backup & recovery — distinct from soft-delete

B.6 Key rotation

• Configure an auto-rotation policy per key: an expiry (ExpiresIn, e.g. 720 days) plus a Rotate lifetime action (e.g. after 540 days) that auto-creates a new key version; Rotate Now triggers an on-demand rotation. Auto-rotation creates a new version with no downtime — old versions remain so existing ciphertext can still be unwrapped.
• Requires the Key Vault Crypto Officer role (or the equivalent rotation permissions under the access-policy model).
• Best practice: rotate at least every two years, more often per compliance. Azure Policy “Keys should have a rotation policy…” audits compliance.
• Auto-rotation enables end-to-end, zero-touch CMK rotation for consuming services (Storage, SQL). Consuming services may take 1–24 hours to pick up a new key version.
• Certificates auto-renew via the issuer policy. Secrets have no native auto-rotate — use the Azure Functions + Event Grid pattern (single-credential, or dual-credential for zero-downtime rotation of resources with two key sets, e.g. storage account keys).

B.7 Network settings

• Firewall — default allow-all; restrict to selected virtual networks (service endpoints) and IP ranges. “Allow trusted Microsoft services to bypass this firewall” lets first-party services (Storage CMK, SQL TDE, Azure Firewall, etc.) reach the vault.
• Private Link / private endpoint — give the vault a private IP in your VNet (private DNS zone privatelink.vaultcore.azure.net), removing the public endpoint. Preferred over firewall for production. Networking primitives themselves live in Secure Networking & Edge Protection.
• Disable public network access — the hardest posture; vault reachable only via private endpoint. Enforceable via Azure Policy “Azure Key Vault should disable public network access.”
• All data/management/auth traffic is HTTPS on 443 (occasional 80 for CRL retrieval). Monitor with Microsoft Defender for Key Vault + Sentinel for anomalous access (see Posture & Governance and Security Operations).

B.8 AZ-500 (operate) ↔ SC-100 (architect) — Key Vault

Part C — Azure Storage

C.1 Access control / authorization (three mechanisms, ranked)

A request to a storage account is authorized by one of:

1. Microsoft Entra ID (OAuth) + Azure RBAC (preferred) — data roles such as Storage Blob Data Reader/Contributor/Owner and the Storage File Data SMB Share roles (plus ABAC conditions on blobs). No keys to leak; integrates with PIM and Conditional Access.
2. Shared Access Signature (SAS) — scoped, time-limited tokens, distinguished by what signs them:
   • User-delegation SAS (best) — signed by an Entra-ID-issued key, Blob only; revoked by revoking the delegation key; no account secret is shared.
   • Service SAS — signed by the account key; one service; can bind to a stored access policy for server-side revocation.
   • Account SAS — signed by the account key; spans services; broadest blast radius if leaked.
3. Shared Key (account access keys) — full account access. Two keys (key1 / key2) exist so you can rotate with zero downtime (serve on key2 while key1 rolls). Disable where possible; if used, rotate regularly and store in Key Vault.

Additional hardening: turn off anonymous public blob access (default off on new accounts), require HTTPS-only (supportsHttpsTrafficOnly), and set a minimum TLS version.

C.2 Managing storage account access keys

• Keep two keys so you can rotate without downtime: switch clients to key2, regenerate key1, switch back. Automate via the Key Vault dual-credential rotation pattern (§B.6).
• Prefer eliminating keys entirely (Entra RBAC / user-delegation SAS). Defender for Storage specifically flags suspicious activity from entities with no identity using leaked / over-permissive SAS (§C.7).

C.3 Access to Azure Files

Azure Files supports identity-based SMB authentication via exactly one identity source per storage account (it applies to all shares in the account), enforced with share-level Azure RBAC roles plus NTFS ACLs:

• On-premises AD DS — the storage account is domain-joined; requires Entra Connect sync of identities.
• Microsoft Entra Domain Services — managed domain.
• Microsoft Entra Kerberos — for hybrid (and cloud-only) identities, using AES-256.

The fallback is the storage account key (full access). For Files over REST, SAS is used; for SMB, Kerberos. Tools: SMB mount, REST, AzCopy, Azure File Sync.

C.4 Access to Azure Blob Storage

Entra RBAC, SAS, Shared Key, or anonymous (discouraged). Choose by caller type: apps → managed identity + RBAC; external partners → user-delegation SAS; never anonymous for sensitive data.

C.5 Data-threat protection (soft delete, versioning, immutable / WORM)

These defend against the ransomware / accidental-deletion / tamper threat — a top SC-100 priority (“design resiliency for ransomware”). Enable blob soft delete first, before applying immutability.

• Soft delete — at blob and container level; deleted data recoverable within a retention window (a recycle bin).
• Versioning — automatically preserves the prior state on each overwrite/delete; protects against malicious overwrite, not just delete.
• Point-in-time restore (PITR) — rolls a block-blob container back to an earlier moment; requires versioning + change feed + soft delete; incompatible with version-level WORM and last-access tracking.
• Azure Backup for blobs/files — operational and vaulted tiers via a Recovery Services vault (see the Capstone (BCDR) unit).
• Immutable storage (WORM) — the strong tamper/ransomware guarantee:

• Container-level WORM — policy at container scope only; no prerequisite.
• Version-level WORM — policy at account / container / version; requires blob versioning; precedence Version → Container → Account; not supported on hierarchical-namespace (ADLS Gen2) accounts, nor on accounts with NFS 3.0 or SFTP enabled; incompatible with PITR and last-access tracking.
• Unlocked vs locked: a new time-based policy is unlocked for testing and provides NO delete protection; locking makes it compliant with SEC 17a-4(f), FINRA 4511, and CFTC 1.31 (validated by Cohasset Associates) and is irreversible (it can only be extended).
• Limit: 10,000 containers per account with unique time-based policies (an account-level policy can exceed that via inheritance).
• Pair immutable + soft delete + versioning for a ransomware-resilient backup target.

C.6 Encryption at rest, in transit

• Storage Service Encryption (SSE) — always on, 256-bit AES, FIPS 140-2 compliant, cannot be disabled, MMK by default.
• Customer-managed keys (CMK) — key in Key Vault or Managed HSM; the account’s managed identity (system- or user-assigned) needs wrapkey, unwrapkey, get, and the vault needs purge protection.
  • Blob and Files are always CMK-protected once CMK is configured.
  • Queue and Table are CMK-protected only if the account was created with an account-scoped encryption key (a creation-time decision).
  • Use an encryption scope for per-container/blob CMK granularity. Cross-tenant CMK is supported (ISV scenario). Azure Policy can require CMK.
• Infrastructure (double) encryption — second 256-bit AES layer, separate Microsoft-managed key, different algorithm — enable at account or scope creation only (§A.2).
• In transit: “Require secure transfer” forces HTTPS; set a minimum TLS version.

C.7 Microsoft Defender for Storage (detective layer)

Encryption and immutability are preventive; Defender is detective. Agentless, enabled at subscription scope (auto-covers existing + new accounts, with opt-out) or per-resource; protects Blob, Files, Data Lake; no diagnostic/resource logs required.

• Activity monitoring — ML + Microsoft Threat Intelligence detect anomalous access, exfiltration, malicious IPs/Tor, and suspicious SAS use by entities with no identity (leaked / over-permissive SAS).
• Malware scanning — Microsoft Defender Antivirus; on-upload (scans each blob as written) and on-demand; billed per GB scanned.
• Sensitive data threat detection — integrates Microsoft Purview sensitive-information types & labels to prioritize alerts on the data that actually matters (toggle at no extra cost).
• The current plan supersedes Defender for Storage (classic) — migrate for malware scanning + sensitive-data detection.

C.8 AZ-500 (operate) ↔ SC-100 (architect) — Storage

Part D — Azure SQL, SQL Managed Instance, Synapse & Cosmos DB

D.1 Microsoft Entra database authentication

• Microsoft Entra authentication (recommended) — centralized identity, SSO, MFA, Conditional Access, no password proliferation, single-place rotation, group-based access. Identities can be users, groups, service principals, or managed identities (the recommended app pattern).
• SQL authentication — username/password. Windows authentication is not supported in Azure SQL Database.
• Critical nuance: Entra auth for Azure SQL does NOT integrate with Azure RBAC. You must create the identity as a database principal — CREATE LOGIN ... FROM EXTERNAL PROVIDER (server principal) or CREATE USER ... FROM EXTERNAL PROVIDER (contained user). The SQL Server Contributor / SQL DB Contributor Azure roles govern management-plane operations, not database connectivity.
• Prerequisite: set a Microsoft Entra admin on the logical server, then create the contained DB users. Enabling this is the AZ-500 task.

D.2 Database auditing

• Tracks DB events; destinations are Azure Storage account, Log Analytics workspace, or Event Hubs (you can target multiple). Configurable at server level (applies to all DBs) or database level. Feeds Microsoft Sentinel for hunting (see Security Operations).
• Storage-destination auth: use Managed Identity (system- or user-assigned) — Microsoft warns against storage access keys (compromise risk).
• Default retention = 0 (unlimited). Changing it only affects logs written after the change.
• Entra failed logins do NOT appear in the SQL audit log — they are recorded in the Microsoft Entra admin center. (Successful logins are audited in both; the gateway rejects bad Entra credentials before the DB is reached.) When targeting Log Analytics / Event Hubs, a diagnostic-settings resource is auto-created — if deleted, auditing fails silently; alert on its deletion.
• Immutable audit logs: write to a WORM container with “Allow protected append writes” set to Append blobs or Block and append blobs (the None option is unsupported), and the storage retention must be shorter than the SQL Auditing retention.

D.3 Transparent Data Encryption (TDE) — encryption at rest

Encrypts the whole database at rest (data + log + backups) transparently with AES, via the Database Encryption Key (DEK). On by default for newly created Azure SQL databases. Applies to Azure SQL Database, SQL Managed Instance, and Azure Synapse Analytics dedicated SQL pools. Protects against offline theft of files/backups; the service itself can read plaintext (it is at-rest only). Two models:

• Service-managed — the DEK is protected by a built-in server certificate (AES-256) that Microsoft rotates.
• Customer-managed (CMK / BYOK) — the TDE protector, a customer-managed asymmetric (RSA 2048+) key in Key Vault or Managed HSM, wraps the DEK and never leaves the vault. The logical server’s managed identity (system- or user-assigned) needs Get / WrapKey / UnwrapKey on the vault. Configurable at server level (inherited by all DBs) or per database (enables ISV/multi-tenant, even cross-tenant, key isolation).

D.4 Always Encrypted (AE) — encryption in use (client-side)

Column-level encryption performed by the client driver; the encryption key is never exposed to the SQL engine, so DBAs, cloud operators, and anyone server-side see only ciphertext. Two-key hierarchy: a Column Encryption Key (CEK) encrypts the data, and a Column Master Key (CMK) wraps the CEK and lives in Azure Key Vault or the Windows Certificate Store.

• Deterministic encryption — same plaintext → same ciphertext; supports equality, joins, grouping, indexing, but leaks value patterns.
• Randomized encryption — stronger; no server-side operations without enclaves.
• Always Encrypted with secure enclaves — a trusted execution region inside the engine that can handle keys, enabling range / LIKE / pattern / sort queries and in-place encryption/re-encryption without exporting data. (Enclave operations require randomized encryption and compatibility level 160+.)

D.5 Dynamic Data Masking (DDM) — obfuscation on read

A policy-based presentation-layer feature: masks designated columns in query result sets for non-privileged users; the stored data is unchanged — it is NOT encryption. Mask functions: default (full), email, random (numeric), custom string (partial). Applies to Azure SQL Database, SQL Managed Instance, Synapse dedicated pools, and Fabric SQL.

D.6 Data Discovery & Classification

Built into Azure SQL Database, SQL Managed Instance, and Synapse — discovers and labels sensitive columns and computes real-time result-set sensitivity for auditing. This is the Azure-data flavor; Microsoft 365 sensitivity labels / DLP and the broader Purview Data Map belong to the governance unit (Posture & Governance). SC-100 drives the controls (DDM, AE, auditing) from these classification labels.

D.7 Synapse & Cosmos DB specifics (SC-100 breadth)

• Synapse — TDE (service- or customer-managed) on dedicated SQL pools; DDM supported; Entra pass-through to the data lake; serverless-pool storage access via database-scoped credentials (Managed Identity / SAS / service principal). Auditing to a firewalled storage account requires the workspace’s system-assigned MI with Storage Blob Data Contributor (UAMI not supported).
• Cosmos DB — SSE always on; CMK = automatic double encryption on transactional storage (not Synapse Link / continuous backup, §A.2); server-side Dynamic Data Masking for NoSQL; RBAC + private endpoints for access. Threat protection via Microsoft Defender for Azure Cosmos DB — API for NoSQL only — detecting SQL-injection variants, anomalous access (Tor / odd geo / unusual apps), and suspicious key-listing / exfiltration.

D.8 Microsoft Defender for Databases (detective layer)

Four separately-priced offerings:

• Defender for Azure SQL Databases
• Defender for SQL Servers on Machines
• Defender for Open-Source Relational Databases (PostgreSQL / MySQL / MariaDB, incl. AWS RDS)
• Defender for Azure Cosmos DB (NoSQL API only)

For SQL = Vulnerability Assessment (finds misconfigurations, excessive permissions, missing hardening) + Advanced Threat Protection (SQL injection, anomalous logins, unusual access patterns). One enablement covers all DBs on a server/MI; managing it needs the SQL Security Manager role (or DB/server admin). Alerts surface in Defender for Cloud and stream to Microsoft Sentinel for correlation/SOAR.

D.9 Choosing the encryption layer — the design heuristic

D.10 AZ-500 (operate) ↔ SC-100 (architect) — SQL / Synapse / Cosmos

Part E — The operate ↔ architect throughline

The unifying exam heuristic

For every control in this unit: AZ-500 = “configure/enable the knob”, SC-100 = “decide whether this knob is the right control for this data class, compliance regime, and threat — and how it composes with the others.” Master the knobs and the layering logic together and you pass both with one pass through this unit.

Part F — High-yield gotchas (highest miss-rate)

Part G — Cross-references

• Prerequisite: Identity, Access & Privileged Access — RBAC, managed identities, PIM, Conditional Access (every authorization here resolves through it).
• Adjacent units:
  • Secure Networking & Edge Protection — Private Link / Private Endpoints for Key Vault and Storage; TLS.
  • Secure Compute, Containers & Endpoints — disk encryption (ADE, encryption-at-host, confidential disk) that also consumes Key Vault keys via Disk Encryption Sets.
  • Posture, Governance, Multicloud & Compliance — Defender for Cloud posture / Secure Score; Microsoft Purview data discovery, sensitivity labels & DLP; Azure Policy.
  • Security Operations: Sentinel, Defender XDR, SOAR — Sentinel / Azure Monitor / Log Analytics for auditing and alert response.
  • Capstone: Apps, M365 Data Protection & Strategy — Backup, BCDR & ransomware-resilience strategy.
• Recall & index: Cheat Sheet — Unit 2 · High-ROI — Unit 3 · 99-Appendix · Strategy & Map

Part H — Authoritative references

• Storage service encryption (SSE)
• Enable infrastructure (double) encryption
• Double encryption at rest
• Storage customer-managed keys
• Cosmos DB customer-managed keys & double encryption
• Managed disk encryption (double encryption at rest)
• Key Vault RBAC guide · RBAC vs access policy
• Managed HSM access control · Managed HSM overview
• Key Vault recovery (soft-delete & purge protection)
• Key Vault backup & restore
• Configure key rotation
• Key Vault network security
• Manage storage account access keys · Prevent Shared Key authorization
• Storage SAS overview · Azure Files identity-based auth
• Immutable storage for Blob Storage · Version-level WORM
• Transparent Data Encryption (TDE) · TDE with BYOK
• Always Encrypted (Database Engine) · Always Encrypted with secure enclaves
• Dynamic data masking · Entra authentication for Azure SQL
• Azure SQL auditing
• Microsoft Defender for Storage · Microsoft Defender for Databases
• AZ-500 study guide · SC-100 study guide